Deploying fully autonomous vehicles has been a subject of intense research in both industry and academia. However, the majority of these efforts have relied heavily on High Definition (HD) prior maps. These are necessary to provide the planning and control modules a rich model of the operating environment. While this approach has shown success, it drastically limits both the scale and scope of these deployments as creating and maintaining HD maps for very large areas can be prohibitive. In this work, we present a new method for building the HD map online by starting with a Standard Definition (SD) prior map such as a navigational road map, and incorporating onboard sensors to infer the local HD map. We evaluate our method extensively on 100 sequences of real-world vehicle data and demonstrate that it can infer a highly structured HD map-like model of the world accurately using only SD prior maps and onboard sensors.