FHWA conducted a study to explore the possibilities for AR technology in roadway construction.

Field operations in the transportation sector present many challenges. Among them are the lack of real-time and integrated information, gaps between planned solutions and practical implementations, quality assurance, and communications among project participants. As a result of the rapid advancement in computer interface design and hardware, augmented reality (AR) may be a tool to help overcome some of these obstacles.

 

 

AR offers an immersive technology that overlays virtual computer-generated information with the real environment in real time, enhancing the user's perception of reality and enriching the provided information content. This blending of project-specific information with the real-world site view can assist project managers and engineers with the delivery of their projects in a safe and timely manner and with greater efficiency and accuracy. In addition, with the ability to navigate through all phases of a construction project, managers can detect errors before they occur or change the design and construction details.

"Considering AR's benefits and success in the entertainment and video game industries, leveraging AR appears to be an opportunity in construction management for highway infrastructure assets," says Dr. Kelly Regal, Associate Administrator, FHWA Office of Research, Development, and Technology.

Over the last year, FHWA conducted a comprehensive study to investigate available AR technologies, their reliability and practical application, and how these technologies can be applied to construction management. The results of that study follow.

Hand-held or Head-mounted?

There are two basic categories of display types in AR systems: hand-held mobile devices, such as smartphones and tablets, and head-mounted display (HMD) devices, such as headsets or glasses. Primary differences include the way each device displays imagery to users and how the devices track their position relative to the real world.

Hand-held devices are typically video see-through displays that use the back-facing cameras on the device to capture video of the real-world environment and display that image on the front screen. With these displays, the device needs to be held close to eye level and at arm's length to capture the widest field of view—which can be difficult over long periods of time and challenging in a construction site environment. Hand-held devices typically use a global navigation satellite system (GNSS) to determine the initial user location within a few meters, and then use the inertial movement of the device to change the view as the device is moved around.

 

 

Some vendors have demonstrated prototype applications that optically track the imagery in the video feed to support registration with the real-world view and track movement. Many commercially available AR applications use a marker-based positioning tool, where a target is placed in the real world and viewed by the video feed to register real-world position relative to the virtual model.

Several off-the-shelf AR applications enable the user to display a 3D model over the live video feed of a mobile device. The application is given a "target" image, either graphical (for example, a plan or rendering) or a photograph. The image is printed out, and the application on the mobile device recognizes the target image and displays the 3D model in alignment as defined in the application. The model view is locked to the position of the target image in the video feed.

HMDs are typically optical see-through displays where the device's view of the real world is overlaid by a virtual computer-generated image. These devices offer more immersive and realistic experiences than hand-held devices because the real-world view is direct and the virtual view is typically stereoscopic. The scene changes as the user's head moves. Some HMD devices have built-in audio commands to change the view parameters. The audio controls help maintain hands-free operation. HMDs are typically outfitted with more sophisticated and accurate tracking technology than off-the-shelf hand-held devices, as the blending of virtual imagery and the direct view out of the device requires more precise alignment of the imagery for true immersion and user comfort. Like hand-held devices, the user's initial positioning with an HMD relies on GNSS tracking or a marker in the field.

 

 

Some AR HMD devices can leverage the 3D scanning hardware used for AR tracking to capture existing 3D data in the field. The user views these virtual model elements superimposed over the real-world view and can line up the model elements to identical elements in the real world. The precision of the data depends on several factors, including available site survey information and the scanning precision of the sensor hardware.

"On a typical highway construction site with good survey targets and a robust 3D-model-based workflow, this ability to capture accurate data in real time using only an HMD device can become an option for site inspections," says Katherine Petros, leader of the Infrastructure Analysis and Construction Team in FHWA's Office of Infrastructure Research and Development.

Tablet devices are making more rapid advances than HMD devices in the highway construction field because they have comparatively fewer display and viewing limitations. The tradeoff is less accurate tracking technology, and therefore, less precision of the registration of virtual to real-world imagery. However, industry researchers are trying to develop tablet devices with integrated special hardware and software to provide greater precision tracking and better registration of virtual 3D data to the real-world view.

Challenges for AR Systems in Construction Environments

Construction sites—especially highway construction sites—are particularly challenging for AR systems. Highway projects are typically made up of large, smooth, and flat objects lacking in fine details, making it more difficult for AR systems to track their location within the scene. Additionally, it is difficult for 3D modeling applications to represent these types of objects to be easily recognized by the user and the AR system. As a result, these elements require more preparation for their use in AR systems.

In addition, there are several technological barriers. AR systems require significant processing power and enough onboard storage to concurrently support tracking processes and the real-time display of the virtual 3D model. To be useful in a construction environment, AR devices must be standalone and portable, which means processing for tracking and display must be on board the device, not augmented by an external source. Some systems are tethered to a separate wearable computing device that reduces the necessary weight on an HMD. Larger 3D models, more accurate tracking, and increased display quality require more processing power. As AR systems evolve, there will be a tradeoff between performance of the system and the size, weight, and comfort of the AR device.

 

 

The brightness of the real-world environment presents a key challenge with HMDs. Typical highway construction sites are outdoors and bright, which limits the quality and usefulness of most current AR HMD devices. Bright, open environments are more challenging for the tracking technology to follow. Hand-held devices control the real-world display on the screen—the video display and virtual displays are better matched in overall brightness, providing better quality and greater realism.

"In the near term," says Adrien Patané, a regional manager with an AR technology designer and manufacturer, "hand-held devices likely provide better opportunities for AR on outdoor construction locations."

Another drawback of both hand-held and HMD devices is the limited field of view of the overlaid virtual model presented to the user. Users must pan around with the device or turn their head back and forth to view a large area, which could be prohibitive over long periods.

Another technological challenge with AR systems is occlusion, or masking of hidden elements, which may be needed in complex construction environments. When the occlusion is ignored or displayed poorly, it negatively impacts the realism and immersive quality of the displayed scene.

AR systems also present difficulties because of the safety risks on a construction site. Like any display device, some user attention will be focused on the device and not entirely on the surroundings. Hand-held devices are held in front of the user, require the use of at least one hand, and can block visibility. HMDs can limit the user's peripheral view and block ambient sounds. It may be possible to address this challenge in the future if devices could be designed to recognize safety issues and risks for the user by knowing the user's precise location on the site.

AR Applications for Highway Construction

Industry and technology manufacturers are developing AR applications for construction to display annotations and graphical information that enhance the understanding of real-world objects, combined with increased locational accuracy. In addition, 3D design and construction models can be overlaid, in their real-world position, in the view of the existing environment.

AR technology can display what is not yet constructed, enabling users to see 3D design models in the real-world context. It provides the user with the opportunity to compare design alternatives in context, check relationships between existing and future elements, monitor site logistics and equipment movements, and illustrate construction methods and sequencing.

After construction, AR can overlay and compare 3D design models (design intent) onto the end result in the field to inspect the construction, monitor compliance with codes and standards, and check quantities and work progress.

AR also can display existing elements that are not visible to the user in the real world, such as buried utilities or structural components or other elements obstructed from the current view. AR can show abstract information, such as alignment information, easements, site and right-of-way boundaries, environmental boundaries such as flood levels or sea-level rise data, potential work zone hazards, and metadata tagged to real-world objects. AR can also display unsafe areas and risks or guide users securely through a construction site.

AR systems prove useful for construction inspection—for example, for measuring areas of newly installed concrete pads to calculate contractor payment amounts. The user can capture points with the AR device using finger gestures and a cursor placed over real-world points, and the device measures the areas of the surface enclosed. AR technologies can also be complemented by distance measurement capabilities, which enhance engineering and inspection functions by being able to observe distances between reality and proposed designs.

Most platforms for 3D design applications support review and collection of data in the field through mobile devices. The field data are then synchronized with project models and data in the office. This provides an opportunity for collaboration between the field and office and ensures that the user with the device always has the latest, correct version of the model, which may also serve as a digital as-built to be used in future. While most of the tools offer ways of optimizing models and model display on mobile devices with more limited processing power and storage, this optimization will be even more important for AR devices and applications that require enough graphics performance to support real-time stereoscopic rendering of the 3D models.

UDOT's Perspective

Since 2016, the Utah Department of Transportation (UDOT) has been awarding select projects using 3D models as the legal document (MALD). To date, UDOT has awarded 11 projects with MALD, and fully constructed 8 of them. Although most of these projects also have included paper plan sets for information only, UDOT inspectors and contractors have used survey rovers and mobile devices (with the 3D design model loaded) in lieu of the plans sets. In fact, while most construction crews began projects referencing the paper plan sets, all have stated that the use of mobile devices was easier and more efficient than using the plan sets.

In 2018, UDOT awarded its first project with MALD only, without creating and printing plan sheets. That project success led UDOT to forgo cutting sheets on later projects with MALD. The success of using mobile devices in the field is a significant precursor to the promise of AR use.

UDOT has used vendor apps on tablets in the field to inspect projects with MALD. However, these tools have not been user-friendly, and model authors have complained about added time when checking that their design information is transferred to the mobile device.

UDOT's geographic information system (GIS) group has developed user-friendly solutions that can be applied in the field with a phone or tablet. UDOT construction inspection crews have been pleased with the tools, which led to UDOT developing additional workflows for GIS tools in the field. In addition, UDOT is experimenting with the ability of these tools to also write digital information to a central set of databases, expecting the practice to replace paper or PDF plan sheet as-builts.

"Expanding the availability of 3D design models in the field with vendor apps or GIS tools will lead to a smooth transition to AR, likely on multiple platforms," says George Lukes, a standards and design engineer with UDOT.

UDOT has worked with vendors on construction projects with AR. The tool UDOT tested could extract all of the features in the 3D design model. While the user held a mobile device, the features were projected on the screen in real time as the user rotated or moved the mobile device. When the field demonstration was done in late 2018 and early 2019, important attributes such as striping color, sign size, and draining box dimensions from the 3D design model were not yet available on mobile devices with the vendor's AR app. Even without the attributes, crews commented on the value and were excited at the prospect of using AR with 3D design model attributes included.

 

 

With an improved tool, UDOT expects to test out AR use again on a project in summer 2020. UDOT anticipates that most, if not all, of the 3D design model attributes will be included in the AR app.

For UDOT, the most valuable use for AR will probably be for underground utilities during design, construction, and asset management and operations. Although UDOT anticipates AR to be useful and valuable, there will be short-term gaps with most, if not all, information in 3D design models. Until a fully geospatial utility database is populated, the geospatial location of angle points, valves, and other details often will not be known during the design phase of the project.

UDOT is currently working toward populating a utility database. One benefit of such technologies is that they can lead to the proper recording of utility placement during construction. Once the utilities are observed in the field by AR devices, their locations and attributes are recorded as digital as-builts for future applications.

"Although it may take a significant amount of time to populate an underground utility database, the potential for substantial reduction or elimination of delays due to utility conflict is impressive," says Lukes. "AR will undoubtedly increase construction efficiency with accurate representations of the underground utilities."

FDOT's AR Modeling

In Florida, AR is reshaping the how District Five of the Florida Department of Transportation (FDOT) delivers projects. The agency uses the technology in almost every aspect of a project. Starting in the design phase, District Five is using AR to review 3D design models. The AR technology not only enables designers to identify errors, it also helps them to easily identify constructability issues with a design.

For example, designers have used AR to identify conflicts between drainage structures and existing utilities. While these conflicts can be seen in a computer-aided design and drafting environment, constructability of the inlet is not always as obvious. Nearby utility poles or other hard features can sometimes impact the constructability of the drainage structures. AR makes these conflicts easily visible.

FDOT also uses AR technology to review and visualize a model and get feedback from stakeholders in the design phase. Instead of the stakeholders reviewing a project on a 2D sheet of paper, they are able to view an AR model in the field. This visual overlay results in a faster review and a better overall understanding of the project.

FDOT is moving into its first construction project using AR. The team is using automated machine guidance to construct a majority of the elements. The project has no plan sheets and instead has a signed and sealed 3D model as the contract document. This puts additional emphasis on the model compared to other projects. AR will enable crews to quickly check constructed elements against the proposed model. The AR technology will also enable contractors to review form work before concrete is poured. Any element that needs to be laid out during construction can be quickly checked against the proposed model with AR technology.

After construction, the benefits of AR continue. FDOT's maintenance offices will be able to use the 3D as-built record for future maintenance and rehabilitation with a high degree of accuracy.

Michigan and California

Other States are using AR technologies to improve their processes and outcomes as well.

 

 

The Michigan Department of Transportation used AR technologies for visualization of bridge replacements on the I–94 Advanced Bridges Project in downtown Detroit. The ability to visualize the new bridge design over the existing aging bridge, including abutments and structural details, enabled the project team to review proposed design elements overlaid on reality. This provided the designers with unique perspectives and contextual placement of their 3D designs. Future applications of AR technology could include verifying that existing 3D models containing underground utilities are accurate, visualizing potential conflicts between the proposed design and existing facilities, and communicating the proposed design to affected project stakeholders.

 

 

 

In Sacramento, CA, a recent construction manager/general contractor study of new bridge construction incorporated AR technologies. The team could review the 3D model in the office and then use the AR model onsite for visualization of bridge pilings, right-of-way, and structure components, as well as clearly identify an underground high pressure jet fuel line that ran parallel to the new bridge.

FHWA Research Findings

AR hardware, software, and applications are rapidly changing and improving. The findings of the FHWA research study of current technologies, opportunities, and challenges all point to potential future directions for AR use in highway construction. However, future applications will require investment in specific hardware and software frameworks. The FHWA study identified the following top five potential applications in terms of possible impact on transportation construction and feasibility of development:

1. Supporting right-of-way acquisition and providing project visualization to property owners to better understand project impacts and design options.
2. Visually annotating the schedule and quality variances in the field to ensure that all parties view the same issues.
3. Verifying proper installation by providing the right information and format for inspectors in the field.
4. Supporting training and certification for construction inspection.
5. Supporting automated compliance checks of codes and standards for installed items through machine learning.

 

 

These identified potential applications can serve as starting points for transportation agencies and developers to focus their AR development and implementation efforts in the near- and long-term future. The most important requirements for each application are the software development; the coordination of hardware as an integrated working solution; and the application for the user interface, data input and output, and 3D model management. The application and hardware should be seamless so that the users can focus on the tasks they need to accomplish.

The lack of information on return on investment to justify AR implementation is a challenge. This is a traditional barrier to new technologies; however, independent field trials have helped address the concern and further the development and implementation of the technologies. Multiple objective field trials that examine the benefits and costs of AR based on empirical results will further alleviate this obstacle.

"Just as innovations in automated machine guidance, inspection, and quality assurance using unmanned aerial systems are changing highway construction for the better, advancements in AR technologies offer the potential to facilitate construction inspection processes," says Hari Kalla, FHWA's Associate Administrator for Infrastructure. "Innovations such as 3D-model-based workflows that are being advanced through FHWA's Every Day Counts initiative will provide opportunities for using AR on highway construction projects in the future."

---

Hoda Azari is the manager of the Nondestructive Evaluation (NDE) Research Program and NDE Laboratory at FHWA's Turner-Fairbank Highway Research Center. She holds a Ph.D. in civil engineering from the University of Texas at El Paso.

Kevin Gilson is the director of design visualization in the project visualization group with an international transportation consultancy firm. He oversees visualization production, technology development and innovation, implementation of building information modeling, and 3D-model-based design processes within the firm. He holds an M.A. in design from the University of California, Berkeley.

For more information, contact Hoda Azari at Hoda.Azari@dot.gov.