USA Banner

Official US Government Icon

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure Site Icon

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

U.S. Department of Transportation U.S. Department of Transportation Icon United States Department of Transportation United States Department of Transportation

Public Roads - March/April 2016

March/April 2016
Issue No:
Vol. 79 No. 5
Publication Number:
Table of Contents

The World of Tomorrow Is Today

by Steven Lottes, Kornel Kerenyi, and Cezary Bojanowsk

High-performance computing is helping pave the way to the bridges of the future. Read on for the straight skinny from FHWA’s Hydraulics Research Program


FHWA researchers used high-performance computational capabilities to do a virtual test of a scour countermeasure proposed for installation at this California railroad bridge, where countermeasures to control flood erosion are complicated by environmental restrictions.


Many companies and organizations are now using advanced computer analysis to reduce the costs of developing products, improve their safety and reliability, and decrease the time needed to develop them. Computational fluid dynamics and computational structural mechanics now are routinely employed in the design of vehicles, ships, and aircraft. Computational analysis is applied to as many of the products’ components as possible, including streamlining body shape to reduce drag and optimizing engine components to achieve fuel and cost savings.

Other industries and groups using computational advances include food and chemical processing, petroleum exploration and production, and medical research, including biomechanical engineering. In meteorology, high-performance computing is yielding much more accurate weather forecasts.

The successful application of modern computational mechanics in a wide variety of industries suggests that it can and should be applied to transportation research and development. To this end, the Federal Highway Administration’s Hydraulics Research Program began expanding its computational capabilities as early as 2007.

Prior to these applications, researchers at FHWA’s J. Sterling Jones Hydraulics Laboratory at the Turner-Fairbank Highway Research Center in McLean, VA, would conduct physical experiments using scaled models placed within a large hydraulic flume. Conducting these physical model experiments could be costly and lengthy. As a replacement approach, the researchers currently are using high-performance computing capabilities provided by Argonne National Laboratory’s Transportation Research and Analysis Computing Center (Argonne TRACC). In addition to cost and time savings, another advantage of using advanced computing is the ability to conduct full-scale simulations of real-world hydraulic structures such as bridges and culverts, while avoiding difficult scaling issues that occur when using an experimental approach.

“Given ongoing funding challenges, spending available funds as effectively as possible is important,” says Michael F. Trentacoste, FHWA associate administrator for research, development, and technology. “Not only can transportation researchers use computer analysis to conduct simulations of new structures, but also to evaluate old structures and scour countermeasures to see if they can withstand more frequent and larger weather events.”

What Exactly Is Advanced Computing?

FHWA researchers are conducting advanced computing studies using these high-performance computer clusters at Argonne National Laboratory’s Transportation Research and Analysis Computing Center.


Currently, FHWA’s hydraulics researchers are conducting detailed and automated experimental work to calibrate computational fluid dynamics simulations, and they will be doing even more in the next 10 to 15 years. The computer simulations in a virtual world can be laboratory scale or full scale and include as much real-world detail as needed to compute storm surges, flood flows at bridges and culverts, and other events.

Most of the problems to which computational fluid mechanics is applied require far too much data for a typical or even a high-end office desktop computer. One example is assessing whether bridge foundations could be undermined and experience failure during a flood. This assessment requires computing the flow one-third of a mile (0.54 kilometer) upstream and downstream of a bridge to determine erosion forces.

For an accurate assessment of the risk of foundation failure, this kind of computation must solve equations governing the flow and related forces at millions or tens of millions of points distributed throughout the volume of a river. The huge amount of data on velocity and forces at millions of points is usually processed to produce detailed visualizations of what is happening in the flow. In addition, the data must be reduced by averaging or summing to obtain a much smaller number of numerical results, such as the flood forces on a bridge. This reduction is similar to high-resolution versus low-resolution photographs taken by a digital camera. High resolution gives a much better picture of what is happening.

Modern high-performance computer clusters are able to solve these kinds of large problems. The clusters consist of a large number of computers in racks, cabled together with both gigabit per second Internet plus an extremely high-speed InfiniBand interconnect. The high-speed interconnect is the key to enabling the machines to function together as one large computer.

In the case of a large flood flow problem, the analysis software breaks it into many smaller pieces and spreads them over a large number of processors. Each processor works on only a small part of the problem. The solutions to the pieces of the problem are highly dependent on what is happening at the interfaces with neighboring pieces, like a vortex moving with the flow from an upstream piece into a downstream piece. The needed information at the interfaces is communicated to processors working on neighboring pieces through the high-speed interconnect.

Using this methodology, high-performance computer clusters can solve problems in a couple of days--or even a few hours in some cases--that would take months on a desktop computer. And, frequently, these problems are too big to fit in the typical desktop computer’s memory.

Several recent and current examples of this type of applied research illustrate some of the ways that FHWA and Argonne TRACC researchers are using high-performance computer clusters and computational mechanics to help solve transportation infrastructure problems, assess risks, and improve designs.

Assessing Scour Countermeasures at A California Bridge

The problem: How can you optimize a new design for a scour countermeasure when you have only enough time to set up and run a few physical model tests in the laboratory at one-thirtieth scale?

In the case of a California bridge, the answer turned out to be using high-performance computing to run more than 50 simulations of flood flows with varying full-scale design alternatives on a computer cluster. After FHWA and Argonne TRACC researchers selected the best candidate from this large number of simulations, researchers from the U.S. Army Corps of Engineers built a scale model and tested it in the Corps’ laboratory to confirm that the optimized design provided the needed protection against scour.

The BNSF (formerly the Burlington Northern and Santa Fe) Railroad Bridge over the Santa Ana River, downstream of Prado Dam in Riverside County, CA, is classified as scour critical. This classification means that riverbed erosion around the foundation during major floods could compromise the bridge.

In addition, the presence of the threatened Santa Ana sucker fish in the river creates constraints on the type of countermeasures that could be used to protect the bridge. In fact, requirements for protecting the fish eliminate most of the common approaches, including riprap (stones or chunks of concrete piled together to prevent erosion), which would create a serious fish passage problem. To protect the bridge and satisfy the environmental constraints, the U.S. Army Corps of Engineers, Los Angeles District, developed a new countermeasure design.

The proposed design encases the four central sets of existing piers with driven sheet pile and the construction of triangular concrete pier extensions extending 50 to 200 feet (15 to 61 meters) upstream at each pier group. The pier extensions are tapered from a width of 26 feet (8 meters) at the piers to 2 feet (0.6 meter) at the noses of the pier extensions. The design goal was to shift the potential for scour upstream away from the bridge support piers and reduce local scour at the extension noses by using narrow sloping noses that direct the flow upward.

Researchers at Argonne’s computing center performed a full-scale, three-dimensional computational fluid dynamics study of the pier extensions and design alternatives for the guide walls to find an optimal configuration for various flood flow conditions. The results of the study indicated that a pier extension length of 100 feet (30.5 meters) is sufficient to protect the piers. If the design length had not been optimized using computational fluid dynamics, the final design would likely have included safe but much more costly 200-foot (61-meter) extensions.


This artist rendering shows the pier extension concept for the California bridge after long-term erosion. The current riverbed and flood plain elevations will result in most of the left-most extensions being buried at completion of the project.


The recommended orientation was about 11 degrees west of the centerline of the channel to better guide water coming from the west flood plain into the bridge approach flow during floods. Analysis showed that an initial design for the west guide wall, based on the guidelines provided in the Hydraulic Engineering Circular numbers 20 and 23, would not guide high water moving off the west flood plain smoothly back into the channel as it approached the bridge, and that would increase erosion risk at the piers.

Using computational fluid dynamics analysis, the Argonne researchers developed a longer and more curved guide wall conforming to the local topology for the west abutment. They determined that the longer wall would perform much better, so it became part of the final design. The U.S. Army Engineer Research and Development Center in Vicksburg, MS, tested the design with a one-thirtieth scale physical model. Construction of the project is scheduled to start in fall 2016. The project currently is in the design phase, which will be completed in summer 2016.

Complete details can be found in the Argonne report titled Three Dimensional Analysis of Pier Extension and Guide Wall Design Alternatives to Mitigate Local Scour Risk at the BNSF Railroad Bridge Downstream of the Prado Dam (ANL/ESD-15/7), available from the U.S. Department of Energy’s Office of Scientific and Technical Information at


The U.S. Army Engineer Research and Development Center in Vicksburg, MS, installed these optimized triangular pier extensions at one-thirtieth scale for testing after a computational fluid dynamics analysis tweaked the design for the California bridge.


Sizing Riprap Installations

The technical procedures for sizing of riprap for scour countermeasures is based mostly on limited field observations and scaled laboratory tests under ideal controlled conditions. The actual size of riprap required for many field applications is too large for testing in the laboratory. As a consequence, significant uncertainty remains in the formulas for sizing riprap.

The problem: How do you determine riprap size for major projects or assess a riprap installation when many interrelated factors such as local riverbed bathymetry, pier orientation in the river, and the distribution of flood flow velocities are all important? Simulations on high-performance computer clusters can now account for all of the interrelated flow physics and geometry to determine how big the riprap needs to be and assess the capability of an arrangement to mitigate scour risk.


Shown is computational fluid dynamics modeling of the proposed pier extension to reduce scour at the BNSF Railroad Bridge over the Santa Ana River. Red shows the high shear stress that will result in riverbed erosion. The area at the pier with extensions has lower shear stress, which will reduce erosion during a major flood.


A project using advanced computing is determining the size and placement of riprap needed to protect this pier at the Middle Fork Feather River Bridge in Plumas County, CA.


Shown here is a multibeam sonar scan of an existing installation of a riprap countermeasure at the Middle Fork Feather River Bridge in Plumas County, CA. The sonar scan was conducted to study the effectiveness of the riprap installation.


This computational fluid dynamics and structural mechanics modeling shows the existing installation of the riprap countermeasure at the Middle Fork Feather River Bridge.


The FHWA and Argonne TRACC researchers developed an alternative computational technique that couples structural mechanics software with fluid dynamics software to anticipate incipient motion of large rocks in river environments. The coupling method enables automated calculations of the large displacements of rocks that occur because of the constantly changing fluid forces. The software facilitates precise calculations of pressure distributions on the surface of the rocks in the riprap installations, while predicting the resulting motion of rocks because of those forces, as well as collisions between the individual rocks.

Argonne’s high-performance computers are able to handle geometries of large stretches of rivers with highly detailed representations of bridge piers and dozens of movable rock riprap installations. The FHWA and Argonne TRACC researchers applied the results to verify the size and placement of riprap that had been installed to protect piers at the Middle Fork Feather River Bridge located on State Route 89 in Plumas County, CA, near the towns of Blairsden and Graeagle, close to the intersection with highway 70. The bridge was constructed in 1955 and is approximately 223 feet (68 meters) long and 32 feet (10 meters) wide. The researchers determined threshold water velocities for riprap motion to address possible extreme weather events. In future, the procedure is planned for use at other bridges.

Hydraulic Capacity of ADA-Compliant Grates

When making improvements to sidewalks and crosswalks in urban areas, the Minnesota Department of Transportation (MnDOT) needed to use new grate styles to comply with the Americans with Disabilities Act of 1990. ADA-safe grates are required wherever catch basins are located in a pedestrian access route. The ADA accessibility guidelines require that the openings in those grates prevent a 0.5-inch (1.3-centimeter) sphere from passing through. The grates are mainly used on retrofit projects where relocating the drainage structures is too costly.

The problem: The manufacturers of ADA-compliant grates, however, were unable to provide information on their hydraulic capacities. To quantify those capacities, researchers traditionally would run physical model testing, varying the flows and slope in a hydraulics laboratory. MnDOT opted for a less expensive and faster approach by modeling the grates at full scale using computational fluid dynamics on Argonne’s high-performance computer clusters.

“MnDOT uses ADA-safe inlet grates for some inlets,” says Lisa Sayler, P.E., assistant State hydraulics engineer with the MnDOT Bridge Office. “These grates have smaller openings, and presumably lower hydraulic capacity, than the current standard MnDOT grates, but the exact capacity is unknown for ongrade locations. The computational fluid dynamics analysis provided a quick and cost-effective way to analyze the grate capacity. The results produced by the analysis ensure that the retrofit project does not reduce the overall drainage capacity.”

The researchers performed a computational fluid dynamics study of an ADA-compliant grate compared to a noncompliant vane grate. The study included 21 cases with varying street, cross street, and gutter slopes for each grate. The research team built the geometry for the simulations at full scale, an approach that could not be easily accomplished in a laboratory with a flume.

Hydraulic performance of the ADA-compliant grate was below that of the vane grate for all but the lowest volume flow rates of rainwater drainage, and the performance deficit rapidly grew larger as the flow rate increased. The narrow slots of an ADA-compliant grate, limited to a width of 0.5 inch (1.3 centimeters), create more resistance to flow through the grate than those of noncompliant grates with wider slots. Grate hydraulic performance correlated well with the upstream Reynolds number of the approach flow. To handle runoff, MnDOT is using a combination of traditional grates with larger slots and ADA-compliant grates.

Complete details are available in the Argonne report Hydraulic Capacity of an ADA Compliant Street Drain Grate (ANL/ESD-15/25), available from the U.S. Department of Energy’s Office of Scientific and Technical Information at


The Minnesota Department of Transportation used computational fluid dynamics to analyze the hydraulic capacity of ADA-compliant grates such as this one.


This computational fluid dynamics modeling of an ADA-compliant grate shows the water surface with grate not capturing all of the water that flows directly over it.


Measuring Erosion In Stream Beds

The problem: Historically, foundation scour has been a leading cause of bridge failures.FHWA and others have investigated scour and developed equations to predict the potential to scour.However, the existing equations, based on hydraulic modeling in sand, tend to overpredict scour.This conservative overprediction can result in overly deep and expensive foundations or reliance on unneeded countermeasures to prevent the scour, especially in cohesive soils.

A device incorporated into standard geotechnical drilling equipment to measure the erosion resistance of sediment material also is used to measure continuously the erosion resistance of fine-grained subsurface cohesive soils in situ.This measurement strategy is required because of the extreme difficulty associated with accurately characterizing the erosion resistance of such soils. This device is part of a new risk-based scour design methodology that FHWA’s Hydraulics Research Program is currently developing. The research to develop this device and to make it market ready for the Every Day Counts initiative was one of the agency’s strategic objectives for 2014.


These FHWA researchers are conducting field tests of an in situ scour testing device.


This computational fluid dynamics model shows the erosion head that is used for the in situ scour testing device, with the head contour optimized to maximize its efficiency by minimizing pressure drop in the head.


The FHWA hydraulics team designed and manufactured an erosion head for the in situ scour testing device to fit inside standard geotechnical sampling tubes. Sonar sensors monitor the erosion rate and maintain a required “working gap” at the soil-head interface.The researchers used computational fluid dynamics modeling to optimize and streamline the shape of the erosion head and to determine the system’s capacity. The team also conducted several simulation runs using computational fluid dynamics to explore the flow limits of the erosion head and to fine-tune the flow requirements for in situ field tests.

Currently, the lab is conducting the first phase of the field tests. The second phase will start in the summer of 2016, and the researchers expect to complete the project with a market-ready device available in 2017.

Energy Dissipaters for Drainage on Steep Slopes

The problem: Energy dissipation for culverts discharging on slopes with a horizontal to vertical ratio of 2:1 (2H:1V) or greater is an ongoing issue. On steep slopes the discharge of collected water at a single pipe opening results in a high-speed stream that rapidly erodes the hillside.

The commonly used solution, riprap, works well for milder slopes but is unsuitable for steep ones. Not only is the riprap unstable by itself, but also it is difficult to install where access for heavy construction equipment is limited.

Culverts discharging on 2H:1V or steeper slopes are prone to failure because of the headcut at the outlets potentially migrating upstream. A headcut is the erosional process that occurs at abrupt vertical hydraulic drops. This erosional process takes place mostly because culverts channel the naturally occurring sheet flow--the film of rainwater that forms on roads and runs off to the roadside--and convert it into a point discharge, thus increasing velocities and shear stress. Protecting these outlets using traditional measures such as riprap is difficult because of the access issue.

FHWA and Argonne TRACC researchers are studying methods to dissipate energy at culvert outlets discharging flow onto steep slopes. The researchers are using computational fluid dynamics modeling to study various conceptual solutions and their effectiveness in dissipating energy. These concepts include an upward-inclined ramp and a slotted discharge pipe to redistribute the flow over a much larger area. The models have shown that these concepts will likely perform significantly better than current methods.

Currently, engineers with FHWA’s Eastern Federal Lands Highway Division are identifying potential field sites for a test installation of the two alternatives. Once FHWA identifies an appropriate site, the researchers will add digitized bathymetries of the topography to the computer model and will then conduct full-scale simulations. The results of those simulations will guide the field installation.

Reaping the Benefits

Shown here is the 3–D printed erosion head that FHWA used for the in situ scour testing device.


This computational fluid dynamics model shows an upward-inclined ramp concept with slots and guide vanes to spread the water flow exiting the red-colored culvert pipe onto a steep slope.


This computational fluid dynamics model of the water flow from a culvert discharging onto an upward-inclined ramp shows the water spreading to reduce erosion of the hillside.


Another alternative is a horizontal slotted discharge pipe. A computational fluid dynamics model shows water spreading out over the slope to reduce erosion.


These examples illustrate a few ways in which FHWA is using advanced computational analysis to achieve applied research goals and optimize project designs. The software capabilities needed to do these analyses were unavailable a decade ago, and even the more limited computational mechanics and 3–D design software that was available at that time was very expensive. Generally, it was used only for high-profile projects such as those at the National Aeronautics and Space Administration or where huge economies of scale could justify the cost, such as in the automotive and aircraft industries.

Today, FHWA and Argonne TRACC researchers have applied advanced computational methods tooptimize the design of a new scour countermeasure, determine riprap size and assess riprap configuration, determine the hydraulic capacity of an ADA-compliant street drain grate, streamline pier extensions, aid in the development of an in situ sediment erosion testing device, and investigate and optimize various concepts to dissipate the flow energy at drainage outlets on steep slopes.

“We expect that projects and applied research studies in the FHWA hydraulics research program will employ 3–D computational fluid mechanics in the future,” says FHWA Associate Administrator Trentacoste. “The hydraulics laboratory is integrating the use of advanced computational analysis on high-performance computer clusters into its operations and plans to use it on all projects that would benefit from it.”

Steven Lottes, Ph.D., leads the computational mechanics group at Argonne TRACC. He has more than 25 years of experience in the modeling and analysis of complex multiphase flow systems. He plans, coordinates, and conducts computational research on transportation applications at Argonne’s TRACC and provides technical support to the center’s user community.

Kornel Kerenyi, Ph.D., is the manager of FHWA’s Hydraulics Research Program in the Office of Infrastructure Research and Development. Kerenyi coordinates FHWA’s hydraulic and hydrology research activities with the National Hydraulic Team, State and local agencies, academia, and various other partners and customers. He also manages the J. Sterling Jones Hydraulics Laboratory at the Turner-Fairbank Highway Research Center in McLean, VA.

Cezary Bojanowski, Ph.D., is an expert in computational mechanics at Argonne TRACC. His work includes computational fluid dynamics modeling of free surface flows applied to transportation infrastructure, fluid structure interaction in scour countermeasures, response of bridges to extreme loadings, and computational analysis of crashworthiness and vehicle occupant safety.

For more information, contact Kornel Kerenyi at 202–493–3142 or, or Steven Lottes at 630–252–5290 or