Before FHWA's newest CARMA vehicles could join the fleet, they needed to be outfitted with the right equipment.

In 2019, the Federal Highway Administration added four CARMA-equipped passenger vehicles to its fleet of test vehicles. Before the vehicles were delivered to the Saxton Transportation Operations Laboratory at the Turner-Fairbank Highway Research Center, FHWA conducted verification and acceptance testing in Morton, IL. There, FHWA engineers and researchers customized the four vehicles with aftermarket, highly automated vehicle kits using open source software to achieve Society of Automotive Engineers (SAE) Automation Level 2. (For more information about the CARMA Platform, see "CARMA^SM: Driving Innovation" in the Winter 2020 issue of PUBLIC ROADS.)

For the first time, the U.S. Department of Transportation tested SAE Automation Level 2 vehicles, and this advanced the process of achieving cooperative driving automation (CDA) used in support of transportation systems management and operations research. The Volpe National Transportation Systems Center provided validation for FHWA's automated driving systems (ADS) verification and acceptance test plan. Because of specific feature implementation required for each model, testing ADS at this level posed many challenges, such as verifying the vehicle calibrations involved with the different sensors that impact ADS performance and ensuring the fail-safe mechanisms are properly tuned with the different types of by-wire controllers.

Selecting the Vehicles

Prior to purchasing two sport utility vehicles (SUVs), one minivan, and one midsize sedan to add to the fleet of CARMA-equipped vehicles, FHWA considered a variety of passenger vehicles. FHWA performed two rounds of a trade study aimed at choosing the most appropriate vehicle type and model to implement SAE Level 2 capabilities and to extend to higher SAE levels. The research team evaluated 16 vehicle models on 15 different attributes.

Researchers weighted four comparison criteria the most heavily. The first was the cost of instrumentation, including any reverse engineering and design work. Second was the FHWA team's knowledge and ability to access the vehicle's controller area network (CAN) messages, which provide critical information about the vehicle's state at any given moment. Researchers also heavily considered each vehicle's existing potential for by-wire integration and customization (the ability to control the vehicle through electronic signals vs. mechanical motions). The final weighted criterion was passenger comfort, including capacity.

"We recognize the importance of interoperability and consistency as major aspects for the future of CARMA research and usability," says Taylor Lochrane, the CARMA technical program manager. "FHWA selected the four passenger vehicles to accommodate the CARMA Platform and conduct research on different types of controllers and across diverse vehicle types, makes, and models."

FHWA selected industry leaders based on their expertise building and supplying platforms and components used within ADS.

The Women Leading the CARMA Collaborative

FHWA initiated the CARMA Collaborative to engage with stakeholders and collectively support the future of automated vehicles and connected infrastructure in the transportation industry to improve safety, efficiency, and mobility. The CARMA Collaborative is a largely female-driven initiative. In the past 2 years, the team:

• Established FHWA's CARMA branding and promoted the CARMA engineers' efforts, which enabled the development and public release of the CARMA Platform.
• Created a robust array of CARMA content and collateral to inspire stakeholder awareness and engagement.
• Collaborated with academic, industry, public agency, and Federal stakeholders to advance transportation with CDA.
• Cultivated relationships at conferences supported with innovative exhibits, including the Transportation Research Board Annual Meeting 2019 and 2020, South by Southwest 2019, and the Automated Vehicle Symposium 2019.

Overall, the women leading the CARMA Collaborative enable conversations and foster relationships across stakeholders to advance transportation safety, efficiency, and mobility.

Adding the Automated Equipment

For enhanced CARMA research and development, FHWA installed several aftermarket hardware devices to provide additional radar data collection, improved detection and navigation, and vehicle-to-everything communication.

On the exterior of the vehicle, high-resolution three-dimensional light detection and ranging (LiDAR) provides a 360-degree surround view along with real-time data for navigation and object detection. A multimode, electronically scanning radar combines a wide field of view at midrange with long-range coverage for object detection. Four rear and side detection radar sensors provide awareness of approaching vehicles as well as support for blind spot detection applications. For enhanced positioning solutions, two global navigation satellite system (GNSS) antennas track the maximum number of satellites in any environment. Additionally, vehicle-to-everything communication is managed with an onboard unit with two dedicated short-range communications antennas. Lastly, global navigation, cellular, and Wi-Fi antennas were installed on the exterior of the vehicle.

Inside the vehicle, FHWA installed multiple cameras: one camera for collision detection and lane marking identification, and two cameras for object and traffic light detection. Light bars are located in the front and back vehicle windows. In the middle console, each vehicle is equipped with a push-button shifter, a tablet, and a red emergency stop button to disable automated controls.

FHWA outfitted each vehicle with a different by-wire control kit that provides electronic control of the vehicle's brake, throttle, steering, and shifting to enable testing for automated vehicle applications. The team incorporated LED lighting on the vehicles' floors for immediate visual indicators: green shows manual driving mode, blue signifies by-wire mode, and red denotes a fault or failure of the by-wire system.

The team installed speed and steering control (SSC) software packages in the Linux-based computer to provide a common interface to tune the speed and steering actions of the by-wire controllers. Using a robot operating system (ROS), the SSC software enables research applications to send messages to this package with desired vehicle path curvature, maximum curvature, desired speed, acceleration limits, deceleration limits, and desired gear. On top of the SSC interface, Autoware,^TM an open-source software, provides the necessary automated driving functionality for localization, motion planning, and object detection and avoidance. With by-wire, SSC, and Autoware installed, the vehicle is able to perform waypoint following. Using a prerecorded set of waypoints (geographic coordinates that define a route of travel), desired speed, and a high-definition map of the test facility, the vehicle was able to pinpoint its location on the map and drive itself using lateral and longitudinal controls.

These integrated aftermarket subsystems serve to develop the next phases of the CARMA Platform enabling cooperative driving at SAE Automation Levels 2 and 3 for the benefit of transportation systems management and operations research.

"The goal of CARMA is to accelerate understanding of the benefits of CDA through testing," says FHWA Administrator Nicole R. Nason. "We want automated vehicles to work together, and increasing the SAE levels of automation available on research vehicles is an important next step to advancing transportation safety, efficiency, and mobility."

Developing the Acceptance Test Plan

FHWA developed a detailed test plan to verify that each vehicle's hardware, software, safety requirements, and design specifications were met prior to acceptance and delivery. The document included the scope, approach, test facility or environment, schedule, roles and responsibilities, and detailed test cases.

Researchers started with a template of test case scenarios and objectives. The collection, review, and discussion of the vehicle requirements, design specification documents, and original equipment manufacturers' manuals for the installed components were essential to gain a deeper understanding of the features and functionality of each component. FHWA collaborated with Volpe and subcontractors to refine the scope and develop detailed test case scenarios and procedures.

Hardware testing involved the inventory and inspection of each hardware unit supplied by the contractor, including the strength of hardware mounting and its proper location. The team measured the location of each aftermarket sensor to ensure an optimal position when compared to the manufacturer's standards. Testing also included verifying that the vehicle had sufficient power distribution for all equipment, inside and out.

Safety testing confirmed that customizations did not impact the vehicles' stock safety features and that the by-wire fail-safe mechanisms were working properly. With system safety and risk minimization as the primary objectives, the team reviewed two National Highway Traffic Safety Administration publications during the development of the test plan and procedures: A Framework for Automated Driving System Testable Cases and Scenarios (DOT HS 812 623) and Automated Driving Systems 2.0 — A Vision for Safety (DOT HS 812 442). To ensure safe operations, researchers incorporated the operational design domain and object and event detection and response into the description of each test case.

Finally, to enable testing the new passenger vehicles at higher automation levels and assisting with future operational and safety design requirements, the team added elements of SAE Levels 3 and 4.

FHWA created functional test cases to evaluate the basic functions and operations of the installed aftermarket hardware and software. These test cases also verified the range, tracking, and calibrations of all the sensors and cameras and assessed the by-wire and SSC capabilities from low to high speed. Functional test cases also examined Autoware localization, waypoint following, camera-LiDAR calibration, and object detection.

Test plan development required multiple reviews and iterations, and the plan created is continually updated to capture evolving requirements during the Agile development lifecycle of a research and development program. Additionally, the test cases and procedures are constantly refined based on the outcomes and lessons learned from each verification test. The test plans are available at https://usdot-carma.atlassian.net/wiki/spaces/CAR/overview.

Introducing CARMA Support Services

CARMA Support Services kicked off in late 2019 to provide support services for the implementation of the CARMA product suite. CARMA Support Services established a CARMA Help Desk and email to respond to common issues from CARMA users, researchers, and deployers while also monitoring user feedback for future improvements.

Executing the Acceptance Testing

Prior to the test event, a readiness review examined the contractor's preliminary test results, the logistics of the local test facilities, the test priorities, and the coordination of the personnel and equipment. In April 2019, testing began with the two CARMA-equipped SUVs. A second iteration of testing began in June 2019 with the minivan and midsize sedan. Each of these test events spanned a week. During the first day, classroom and hands-on training of the by-wire controller, SSC, and Autoware prepared the team for subsequent tests. Before beginning the more complicated testing of the by-wire and ADS features, the team completed important hardware and safety inspections. At the end of each day, the team discussed and documented the issues to be addressed the following day based on priority and severity, including whether the test cases would be executed sequentially or in parallel. Toward the end of each week, the team prioritized addressing test cases and high severity issues for vehicle acceptance and delivery.

Lessons Learned from Testing

The incremental testing approach minimized risk from scheduling and testing resource constraints. The range of requirements tested necessitated a plan for varied staff support. The team needed a diverse background of mechanical, electrical, software, and robotics engineers combined with the proper equipment to ensure that questions and concerns could be quickly resolved.

Safety is always the number one priority, and the engineers received driver accident avoidance training in addition to automated vehicle operation training prior to operating the newly equipped vehicles. Safety drivers operated the vehicles to test the ADS features alongside test engineers who understood the automated operations and capabilities.

The team extensively tested the fail-safe mechanisms. Because of the different types of controllers and the implementation of the by-wire system, the performance and operation varied by vehicle. Each system had four types of fail-safe or override: throttle, brake, steering, and an emergency stop button. The team found that calibrations across by-wire systems differed due to operational variances within fail-safe mechanisms and override procedures to take manual control of the vehicle at higher speeds.

"In the end," says Lochrane, "the team verified that each system was properly tuned to provide sufficient results: the safety driver could easily and naturally use any of these mechanisms to immediately take full control of the vehicle in order to ensure its safe travel at all times."

In addition, confirming the proper calibration of the LiDAR and the proper configuration of Autoware was very important. Remembering to collect ROS bags prior to each test case was significant for simulating and troubleshooting issues that occurred.

Moving Forward

Including the newest four vehicles, the initial CARMA-equipped passenger fleet has set the foundation to advance the FHWA team supporting the CARMA Program to higher SAE automation levels with extensive safety considerations. The next test fleet will include four heavy trucks. Class 8 vehicle dynamics will challenge the team to find a proper test facility as well as to create new test cases for both tractor and trailer that address complications during sensor calibration while testing under SAE Automation Levels 1 and 2.

"CARMA aligns with the Department of Transportation's approach to multimodal automation," says FHWA Deputy Administrator Mala K. Parker, "and this program enables transportation innovation and safety with automated technology."

Deborah Curtis is a research transportation specialist at FHWA. She has more than 28 years of experience leading projects related to traffic signal systems, intelligent transportation systems, and, most recently, cooperative automation. She has a B.S. in civil engineering from West Virginia University.

Mae Fromm is the senior software engineer for the Saxton Transportation Operations Laboratory. She has a B.S. in decision sciences and management information systems with a minor in information technology from George Mason University. She has 17 years of software development experience and currently supports CARMA software development and testing activities.

Laura Dailey is the communications manager for the Saxton Transportation Operations Laboratory, overseeing marketing and engagement activities. She earned an M.S. from Drexel University and a B.S. with a marketing concentration from Elon University.

For more information, contact Deborah Curtis at Deborah.Curtis@dot.gov or visit https://highways.dot.gov/research/operations/Cooperative-Driving-Automation.