University Conducting HSIP Project Evaluations Using Empirical Bayes

Original publication: HSIP Noteworthy Practice Series, HSIP Project Evaluation; FHWA-SA-11-02; 2011(PDF, 2.3MB)

Wisconsin DOT contracted with the University of Wisconsin Traffic Operations and Safety (TOPS) Laboratory to investigate multiple project evaluation methods through a research grant for HSIP evaluation support. Initial research efforts included project evaluations based on before and after collision maps using the software Intersection Magic and before-after evaluations using benefit-cost analysis. From the beginning of the research, the intent was to use Empirical Bayes (EB) analysis in the project evaluations, but Wisconsin did not have safety performance functions (SPFs), which are required for the EB method. However, once the State acquired the SafetyAnalyst software, the TOPS Laboratory was able to incorporate the EB method into the project evaluations by using the SPFs contained in SafetyAnalyst. The SPFs in SafetyAnalyst were developed using national data and are intended to be calibrated to local conditions. While it was not possible to calibrate the SPFs to Wisconsin conditions due to lack of data, the TOPS Laboratory uses the SPFs to provide a comparison of performance in Wisconsin to that of the nation.

The TOPS Laboratory developed a process to extract the appropriate crashes (by location, type, and year) from the Wisconsin crash database based on the project location and scheduled start and completion dates for evaluation purposes. HSIP projects are evaluated based on five years of before data and three years of after data. Fatal and injury crashes are the focus of the evaluation, but the analysis also considers target crash types based on the nature of the improvement.

The TOPS Laboratory conducts a benefit-cost analysis based on results of both a simple before-after evaluation and an EB analysis to evaluate the projects from an economic perspective. This provides a simple comparison of the results of the two evaluation methods (In the table shown, “S. No.” refers to the site number for the project evaluated, and the “FOS (financial operating system) ID” is used by Wisconsin DOT as the specific project identifier.) and demonstrates how a simple before-after evaluation can overestimate the safety benefits.

Table 1. Benefit-Cost Analysis

Key Accomplishments

• Developed a project evaluation process incorporating Empirical Bayes analysis into all HSIP project evaluations.
• Demonstrated the importance of using statistical evaluations to reduce the overestimation of safety benefits due to regression-to-the-mean bias.

Results

Originally, engineers in Wisconsin were reluctant to use EB. However, with the assistance of the TOPS Laboratory, the Wisconsin DOT was able to successfully implement a project evaluation process incorporating EB analysis and to receive buy-in at the regional level. The TOPS Laboratory demonstrated the importance of statistical EB techniques in project evaluations through a comparison benefit-cost analysis using simple before and after results to before and after using EB. The results demonstrate the EB analysis reduces the overestimation of safety benefits due to regression-to-the-mean bias.

Contact

Andrea Bill
Traffic Safety Engineering Research Program Manager
Traffic Operations and Safety (TOPS) Laboratory
University of Wisconsin
608-890-3425
bill@wisc.edu

Project Evaluation Using Empirical Bayes

Original publication: HSIP Noteworthy Practice Series, HSIP Project Evaluation; FHWA-SA-11-02; 2011(PDF 2.3MB)

Incorporating the Empirical Bayes (EB) method into project evaluations reduces the potential overestimation of safety benefits due to regression-to-the-mean. While the EB method is not difficult in itself, it requires safety performance functions (SPF) for the type of facilities on which projects are being evaluated.

SPFs were originally developed by Colorado Department of Transportation (CDOT) for use in the network screening process. While crash rates are commonly used to measure safety, the crash rate implies a linear relationship between safety and exposure, which can often be misleading since rates change with Annual Average Daily Traffic (AADT). To capture how this rate change takes place, design engineers at the CDOT started to calibrate SPFs in the late 1990s, as part of the development of the Level of Service of Safety (LOSS) concept. LOSS is used to identify locations with potential for safety improvement and reflects how a roadway segment is performing in regard to its expected crash frequency and severity at a specific level of AADT, based on the SPF. By 2001, CDOT had calibrated SPFs for all public roadways (state and local) in Colorado, stratified by the number of lanes, terrain, environment, and functional classification. In 2009, CDOT in collaboration with consultants developed SPFs for all intersection types.

The development of SPFs has not only advanced CDOT’s network screening process, it also has enabled CDOT to institutionalize the use of the EB method as a standard procedure for safety evaluation analysis. Colorado has traditionally used a simple spreadsheet with three to five years of before and after data to conduct project evaluations. CDOT is currently working on applying an EB correction to evaluate sites on an SPF graph as shown. The use of the EB method is particularly effective when it takes a long time for a few crashes to occur, as is often the case on Colorado rural roads.

Key Accomplishments

• Developed SPFs for all roadway facility and intersection types in the state.
• Institutionalized the use of the Empirical Bayes method as a standard procedure for safety evaluation analysis to reduce effects of regression-to-the-mean.

Results

CDOT developed SPFs for all state and local roadway facilities and intersection types. The development of the SPFs has enabled CDOT to fully institutionalize the EB method for all safety analysis at CDOT and reduce the effects of regression-to-the-mean.

Contact

Bryan Allery
Colorado Department of Transportation
303-757-9967
bryan.allery@dot.state.co.us

Jake Kononov
Colorado Department of Transportation
303-757-9973
jake.kononov@dot.state.co.us

Safety Warrants and Spot Safety Index

Original publication: HSIP Noteworthy Practice Series, HSIP Project Identification; FHWA-SA-11-02; 2011(PDF, 2.7MB)

The North Carolina Department of Transportation (NCDOT) started to identify shortcomings in its problem identification method in the mid-1990s. The previous method focused on identifying locations with a potential safety issue based on factors such as crash frequency, crash rate, and crash severity. In many cases, the locations identified did not exhibit a correctable crash type and were congestion related issues. For example, NCDOT repeatedly identified signalized intersections exhibiting a high frequency of rear-end collisions, but attributed the collisions to congestion and driver inattention rather than a roadway factor.

Intersection Warrants:

I-1: Frontal Impact

I-2: Last Year Increase

I-3: Frequency with Severity Index Min

I-4: Night Location without Streetlight

I-5: Chronic Pattern

Section Warrants:

S-1: Run Off Road- Wet Conditions

S-2: Run Off Road

S-3: Wet Road Conditions

S-4: Non-Intersection Night Location without Streetlight

Bridge Warrant:

B-1: Bridge

Bike/Ped Intersection Warrants:

P-1: Last 3 Years (pedestrians)

P-2: Darkness with Streetlights

P-3: Alcohol Involvement

P-4: Chronic Location

X-1: Last 3 years (bicyclists)

X-2: Darkness with Streetlights

X-3: Alcohol Involvement

X-4: Chronic Location

Beginning with the 1996 HSIP, a set of safety warrants was established for intersections and roadway segments to target locations exhibiting a pattern of correctable crash types or conditions, as well as locations with a significant increase in crash frequency during the past calendar year. NCDOT has continued to expand and modify the safety warrants throughout the years to improve the identification process.

NCDOT initially screens the network (including local roads) for potential safety improvement locations using four categories of safety warrants: intersections, sections, bridges, and bicycle and pedestrian intersections. The safety warrants are analyzed annually using 5 to 10 years of crash data by querying the crash database. The current warrant criteria are based on crash frequency, severity, conditions, and percentage of target crashes. When a location meets the warrant criteria, it is flagged. As an example, an interstate segment would be flagged based on run-off road crashes if a minimum of 30 total crashes occurred on the segment, the crash rate is greater than 60 crashes per mile, and a minimum of 60 percent of the total crashes were run off the road. After a location is flagged, a weighting factor is calculated based on the warrant criteria. The weighting factors are summed for locations meeting multiple warrants and are used to rank locations to determine which will receive priority for further analysis and investigation by the corresponding Regional Traffic Engineering and Highway Division staff. The Regional Traffic Engineers are responsible for identifying potential countermeasures and developing projects.

All safety projects are submitted to North Carolina’s Safety Oversight Committee, which was established to help select projects to receive Spot Safety Program funding. To provide clear and consistent data-driven selection process, the Spot Safety Index (SSI) was developed as a decision support tool to perform an initial prioritization of all candidate projects from across the state. It ensures safety investments are focused on locations with the greatest need and potential for improvement. The SSI is calculated based on a 100-point scale and is composed of four parts: Safety Factor (60 points), Constructability (5 points – e.g., ROW acquisition needs), Department Goals (5 points) and Division/Region Priority (30 points). The Safety Factor is based on the benefit-cost ratio, Severity Index, and whether the project is identified in the HSIP List or identified through a Road Safety Audit (RSA). An initial list of prioritized projects is developed by ranking projects based on the SSI. However, the Committee must take other considerations into account to develop the final list, including distribution of funding to the 14 districts and the effectiveness of countermeasures identified in the projects based on results from the state’s evaluation group.

Key Accomplishments

• Developed network screening method to identify locations with severe and correctable crash patterns.
• Continued to update network screening process to improve the identification of relevant safety issues and locations.
• Developed systematic project prioritization ranking method that considers benefit-cost analysis, departmental and regional priorities, and ease of constructability.

Results

The development of the safety warrants for use in the network screening process has enabled NCDOT to focus their analysis on the identification of locations with severe crashes and crash patterns correctable by infrastructure safety countermeasures. NCDOT also has successfully established a clear and consistent data-driven process for selecting and prioritizing projects for funding.

Contact

Stephen Lowry
Safety Improvement Engineer
North Carolina Department of Transportation
919-773-2892
slowry@ncdot.gov

Level of Service of Safety and Diagnostic Analysis

Original publication: HSIP Noteworthy Practice Series, HSIP Project Identification; FHWA-SA-11-02; 2011(PDF, 2.7MB)

The Colorado Department of Transportation (CDOT) uses two methods for identifying locations with potential for safety improvement: Level of Service of Safety (LOSS) and Diagnostic Analysis. LOSS is based on the concept of Safety Performance Functions (SPF), while Diagnostic Analysis is developed around the idea of statistical pattern recognition.

Design engineers at CDOT pioneered development of the LOSS concept to quantify the magnitude of the safety problem. A crash rate implies a linear relationship between safety and exposure. While crash rates are commonly used to measure safety, they are often misleading since rates change with Annual Average Daily Traffic (AADT). To capture how this rate change takes place, CDOT engineers began calibrating SPFs in the late 1990s based on the work of Dr. Ezra Hauer. By 2001, CDOT had calibrated SPFs for all public roadways in Colorado, which were stratified by the number of lanes, terrain, environment, and functional classification. In 2009, in collaboration with consultants, CDOT developed SPFs for all intersection types.

Development of SPFs supports the conceptual formulation of the LOSS concept. It uses qualitative measures to characterize the safety of a roadway segment in reference to its expected performance. If the number of crashes predicted by the SPF represents normal or expected crash frequency at a specific level of AADT, then the degree of deviation from the norm can be stratified to represent specific levels of safety. To describe road safety from the frequency and severity standpoint, two different SPFs were calibrated: one for the total number of crashes and the other for injury and fatal crashes. When the magnitude of the safety problem is assessed, it is described from the frequency and severity standpoint. The figure (Kononov and Allery, 2003) illustrates the LOSS concept using an SPF calibrated for total crashes expected on the 6-lane urban freeways. The delineated boundary line is located 1.5 standard deviations from the mean, reflecting a Negative Binomial error structure. Four LOSS categories were introduced:

• LOSS-I - Indicates low potential for crash reduction;
• LOSS-II - Indicates low to moderate potential for crash reduction;
• LOSS-III - Indicates moderate to high potential for crash reduction; and
• LOSS-IV - Indicates high potential for crash reduction.

LOSS reflects how the roadway segment is performing in regard to its expected crash frequency and severity at a specific level of AADT. However, it only describes the magnitude of the safety problem; it does not provide any information related to the nature of the problem itself. To investigate the nature of the problem, Colorado uses Direct Diagnostics and Pattern Recognition techniques.

A comprehensive methodology was developed to conduct diagnostic analyses of safety problems. The Direct Diagnostics and Pattern Recognition methods calculate a cumulative binomial probability of the crash types and related characteristics to identify overrepresented elements in the crash data (e.g., dark conditions, overturning vehicles) that may be related to abnormal crash patterns and crash causation. Direct Diagnostics is used for intersection analysis, and Pattern Recognition is used for roadway segments.

CDOT initially used the combination of LOSS and Direct Diagnostics and Pattern Recognition to identify sites with potential for safety improvement only on safety motivated projects. Beginning in 2001, they are applied to all projects at CDOT, including resurfacing, reconstruction, realignment, widening, Environmental Assessments (EA) and Environmental Impact Statements (EIS). CDOT conducts a statewide analysis using Direct Diagnostics and Pattern Recognition and recalibrates SPFs about every five years.

Key Accomplishments

• Calibrated SPFs for all highways.
• Developed sophisticated predictive and diagnostic tools to maximize crash reduction in the state within budget constraints.
• Institutionalized use of these tools throughout the state of Colorado.
• Achieved unprecedented fatal crash reduction of 36 percent over the seven years of sustained application of these advanced methods on all infrastructure and behavioral projects at CDOT.
• Provided substantive conceptual and analytical input for the development of the Highway Safety Manual (HSM).

Results

CDOT developed sophisticated predictive and diagnostic tools to maximize potential crash reduction in the state within constraints of available budgets and institutionalized use of these tools throughout the state of Colorado. Over the seven years of application of the advanced methods on all infrastructure and behavioral projects at CDOT, the state has achieved an unprecedented fatal crash reduction of 36 percent, without reduction in travel or increase in safety expenditures. Additionally, these efforts provided substantive analytical and conceptual input for development of the Highway Safety Manual.

Contact

Bryan Allery
Colorado Department of Transportation
303-757-9967
bryan.allery@dot.state.co.us

Jake Kononov
Colorado Department of Transportation
303-757-9973
jake.kononov@dot.state.co.us

Illinois Develops SPFs for All State Routes and Intersections

Original publication: HSIP Noteworthy Practice Series, HSIP Project Identification; FHWA-SA-11-02; 2011(PDF, 2.7MB)

While the development of SafetyAnalyst and the Highway Safety Manual was still underway, Illinois decided to incorporate a new analysis technique to assist the state in moving forward with the implementation of SafetyAnalyst. Within a year, the Illinois Department of Transportation (IDOT), with the assistance of the University of Illinois, developed safety performance functions (SPF) for state routes and intersections throughout the state using the Empirical Bayes (EB) method. The SPFs have been used in the HSIP network screening process since 2008 to identify potential locations for safety improvement projects.

SPF equations were developed for 12 peer groups of roadway segments (e.g., rural two-lane highway, rural multilane undivided highway, rural multilane divided highway, etc.) and eight peer groups for intersections (e.g., rural minor leg stop control, rural all-way stop control, rural signalized, etc.). The SPFs are used in the network screening process to calculate a Potential for Safety Improvement (PSI) for all locations. The PSI is the difference between the corrected crash frequency (calculated using the EB method) and the expected crash experience (based on the SPF) for a given traffic volume within the peer group.

Since the focus of the HSIP is to reduce fatalities and serious injuries, the PSI calculation is weighted to emphasize the most severe crashes. The weighted PSI calculations are then ranked in ascending order by location and peer group to identify locations with the greatest safety need or highest PSI value. Once the sites with the greatest potential for safety improvement are identified, the IDOT Districts review the locations and make recommendations for improvement. Candidate HSIP projects on the state roadway system are selected by the District’s Safety Committee and submitted to the Bureau of Safety Engineering.

When the SPFs were originally developed, there was not enough data to develop SPFs for the local roadway system. Illinois has been expanding the crash database for local roadways and, in the near future, the state will begin discussions about the development of SPFs for local roadways, as well as updating the existing SPFs for state roadways. Currently, local roadways are evaluated using an aggregate level analysis to identify potential safety issues (e.g., counties with overrepresentation of a particular crash type, crash severity, behavioral issue, etc.). Local agencies can submit safety improvement projects to the State Safety Committee for funding consideration through the Local Road Program component of the HSIP.

Key Accomplishments

• Developed SPFs for state routes and intersections throughout the state.
• Expanded knowledge and acceptance of analysis techniques.
• Provided data-driven safety decision making tools.

Results

Incorporating SPFs into the network screening process for safety improvement projects has led to several positive outcomes. Although other factors may involved, Illinois has seen a significant reduction in fatalities. In 2009, Illinois had the lowest number of fatalities since 1921. Transportation professionals are embracing the analysis results and making data-driven safety decisions. Using SPFs has helped shift the focus of the state’s program away from the urban, densely populated areas and provided a broader focus for safety projects, including low-cost safety improvements or systemic improvements that may not have been identified using previous analysis methods. Engineers throughout the state have become more familiar and comfortable with the use of SPFs through the state’s efforts, leading to a greater acceptance of SPFs and appreciation for improved quantitative data.

Contact

Roseanne Nance
Illinois Department of Transportation
217-785-5875
nancer@dot.il.gov