Evaluation is critical to determine if a project or group of projects is achieving the desired results and to ensure investments are cost-effective. Evaluation provides a quantitative estimate of the effects on safety, which is valuable information for future planning. Evaluation results enable a state to determine if appropriate countermeasures were used at particular locations, whether any adverse impacts occurred, if corrective action is necessary, and how effective those countermeasures would be for similar sites in the future.

Various methods exist for evaluating projects, but any evaluation should consider a minimum of three to five years of before and after crash data, the target crash type of the improvement, and crash severity (a countermeasure may increase the total number of crashes, but reduce the crash severity). Ideally, project evaluation should incorporate more advanced techniques (e.g., safety performance functions (SPFs), Empirical Bayes (EB) method) to account for natural fluctuations in crashes from year to year and other changes potentially impacting evaluation results.

The majority of states are conducting project evaluations based on a simple before-after analysis, and a few are using evaluation results to develop state-specific crash modification factors (CMFs) for various countermeasures. While simple before-after evaluations are rather easy to perform and may provide a basic understanding of safety changes, they assume any change was due solely to the safety improvement at the site and may misrepresent the true effectiveness of a project due to the effects of regression-to-the-mean. (Regression-to-the-mean bias describes a situation in which crash rates are artificially high (or low) during the before period and would have decreased (or increased) even without an improvement to the site.) The EB method can be incorporated into project evaluations to reduce the effects of regression-to-the-mean. However, very few states have been able to use the EB method since it requires calibrated SPFs. Many states do not have the training, resources, tools, manpower, or necessary data to calibrate SPFs.

Another challenge is that individual states may not have enough installations of a particular countermeasure to develop quality CMFs. The Evaluations of Low Cost Safety Improvements Pooled Fund Study (ELCSI PFS) combines the implementation efforts of multiple states to develop reliable estimates of countermeasure effectiveness. States can independently initiate similar efforts.

Noteworthy Practices

The following cases demonstrate noteworthy practices four states are using in HSIP project evaluations:

• The Colorado DOT developed SPFs for all roadway facility and intersection types in the State, which enabled the DOT to institutionalize the EB method into all safety project evaluations and reduce the effects of regression-to-the-mean. (read more)
• The Florida DOT developed an on-line database application of safety improvement projects that automates the processes for conducting benefit-cost analysis to compare different countermeasures and for conducting safety project evaluations to develop crash reduction factors (CRF). The application has also enabled Florida to develop and continue to refine state-specific CRFs for several countermeasures based on the project evaluation results. (read more)
• The North Carolina DOT created a safety project evaluation group to conduct evaluations on all spot safety projects in the State. The project evaluations provided field engineers with valuable feedback on the effectiveness of safety projects and countermeasures. (read more)
• The University of Wisconsin Traffic Operations and Safety (TOPS) Laboratory, under contract to Wisconsin DOT, developed a project evaluation process incorporating EB analysis into all HSIP project evaluations. The TOPS Laboratory compared benefit-cost analysis using simple before-after analysis results and EB to demonstrate the importance of using statistical evaluations to reduce the overestimation of safety benefits due to regression-to-the-mean. (read more)

To access these full case studies, click on the individual links above or visit the FHWA Office of Safety on-line at: www.fhwa.dot.gov/safety/legislative-safety-programs/hsip