Introduction to Systemic Safety Program Evaluation
Systemic safety programs are relatively new and evolving in the United States, as is the practice of evaluating systemic safety program effectiveness. Evaluating the performance of countermeasures implemented in locations that have no recent crash history—but that exhibit other characteristics that indicate the potential for a severe crash—is challenging, especially for specific locations or corridors. However, quantifying effectiveness is a critical aspect of systemic safety planning. Evaluating program performance is the last element of the Systemic Safety Project Selection Tool, and it provides useful feedback into the systemic safety planning process. Effectiveness information gained through the process of evaluating the performance of implemented countermeasures provides input to agencies that is useful for modifying and evolving their safety programs to prevent and reduce more severe crashes. Evaluating effectiveness also addresses an agency’s responsibility to invest resources in a way that best serves the traveling public and builds confidence that a systemic safety program is a worthwhile investment.
Quantification of effectiveness is critical to generate support for the systemic approach and to build institutional and cultural support to invest funding for this type of analysis and implementation.
Another benefit to be gained by evaluating the safety effectiveness of systemic safety improvements is that positive results may generate support for the systemic approach and build institutional and cultural support to invest funding for this type of analysis and implementation. Building institutional support for systemic safety begins with the knowledge gained through the systemic planning process related to the systemic safety program’s focus on crash types, facility types, risk factors, and countermeasures to all offices within an agency. This promotion of safety throughout an agency is important because planning, design, operations, and maintenance activities all provide opportunities to implement systemic safety countermeasures on focus facilities. In other words, countermeasures do not have to be implemented solely through dedicated safety projects. Through these additional implementation channels for safety improvements, the systemic approach reaches more locations in less time than safety funding alone accomplishes. Sharing the benefits of the approach helps all agency offices understand the justification for adopting changes and the results expected (i.e., fewer severe crashes) if traditional practices are modified to incorporate systemic safety priorities. Performance evaluation results, especially lives saved and injuries prevented, can be compelling information to bring about changes in business practices within agencies.
This section introduces the safety performance evaluation process for a systemic safety program. As the systemic safety evaluation process is new and continually evolving, this section does not present a process or a framework for evaluation. Instead, the following chapter provides an overview of an approach and potential methods, including the data needs and performance measures for these methods. This chapter also discusses several scenarios agencies might face as they evaluate their program. As the systemic approach to safety evolves and implementation continues, additional research is necessary to confirm systemic safety evaluation techniques.
Data Needs
Data required to evaluate the systemic portion of an agency’s safety program depends upon the level of analysis—systemwide versus improved locations. Evaluating improved locations requires the crash, roadway, and traffic data assembled during the systemic safety planning process (Element 1) and for a minimum of three years after implementation, and details about the implementation of specific systemic safety countermeasures. Data required to perform program evaluations include statewide, regionwide or systemwide crash and roadway data within the study area. The countermeasure-specific data are for the actual sites where projects (the term "projects" includes dedicated safety projects as well as safety improvements implemented as one component of a traditional construction or maintenance project like resurfacing or as part of routine maintenance efforts) were implemented and include key, descriptive information about the project type, a detailed definition of the location, site conditions, and documentation of when specific countermeasures were implemented. The Recommended Protocols for Developing Crash Modification Factors report describes the collection and use of evaluation data to assess countermeasure effectiveness (Carter, 2012).
Although many states evaluate their safety program annually to meet Federal reporting requirements, a countermeasure’s effectiveness should not be based on a single year of data.
Crash data support the effort to evaluate countermeasure effectiveness relative to changes in crash frequency. As a result, it is important to consider statistical reliability, which is highly influenced by sample size. Thus, while crash data would likely be documented on a site-by-site basis, data should ultimately be "rolled up" to represent the entire system along which a particular countermeasure was deployed. This "roll up" of crash data to the system level recognizes that, in the systemic approach, some deployment might take place at locations with a history of no or few crashes, which means that change in crash frequency at specific locations does not sufficiently tell the whole story about effectiveness. Also, rolling the crash data up to the system level maximizes the crash data sample size, which provides the greatest chance for statistical reliability. Analyzing at least three years of crash data after implementation also helps to attain a sufficient sample of crashes.
Systemic Safety Performance Measures
The Systemic Safety Project Selection Tool relies on severe crash history and other indicators, such as risk factors, to determine future crash potential of particular sites. The resulting widespread deployment of countermeasures is intended to reduce the future crash potential. Therefore, the systemic safety program’s effectiveness can be measured by reduced system risk. Depending on the availability of data applied in the project identification process, risk might not be easily quantifiable. Instead, changes in the number of severe focus crash types, especially on the focus facility types, become the long-term performance measurement.
The recommended systemic safety evaluation process occurs at three levels, as follows:
- Output: What is the output of the systemic safety program? Is the systemic safety program being implemented as planned and programmed? Are high priority countermeasures being deployed at the right type of locations and at the number of locations planned?
- Focus Crash Type: Has implementation effectively reduced the identified focus crash types? That is, are the severe crashes trending down?
- Countermeasure Performance: Within each crash type, are deployed countermeasures performing as expected?
The following information provides an overview of these evaluation levels by identifying the purpose of each part of the evaluation and describing a general approach. This overview does not provide step-by-step instructions for specific analysis methodologies. This chapter concludes with sources for additional information to support systemic safety program evaluation efforts.
Suggested Evaluation of Data
Countermeasure
- Definition of improvement implemented at site
- Precise implementation location
- Precise implementation date
Crash Data
- Severe crash data for focus crash type before implementation
- Severe crash data for focus crash type after implementation
Before and After Site Conditions
- Roadway or intersection geometry
- Intersection traffic control device
- Shoulder surface width and type
- Road division
- Median width and type
- Speed limit
- Average daily traffic volumes
- Average daily entering vehicles
- Roadway classification
- Area type (rural/urban/suburban)
Systemic Safety Program Output
The first level of evaluation is an interim evaluation because, at such an early stage in program development, the evaluation is of "output," rather than "outcome." A period of at least three years is ideal to evaluate changes in crashes (the outcome); therefore, instead of waiting, agencies should begin annually reviewing their funding decisions to evaluate if selected improvements are consistent with the systemic funding goal (the output). Comparing allocated funding to the planned systemic safety program on an annual basis reveals opportunities to better align the following year’s systemic safety programming with program goals. Agencies might need to adjust funding within the systemic allocation or between the systemic and site-specific allocations.
The process of balancing safety investments between projects identified through the site analysis approach versus the systemic approach involves reviewing past funding practices (Element 2) to gain an understanding of whether funded projects were directed toward the focus crash types and facilities identified through the systemic safety planning process (Element 1). Each funding cycle provides the opportunity to perform an interim evaluation using this same review technique. If there is a preference for and more familiarity with historical practices, projects selected for safety funding could easily stray from the focus crash type, focus facility type, or preferred countermeasure in new and evolving systemic safety programs. If this were to happen, then the outcome could very well fall short of what is expected of a systemic safety program.
An output evaluation of the planned funding and implementation, as compared with actual funding, should consider the following:
- Did systemic safety projects reach the dollar amount set as a goal? If a goal was set for individual focus crash types, how do the planned and actual distributions compare? The answers indicate whether the goals for implementing improvements developed through a systemic risk assessment have been achieved.
- Were projects implemented on the focus facility types, especially locations consistent with the identified risk factors?
- Were investments distributed regionally (e.g., by district) and jurisdictionally (i.e., state versus local) as intended?
- Were the preferred countermeasures successfully selected for most candidate locations? The answer provides feedback about the list of priority countermeasures and the effectiveness of the countermeasure selection process.
Answering Some Common Concerns
Q: What if the program fell short of the overall spending goal and more systemic projects should have been implemented?
A: Consider two possible questions that may help to identify the reason:
- Was agency culture a limiting factor? That is, did funding decisions revert to previous practice?
- Were a sufficient number of sites reviewed to fulfill the funding available?
Addressing issues related to the first question requires continued effort to educate peers about the new systemic safety program, possibly providing assistance with selecting projects for programming. The second question addresses the systemic safety program design. Consider whether the problem might be the criteria used to identify which locations were eligible for project programming. A threshold that is overly strict can identify too few locations. Setting lower thresholds will allow the identification of more locations during the project identification process.
Q: What if plenty of locations were identified, but few projects were -programmed because the countermeasure did not prove to be implementable at most of the locations?
A: A countermeasure may not be implementable at certain locations because of noise or lighting concerns, roadway cross-section, pavement condition or right-of-way constraints. Consider if the preferred countermeasure is appropriate for the focus facility types, especially locations that rank high in the prioritization process. For those locations where it is not appropriate, return to Element 1 to identify other countermeasures. Also review the decision process created in Element 1 that assigned a project to each location—were several applicable countermeasures assigned properly? Make sure the project selection process has appropriate alternatives with clear guidelines on when they can be used.
Q: There was not an issue with allocating funding, but what if I’m not certain enough locations were improved to make a difference in crash reduction or the potential for crash reduction?
A: A key aspect to a systemic safety program is widespread deployment of effective countermeasures. If funding needs to cover more locations, review the preferred and alternative countermeasures and the decision process that identifies the project type for each location. If the preferred countermeasure is too costly, then search for an alternative that is less costly to implement. Also refine the project type selection process to allow greater use of low-cost countermeasures by ensuring criteria do not allow overuse of higher-cost countermeasures. This decision-making process may be enhanced if CMFs are applied to understand crash reduction potential for the improved locations. An agency can estimate the expected benefits based on the crash reduction for all improved locations.
Observed Trends in Crash Frequency or Severity
The second level of evaluation is based on program-level trends that characterize the impact the countermeasures have on safety. This outcome evaluation addresses questions like What happened to the number of severe focus crash types, especially on focus facilities? The systemic safety program focuses on severe crashes using an approach that relies on a long-term outlook across entire roadway systems (e.g., rural freeways, rural county highways, urban signalized intersections). Widespread implementation of systemic countermeasures requires a long-term perspective because several funding cycles may be necessary to fund the improvements, culture change takes time, and at least three years of crash data should be gathered to determine countermeasure effectiveness that is useful for program modifications. Therefore, evaluation also requires a long-term perspective. Continuous long-term tracking is critical for identifying program impacts and useful life.
The NYSDOT has developed a tool to track safety projects to support a "before versus after" evaluation. The tool is called the Post Implementation Evaluation (PIE) System. PIE is organized by project type; documents where the project was implemented and when construction began and was completed; links with the crash record system to document the number of crashes in the before and after periods; and computes a CMF.
As previously discussed, communicating the program results is important to encourage agencies to embrace systemic safety efforts in their daily activities and to promote stakeholder and public support for continuing a systemic safety program. Simple charts and graphs are one method to visually disseminate program results to technical or non-technical audiences. A chart that presents information in an easy-to-understand format allows stakeholders and program managers to comprehend the potential relationship between crash reduction and implementation of systemic safety countermeasures. While not a statistical test, these simple tools easily demonstrate the relationship between actions and results to various audiences, including stakeholders, elected officials, and executive management.
Figure 10 is an example of a chart that displays Minnesota’s statewide traffic fatalities for certain crash types (total, lane departure, and aggressive/speeding driving) along with the time period when related systemic safety improvement programs were initiated (Minnesota DOT, 2012). The figure shows that total fatalities were on a downward trend starting in 2005. However, lane departure and speeding-related fatalities were relatively constant (or decreasing at a rate slower than total fatalities) until their respective countermeasure programs were initiated. Minnesota began a targeted enforcement campaign called Highway Enforcement of Aggressive Traffic (HEAT) in the spring of 2006. High enforcement levels were sustained through mid-2007 and reinitiated in 2009. Figure 10 illustrates that related fatalities dropped each year of the enforcement campaign. Likewise, MnDOT began widely implementing countermeasures to address lane departure crashes during the 2009 construction program. At about the same time, the data appear to indicate that lane departure fatalities began decreasing at a faster rate. These charts were well received by managers and stakeholders and helped them understand the potential impact the programs had on severe crashes.
FIGURE 10. Program Results for Addressing Lane Departure and Speeding-Related Fatalities
In addition to looking at annual totals only, MnDOT separated the data for the portions of the year when HEAT was in effect and when it was not. The analysis revealed that traffic fatalities were occurring at a faster pace than the previous year until HEAT began. Following the initiation of HEAT, the number of fatalities per month decreased and the state finished the year with fewer fatalities than the previous year. This approach focusing specifically on crashes that occurred during the targeted time intervals might better highlight the specific impact of safety programs.
Figure 11 provides a chart showing information about the potential impact of a systemic safety countermeasure on a focus crash type. This chart presents data for a focus facility rather than for all roads in the state, as presented in Figure 10. Figure 11 compares the numbers of total and fatal cross-median crashes (left axis) to the miles of high-tension cable median (HTCM) barrier (right axis) installed between 2005 and 2008 (Illinois DOT, 2009). Prior to HTCM installation, an annual average of 11 total cross-median crashes (range of 6 to 16) and less than 4 fatal cross-median crashes occurred at these locations. The chart data demonstrate that the number of cross-median crashes began decreasing the year after installation began (2006) and continued to decrease to a low of zero during the final year of construction.
FIGURE 11. Illinois Department of Transportation Illustration of Cable Median Barrier Program Results for Treated Locations
This illustration technique was also used to demonstrate the positive results achieved after installing HTCM barrier along Interstate 70 across Missouri (Chandler, 2007). For another example, the NYSDOT report Centerline Rumble Strips on Secondary Highways: A Systemic Crash Analysis includes a simple line chart that compares the number of head-on fatality crashes with the installation of centerline rumble strips (NYSDOT, 2011). The same positive result of implementing systemic safety improvements is clearly discernible from the chart. This example is similar to the Minnesota example in that the chart shows statewide data for head-on fatality crashes rather than just data for the improved locations as the Illinois and Missouri examples show.
Another method to communicate a systemic safety program’s overall effectiveness is the dissemination of performance metrics such as cost effectiveness or benefit-cost ratio data. The cost-effectiveness performance measure expresses the cost invested to prevent each severe crash—that is, dollars spent per severe crash (or fatal crash or fatality) prevented. From a funding perspective, the lower the cost-effectiveness value or the higher the benefit-cost ratio, the more successful the program is at reducing severe crashes. The method is also useful to compare systemic safety programs addressing different focus crash types as a way to normalize each program. Within a focus crash type, calculating cost-effectiveness values or benefit-cost ratios for individual districts, jurisdictions, and facility types allows comparison of the subareas to understand which parts of the program have been the most successful.
Countermeasure Performance
The first two levels of evaluation consider whether or not the systemic component of a safety program is being funded and/or implemented as intended and whether the overall program is a success. The outcome of these evaluations provides feedback about the consistency of an adopted funding goal to direct some fraction of safety investments toward improvements identified through a systemic risk assessment and long-term, systemwide crash trends. The third level of evaluation identifies how individual countermeasures perform on a systemic basis. This information allows program managers to understand the individual parts of the program and identify which countermeasures successfully reduced specific focus crash types and which did not. In the interest of an owner agency’s responsibility to invest resources in a way that best serves the traveling public, this third-level evaluation provides the opportunity for agencies to continue directing investments toward effective countermeasures and discontinue funding countermeasures not achieving desired results. However, given that a systemic safety program very likely will direct investments to facilities with few or no crashes, specific locations or corridors should never be evaluated. Instead, all improved locations should be aggregated and evaluated as a single set. This provides an estimate of the countermeasure’s effectiveness for a typical facility.
Answering Some Common Concerns
Q: What if I do not see a noticeable change in severe crashes at the program level?
A: To see results at the program level, you will first have to improve enough locations so that results are noticeable. If you have not been able to implement enough projects to observe results systemwide, focus on the locations that have been improved and aggregate the data for all improved locations. Remember to collect data for a sufficient amount of time before any countermeasures are implemented to establish a baseline crash frequency so that impacts are observable. This may reveal effectiveness at the implemented locations that simply is not observable systemwide.
If severe crash reductions are still not visible when looking only at improved locations, you may need to move onto evaluating countermeasure performance. This will provide you greater detail on how specific countermeasures have performed, which could reveal that implementation has been too focused on a specific countermeasure that hasn’t proven effective at reducing the severe focus crash type.
More advanced techniques can play a greater role in this third level of evaluation. These techniques include Empirical Bayes (EB) to account for regression to the mean, multivariable regression to account for more than one independent variable, or confidence tests that determine the level at which the results are statistically reliable. When these methods are not options (e.g., a lack of safety performance functions to apply EB adjustments), then simple techniques (e.g., a before-after evaluation with control sites) can increase the confidence of the results. Also, benefit-cost or cost effectiveness evaluations quantify the crash reductions in monetary terms.
It is important to understand that the approach described here and illustrated by Example 11 is the best available at this time. Additional research and future studies will define a best practice to evaluate countermeasure performance for systemic projects.
EXAMPLE 11. Missouri Department of Transportation Evaluation Using Empirical Bayes Methodology
Using the Empirical Bayes evaluation methodology with safety performance functions, the Missouri Department of Transportation (MoDOT) evaluated their Smooth Roads Initiative (SRI), which improved 2,300 miles of roadways with resurfacing, improved markings, and centerline rumble strips or shoulder/edgeline rumble strips (including combinations of these countermeasures) in 2005 and 2006 (MoDOT, 2011). This evaluation computed a benefit-cost ratio for the improvements and the percent reduction in crashes for fatal crashes, fatal plus disabling injury crashes (also known as severe crashes), and fatal plus all injury crashes. The analysis was structured so that each combination of countermeasures was analyzed for each facility type for which it was implemented.
Disaggregating the analysis to this level of detail allowed MoDOT to understand the degree to which individual countermeasures reduced crashes or the potential for crashes on each facility type. This study of 18 countermeasure combinations concluded that all have statistically significant results with benefit-cost ratios substantially greater than 1.0. Of the 18 combinations, the following four were reported as being particularly cost-effective:
- Wider markings with resurfacing on rural multilane undivided highways: benefit-cost ratio = 146
- Wider markings with resurfacing on urban two-lane highways: benefit-cost ratio = 118
- Wider markings and both centerline and edgeline rumble strips with resurfacing on rural two-lane highways: benefit-cost ratio = 36
- Wider markings without resurfacing on urban multilane divided highways: benefit-cost ratio = 29
MoDOT also conducted an evaluation of a subsequent project that installed edgelines along 650 miles of low volume rural, two-lane roads in 2009. During the two years before the countermeasure was implemented, 105 fatal and injury crashes and 576 total crashes occurred along these roadway segments. During the two years afterward, 46 fatal and injury crashes and 327 total crashes occurred along the same segments.
MoDOT used their Countermeasure Evaluation Tool to perform a "Before vs. After" evaluation. The Countermeasure Evaluation Tool is a customized spreadsheet that incorporates Empirical Bayes methodology to estimate the effectiveness of the implemented countermeasure. The tool was created during the development of Missouri safety performance functions.
The MoDOT applied Empirical Bayes methodology with safety performance functions to evaluate the performance of countermeasures implemented to reduce lane departure crashes.
The countermeasure effectiveness is based on a comparison of the expected number of crashes with and without the edgeline treatment for the two years before the installation and the two years after the installation of the edgelines in 2009. Data input into the spreadsheet included segment beginning/end mile posts, Average Annual Daily Traffic, crash frequencies for each segment, and roadway type (i.e., rural two-lane undivided).
With regard to the expected number of total crashes, the analysis revealed a 15 percent decrease in crashes occurred after the countermeasure was put into place, including a finding of significance at the 95 percent confidence level (indicating a high degree of certainty that the edgelines contributed to the reduction in total crashes). Based on the expected number of fatal and injury crashes with and without the countermeasure, the analysis found a 19 percent decrease in crashes; however this was not significant at the 90 percent confidence level. Despite a 19 percent decrease in the expected number of fatal and injury crashes following the deployment of edgelines, the relatively low density of injury crashes prevented the result from being statistically significant. While the result was viewed positively by the MoDOT, the result of statistical significance means there is a lesser degree of certainty that the edgelines contributed to the reduction in fatal and injury crashes.
A detailed breakdown of results by countermeasure and facility type (as Example 11 shows) provides the opportunity to refine a systemic safety program by directing subsequent funding to proven projects and countermeasures. These refinements allow agencies to further their efforts to maximize crash reduction by allocating funding where it will provide the greatest benefit toward reducing of the frequency of severe focus crashes.
A benefit of evaluating countermeasures is that the effort directly leads to developing CMFs that are specific for the systemic safety program. These results are useful for the particular agency’s efforts to incorporate safety into planning, design, operations, and maintenance projects. These CMFs are also useful to augment the common body of knowledge for systemic safety planning. As mentioned previously, many of the current CMFs were developed from programs that treated locations identified through the site analysis approach and may not be relevant to systemic safety programs. The availability of CMFs developed specifically for systemic safety improvements assists all agencies to better predict potential outcomes of safety countermeasure alternatives.
Program managers might also consider alternative performance measures to determine the effectiveness of individual countermeasures. Results such as reduced travel speeds, changes in citations issued, fewer red-light-running violations, and fewer maintenance repairs might indicate that the countermeasure deployment has changed driver behavior. Additionally, feedback from staff who spend a considerable amount of time in the field (e.g., maintenance staff, law enforcement) or citizens can provide early feedback on countermeasure performance.
Answering Some Common Concerns
Q: What if my countermeasure is sufficiently proven effective, but a systemwide reduction in severe crashes is still not visible?
A: Continue implementing the countermeasure. As deployment increases, systemwide results will become visible.
Q: What if my preferred countermeasure is not proving effective?
A: Begin reviewing the process to identify locations for improvements. Review the risk factors to confirm their ability to indicate potential for a severe focus crash type and as necessary, adjust future risk analysis to compensate for any lessons learned.
Q: If my low-cost preferred countermeasure is not effective as the higher cost alternative, how do I achieve a balance in future implementation?
A: You should strive for a balance that maximizes the number of locations improved by selecting the lower-cost countermeasure where appropriate but relies on the higher cost scenario where justified. For the high priority locations, an analysis that computes the potential crash reduction for different implementation scenarios can help identify the plan that should achieve the greatest system results.
Where to Go for More Information
Several resources provide information about countermeasure evaluation practices, both for individual locations and for systemic safety programs. The following lists a few of these resources:
- The Highway Safety Manual (AASHTO, 2010) contains detailed information on countermeasure evaluation in Part B.
- The Art of Appropriate Evaluation, A Guide for Highway Safety Program Managers is published by the NHTSA (2008) for use in evaluating traffic safety programs, especially those that focus on driver education and enforcement. The guide contains practical advice that is applicable to systemic engineering programs.
- More detail on the evaluation approach to MoDOT’s Smooth Roads Initiative program can be found in the Benefit/Cost Evaluation of MoDOT’s Total Striping and Delineation Program: Phase II – Final Report (MoDOT, 2011).
- A Guide to Developing Quality Crash Modification Factors (FHWA, 2010), provides direction to agencies developing CMFs, including selecting appropriate evaluation methods and data considerations.
- The NCHRP report, Recommended Protocols for Developing Crash Modification Factors (Carter, 2012), provides guidance about data to collect and a process to follow for evaluating countermeasures. The report documents protocols that should be used in the development of CMFs, the goal of which is to describe what pieces of the research study should be documented by the study authors and how various potential biases should be addressed.
Summary
Performance evaluation provides useful feedback for safety program decision-making and, as such, is an important aspect of continuing agency efforts to reduce severe crashes. Documenting systemic safety program effectiveness also helps agencies understand the impact a systemic safety program will achieve if incorporated into other standard agency practices for planning, design, operations, and maintenance activities.
Evaluating a systemic safety program comes with unique challenges, especially because implementation is based on risk factors in addition to historical crash performance. Therefore, it is necessary to evaluate the system at a program level and not at individual project sites. The evaluation process includes three levels. The first, an interim evaluation, is a check on output or projects programmed and constructed. This first evaluation is especially important in new systemic safety programs because modifications to the project identification and selection process might be required, and the program will be more successful if these are made sooner rather than later. Tracking long-term performance at the program level is the second level of analysis. Data about systemic countermeasure performance are especially useful for communicating results to stakeholders, executive management, and other interested parties. The third level, evaluating specific countermeasures, allows program managers to understand how individual elements are performing, identify where changes are needed, and adjust future funding to maximize return on investment. As with any evaluation, planning ahead is important. Identifying data needs in advance ensures that all relevant data are gathered at the appropriate time after projects are implemented.