Chapter 1 – Evaluation Planning Checklist
The following checklist is designed to support evaluation planning. If most or all of these activities are completed, the State is prepared for SHSP evaluation.
- Identify evaluation objectives.
- Identify the data needed to address the objectives and perform the evaluation.
- Determine if existing data collection strategies are sufficient for evaluation.
- Identify resources needed to collect data or adjust evaluation objectives if available data are insufficient for evaluation purposes.
- Assign responsibility for generating and distributing evaluation results.
- Document a reporting process to update agencies, partners, and decision-makers on SHSP evaluation results.
- Determine how evaluation results will be applied.
- Document the approach or plan for the evaluation.
Strategic Highway Safety Plan Evaluation Process Model Worksheet – Chapter 2
Process Evaluation – Getting on the Right Track
Column Descriptions
As evidenced by: What is the evidence for the yes (or the no) answer?
What is the Impact: What impact, positive or negative, is this having on our SHSP process?
Opportunities: How can we improve our efforts with respect to this question?
Action Items: What steps will we take to improve our process, when should the steps be completed, and who is responsible for completing them?
The elements examined in process evaluation are: SHSP organizational structure; multidisciplinary, multimodal collaboration; goal and target setting methods; data driven and evidence-based emphasis areas, strategies, and actions; and aligning agency priorities. For each of these elements the primary question driving the evaluation is:
What will it take to convince us our programs/
strategies/actions/investments are on the right track?
The questions below will help determine the effectiveness of current process evaluation efforts and identify areas for improvement. The EPM is based on noteworthy practices and current research and is general in nature. States should feel free to add to or modify these questions to fit their particular situation.
|
SHSP Organizational |
Yes/No |
As evidenced by (observations) |
What is the impact |
Opportunities |
Action Items |
|---|---|---|---|---|---|
|
1. Is the SHSP process supported by an actively engaged organizational structure? |
|
|
|
|
|
|
2. Are top-level managers represented in executive committees or leadership structures/groups established for the SHSP? |
|
|
|
|
|
|
3. Do members of the executive committee or leadership group have the decision-making authority needed to effectively support the SHSP process? |
|
|
|
|
|
|
4. Do members of the executive committee or leadership group assign persons with decision-making authority to the steering committee or working group? |
|
|
|
|
|
|
5. Are multiple transportation modes represented, and do they actively participate on the steering committee/working groups and emphasis area teams? |
|
|
|
|
|
|
6. Has an SHSP program coordinator or manager been assigned? What percentage of this person’s time is dedicated to the SHSP? |
|
|
|
|
|
|
7. Do the leadership and working groups/committees meet as frequently as expected? |
|
|
|
|
|
|
8. Are emphasis areas supported by teams with engaged leaders? |
|
|
|
|
|
|
Multidisciplinary and Multimodal Collaboration Questions |
Yes/No |
As evidenced by (observations) |
What is the impact |
Opportunities |
Action Items |
|---|---|---|---|---|---|
|
1. Are members of the executive or leadership group, the steering committee, the emphasis area teams, and other groups multidisciplinary and multimodal? |
|
|
|
|
|
|
2. Are local/regional/district coalitions supported by the SHSP organizational structure? |
|
|
|
|
|
|
3. Are the necessary disciplines, modes, and agencies (representing the 4 E’s) engaged in SHSP decision-making and implementation? |
|
|
|
|
|
|
4. Do the stakeholders regularly collaborate on decisions that affect SHSP updates and implementation? |
|
|
|
|
|
|
5. Do the necessary stakeholders collaborate and jointly decide on SHSP goal and objective setting methods? |
|
|
|
|
|
|
Goal- and Objective-Setting Methods Questions |
Yes/No |
As evidenced by (observations) |
What is the impact |
Opportunities |
Action Items |
|---|---|---|---|---|---|
|
1. Are data-driven methods, such as trend analysis, used to establish goals and set aggressive, yet achievable objectives? |
|
|
|
|
|
|
2. Are objectives and goals specific, measurable, time bound, and realistic? |
|
|
|
|
|
|
Data-Driven and Evidence-Based Emphasis Areas, Strategies, and Actions Questions |
Yes/No |
As evidenced by (observations) |
What is the impact |
Opportunities |
Action Items |
|---|---|---|---|---|---|
|
1. Is data analysis used to select the emphasis areas? |
|
|
|
|
|
|
2. Are the emphasis area strategies selected through an evidence based process? |
|
|
|
|
|
|
3. Are promising and innovative strategies with less evidence of effectiveness accompanied by an evaluation? |
|
|
|
|
|
|
Aligning Agency Priorities Questions |
Yes/No |
As evidenced by (observations) |
What is the impact |
Opportunities |
Action Items |
|---|---|---|---|---|---|
|
1. Have the various agencies and safety partners incorporated elements of the SHSP into their planning documents? (HSPs, HSIPs, CVSPs, LRTPs, S/TIPs, etc.) |
|
|
|
|
|
Action Item Summary
|
SHSP Organizational Structure – Action Items |
Deadline |
Responsible Person |
|---|---|---|
|
1. |
|
|
|
2. |
|
|
|
3. |
|
|
|
4. |
|
|
|
5. |
|
|
|
6. |
|
|
|
Multidisciplinary and Multimodal Collaboration – Action Items |
Deadline |
Responsible Person |
|---|---|---|
|
1. |
|
|
|
2. |
|
|
|
3. |
|
|
|
4. |
|
|
|
5. |
|
|
|
6. |
|
|
|
Goal- and Objective-Setting Methods – Action Items |
Deadline |
Responsible Person |
|---|---|---|
|
1. |
|
|
|
2. |
|
|
|
3. |
|
|
|
4. |
|
|
|
5. |
|
|
|
6. |
|
|
|
Data-Driven and Evidence-Based Emphasis Areas, Strategies, and Actions – Action Items |
Deadline |
Responsible Person |
|---|---|---|
|
1. |
|
|
|
2. |
|
|
|
3. |
|
|
|
4. |
|
|
|
5. |
|
|
|
6. |
|
|
|
Aligning Agency Priorities – Action Items |
Deadline |
Responsible Person |
|---|---|---|
|
1. |
|
|
|
2. |
|
|
|
3. |
|
|
|
4. |
|
|
|
5. |
|
|
|
6. |
|
|
Chapter 2 – Self Assessment Questions
The following self assessment questions are designed to inform process evaluation. Answering “yes” to a question indicates the State has a well functioning SHSP process in that area of review. Answering “no” indicates improvements can be made.
- Is the SHSP process supported by an actively engaged organizational structure?
- Are top-level managers represented in executive committees or leadership structures/groups established for the SHSP?
- Are members of the executive or leadership group, the steering committee, the emphasis area teams, and other groups multidisciplinary and multimodal?
- Do members of the executive committee or leadership group have the decision-making authority needed to effectively support the SHSP process?
- Do members of the executive committee or leadership group assign persons with decision-making authority to the steering committee or working group?
- Are multiple transportation modes represented, and do they actively participate on the steering committee/working group and emphasis area teams?
- Has a SHSP program coordinator or manager been assigned? What percentage of this person’s time is dedicated to the SHSP?
- Do the leadership and working groups/committees meet as frequently as expected?
- Are emphasis areas supported by teams with engaged leaders?
- Are local/regional/district coalitions supported by the SHSP organizational structure?
- Are the necessary disciplines, modes, and agencies (representing the 4 E’s) engaged in SHSP decision-making and implementation?
- Do the stakeholders regularly collaborate on decisions that affect SHSP updates and implementation?
- Do the necessary stakeholders collaborate and jointly decide on SHSP goal and target setting methods?
- Are data-driven methods, such as trend analysis, used to establish goals and set aggressive, yet achievable, objectives?
- Are objectives specific, measurable, time bound, and realistic?
- Is data analysis used to select the emphasis areas?
- Are the emphasis area strategies selected through an evidence-based process?
- Are promising and innovative strategies with less evidence of effectiveness accompanied by an evaluation?
- Have the various agencies and safety partners incorporated elements of the SHSP into their planning documents? (HSPs, HSIPs, CVSPs, LRTPs, S/TIPs, etc.)
Strategic Highway Safety Plan Evaluation Process Model Worksheet – Chapter 3
Performance Evaluation – Outputs and Outcomes
Column Descriptions
As evidenced by: What is the evidence for the yes (or the no) answer?
What is the Impact: What impact, positive or negative, is this having on our SHSP process?
Opportunities: How can we improve our efforts with respect to this question?
Action Items: What steps will we take to improve our process, when should the steps be completed, and who is responsible for completing them?
Performance evaluation is comprised of: output evaluation and outcome evaluation. For each of these areas the primary question driving the evaluation is:
What will it take to convince us our programs/
strategies/actions/investments are getting us where we need to be?
The questions below will help determine the effectiveness of current performance evaluation efforts and identify areas for improvement. The EPM is based on noteworthy practices and current research and is general in nature. States should feel free to add to and/or modify these questions to fit their particular situation.
|
Performance Evaluation Questions |
Yes/No |
As evidenced by (observations) |
What is the impact |
Opportunities |
Action Items |
|---|---|---|---|---|---|
|
1. Has the current status of all output and outcome performance measures been gathered and reviewed? |
|
|
|
|
|
|
2. Are the performance measures clearly related to SHSP goals and objectives? |
|
|
|
|
|
|
3. Are the numbers and rates of fatalities and serious injuries used as general statistical measures? |
|
|
|
|
|
|
4. Are the numbers and rates of fatalities and serious injuries tracked and reported by emphasis area and compared to previously set objectives? |
|
|
|
|
|
|
5. Have fatality and serious injury objectives been met? |
|
|
|
|
|
|
6. Are observation and/or telephone survey data collected and analyzed to track changes in awareness, attitudes, and behaviors? |
|
|
|
|
|
|
7. Have awareness, attitude, and behavior objectives been met? |
|
|
|
|
|
|
8. Are program-level benefit/cost analyses conducted on certain SHSP programs? |
|
|
|
|
|
|
9. Have the benefits of the program(s) outweighed the costs? |
|
|
|
|
|
Action Item Summary
|
SHSP Organizational Structure – Action Items |
Deadline |
Responsible Person |
|---|---|---|
|
1. |
|
|
|
2. |
|
|
|
3. |
|
|
|
4. |
|
|
|
5. |
|
|
|
6. |
|
|
Chapter 3 – Self-Assessment Questions
The following self-assessment questions are designed to inform performance evaluation. Answering “yes” to a question indicates the State’s SHSP has been effective or successful in this area of performance evaluation. Answering “no” indicates improvements can be made.
- Has the current status of all output and outcome performance measures been gathered and reviewed?
- Are the performance measures clearly related to SHSP goals and objectives?
- Are the numbers and rates of fatalities and serious injuries used as general statistical measures?
- Are the numbers and rates of fatalities and serious injuries tracked and reported by emphasis area and compared to previously set objectives?
- Have fatality and serious injury objectives been met?
- Are observation and/or telephone survey data collected and analyzed to track changes in awareness, attitudes, and behaviors?
- Have awareness, attitude, and behavior objectives been met?
- Are program-level benefit/cost analyses conducted on certain SHSP programs?
- If so, have the benefits of the program(s) outweighed the costs?
Chapter 4 – The Focus is Results Checklist
The following checklist is designed to support the use of evaluation results. If an action on the checklist is in progress or completed, the State is well on the way to using evaluation results to improve the SHSP process and performance.
- Evaluation results have been interpreted and documented.
- Evaluation results were reviewed to identify lessons learned.
- Lessons learned have been used to improve SHSP process and performance.
- SHSP stakeholders are made aware of both successful and unsuccessful programs and strategies as soon as sufficient information is available.
- Evaluation results are being used to increase public understanding of SHSP programs and strategies.
- Evaluation results are being used to help inform elected and appointed officials so they might support increased funding and resources for safety programs and strategies.
- Evaluation results are being used to identify additional safety leaders and partners.
- Evaluation results that identify gaps and weaknesses in SHSP process or performance are being addressed through follow-up actions.
- Evaluation results are used to direct resources to areas with the highest probability of improving safety.