Performance Measures for Impaired Driving Programs

Performance measures and benchmarks provide meaningful and standardized measures in the traffic safety field, and, specifically, in efforts to address impaired driving. Reasons that performance measures are important when addressing impaired driving are listed below.

  • There is a need for meaningful and standardized measures of problems and solutions to facilitate the implementation of appropriate strategies and comparisons across local, regional, national, and international jurisdictions.
  • To increase understanding of the different facets of the impaired driving problem and afford administrators and policymakers insight into available opportunities to address the issue and to measure progress in doing so.
  • They provide an ability to measure progress connected to the ability to effectively fill existing gaps and the implementation of evidence-based strategies to reduce impaired driving.
  • To assist agencies in:
    • setting goals,
    • monitoring and evaluating progress in achieving goals,
    • connecting goals to actions and strategic decisions,
    • allocating limited resources in accordance with need,
    • communicating results, and
    • making informed decisions.
  • To help define priority objectives, focus programs, measure progress, control costs, and increase collaboration.
  • Effectiveness. Has the program, strategy, and/or policy met its stated objectives in terms of its process or activities as well as its outcomes?
  • Efficiency. Has the program, strategy, and/or policy been implemented and delivered in a cost-effective, organized fashion, and in accordance with plans?
  • Quality. Has the program, strategy, and/or policy met the delivery standards developed prior to implementation?
  • Timeliness. Has the program, strategy, and/or policy been implemented according to pre-determined timeframes or were there delays?
  • What are the goals?
  • What measures best demonstrate whether goals are being achieved?
  • What can be measured and are the measures useful?
  • Is there capacity to collect and analyze data for various measures?
  • Are current practices producing desired results?
  • Developed with a consensus-based approach or the establishment of some minimum guidelines to guide the selection of performance measures;
  • operationally defined;
  • useful and valid (i.e., demonstrate an effect on the issue);
  • uniform to facilitate comparisons;
  • sensitive to trends;
  • long-term to make comparisons over time;
  • acceptable and agreed upon by stakeholders;
  • accurate, reliable, and repeatable;
  • what is measured can be easily communicated;
  • collected in a timely fashion; and,
  • cost-efficient.

Law enforcement

  • Number and percentage of impaired drivers involved in alcohol-related crashes and fatal crashes.
  • Number of DWI arrests according to specific factors (e.g., BAC level, crashes).
  • Number of interventions implemented (e.g., saturation patrols, sobriety checkpoints).
  • Number and percentage of patrol officers with specialized DWI training.
  • Number and percentage of arrests leading to a conviction (for the original DWI charge or a reduced charge).
  • Other measures of interest (e.g., location of arrests, cost of enforcement, missed court hearings).

Prosecution

  • Ratio of post-arraignment cases adjudicated vs. cases charged.
  • Number of cases not adjudicated for prosecutorial reasons.
  • Number of dismissals, dispositions, and pleas.
  • Frequency of prosecutorial error or disciplinary actions.
  • Caseload ratio vs. workload ratio (e.g., number of DWI cases assigned to each prosecutor – misdemeanor vs. felony; average number of cases assigned to each prosecutor at any given time).
  • Other measures of interest (e.g., referrals to DWI/Drug Court, hours spent on DWI cases).

Judiciary

  • Number of cases assigned.
  • Number of continuances per case.
  • Number of dismissals (with and without prejudice).
  • Number of appellate reversals of decisions.
  • Average amount of time to resolve a case (pleas, plea agreements, trials).
  • Court clearance rate.
  • Number of offenders ordered to install an ignition interlock device.
  • Number and type of treatment orders for offenders assessed as having substance use issues.
  • Recidivism rates per disposition type.
  • Other measures of interest (e.g. referrals to specialized courts, disposition consistency).
  • DWI Courts
    • number of offenders who participated;
    • offender retention rate;
    • number of offenders referred to treatment and number of offenders who successfully completed treatment;
    • number of offenders who graduated and number of offenders who dropped-out or violated; and,
    • rate of recidivism among graduates.

Supervision

  • Frequency of risk and substance use screening and assessment.
  • Number of referrals to various interventions.
  • Number and types of violations.
  • Recidivism rate.
  • Rate of successful completion of supervision.
  • Caseload ratio vs. workload ratio.
  • Frequency of use of evidence-based principles and best practices.
  • Other measures of interest for practitioners (e.g., frequency of use of specialized risk assessment tools, rate of participation in and successful completion of treatment, frequency of substance use testing)
  • Other measures of interest for the agency:
    • number of cases supervised;
    • average number of cases per officer;
    • number of DWI specialized caseloads;
    • number of DWI offenders per caseload;
    • staff turnover/burnout rate; and,
    • number of specialized DWI trainings offered annually.

Treatment

  • Number of referrals
  • Number of admissions.
  • Average wait times for program admission.
  • Number of discharges.
  • Rate of successful completion (i.e., remaining in treatment for the duration of programming).
  • DWI recidivism rate.
  • Other measures of interest for treatment (mandatory admissions vs. number of voluntary admissions, average length of stay in program).

There are myriad performance measures available in each facet of the DWI system. Some of these indicators are more readily available than others. This availability is often a function of the quality of reporting and/or data collection protocols. Jurisdictions are encouraged to consider the following recommendations.

  • Identify priority measures. Individual jurisdictional agencies are encouraged to determine which performance measures best gauge progress and outcomes and focus on the consistent collection of these data. The inclusion of too many measures initially could create confusion or exceed agency collection and/or analysis capabilities.
  • Locate data. Once priority measures are identified, agencies need to locate the corresponding data needed to track performance for each measure. It is necessary to determine which agency collects the data, whether it requires filtering, and if it is reported consistently across counties, jurisdictions, or agencies. If certain measures cannot be consistently reported and analyzed without developing entirely new data collection protocols, they may not be the best choices for agencies to rely on as initial indicators of performance.
  • Identify points of contact for known data sources. Following the selection of priority measures and the location of the data required to analyze performance, it is important to identify points of contact within the agencies who collect and maintain the data. The identification of these individuals can facilitate the process of reporting, collection, access, and analysis of measures.
  • Develop uniform definitions of performance measures. It is important to develop uniform definitions of common measures to ensure data are comparable, particularly if a jurisdiction is interested in gauging system performance beyond local levels.
  • Create standardize reporting formats for indicators. To encourage practitioners to report data for the purposes of performance measurement, it is necessary to make this process as seamless as possible. The less onerous reporting is on practitioners, the greater the likelihood that they will complete the task. As such, it is suggested agencies create standardized reporting formats for indicators to be collected.
  • Allow for context. To improve performance measurement, request agencies supply their caveats to provide some context for data submitted (e.g., where it originated, the collection purpose) and explain how particular measures are counted (e.g., through a notes field). Providing data context to those who analyze and review performance measurements can help them correctly interpret the data.
  • Reinforce the importance of data collection and reporting of indicators to practitioners. Agencies sometimes overlook the importance of informing frontline practitioners about how data are used to inform decisions and why collection is essential. It is imperative to demonstrate what actionable data entails and to make data meaningful for practitioners. In order to gain buy-in, agencies might consider illustrating the importance of data collection to inform decision-making and the need to enter data into state and national information repositories. In other words, practitioners need an understanding of the tangible purpose behind reporting certain indicators and understand how results and policy decisions are influenced by this information.
  • Facilitate information-sharing among agencies and create linkages. Opportunities for linkages with other agencies in the DWI system are worthwhile to consider and pursue. The creation of information-sharing protocols or processes can establish mutually beneficial relationships and provide greater context for various performance measures. Information-sharing also provides a broader picture of performance throughout the system as a whole and offers the ability to illustrate how performance in one facet can have an impact on another facet of the system.
  • Use performance measurement to cut costs and strategically allocate resources. The use of performance measures is integral to inform decision-making and, as such, agencies should endeavor to take them into consideration when making budgetary decisions in relation to DWI countermeasures, interventions, and policies. For example, resources can be wasted on ineffective programs, policies, or practices, or invested too heavily in one initiative at the expense of others. Performance measurement can provide agencies with an indication of where resources should be allocated to achieve the best outcomes.
  • Link policy to outcomes. In a similar vein to resource allocation, performance measures can be used to develop targeted DWI policy. The justification for decisions, particularly at a policy level, should be informed and driven by data. Performance measures help determine whether agencies are meeting their stated goals and objectives while also providing insight into areas for improvement. Policy can be created or modified to address shortcomings or gaps that are identified through performance measurement.
  • Consider performance measures during the development of data systems. Performance measures must be considered when automated data management systems are being designed; add-ons to include performance measures after the fact are costly. The discussion about performance measures and evaluation reports should occur during the development of the automated system functional standards so issues can be addressed at the outset of new system development or during an update phase.