Maurice Samuels, Senior Evaluation Officer, discusses how selecting measures of impact, context, and outcomes for evaluation reflects values and creates accountability.
One of the ways in which we live our values of informed and inclusive decision making is through measurement. It is important in helping us to learn about the “so what” of our work. It gives us insights to determine if we are on track to reach our intended outcomes and impacts, understand what is working, and identify external factors that are hindering progress.
Measurement is an important tool in helping us to assess the progress of a strategy; it is equally important in helping us rearrange power dynamics and recognize our privilege. Because we use measurement for learning, it inherently holds us accountable for the changes we seek and the degree to which we are living our values. Measurement and accountability proceed arm-in-arm as they are needed for us to make the most informed decisions.
We do not develop the measures in a vacuum but customize our metrics to a strategy. Our process is inclusive, using feedback loops to include diverse perspectives. We intentionally call on the expertise of our grantees, internal and external advisors, Board, and evaluation and learning partners to select measures that are reliable and realistic.
To accomplish this, we ask ourselves several questions in the process of deciding what to measure, such as:
Each evaluation has outcome, impact, and contextual measures. Outcome measures are used for information on the work on-the-ground and close to our funding and support. Impact measures inform us about the ultimate contributions of our work. We know there are external factors that can hinder progress, so we select a handful of measures that inform us about the context we are working in. We follow each from baseline to the end of a strategy, though some may can change based on evolving circumstances.
Outcome measures have given us information on jail populations: admissions, racial and ethnic disparities, length of stay, and flow into prisons. We compare grantee sites to themselves overtime, to each other, and to similar jurisdictions overtime.
For impact measures, we are following national jail trends, the national conversation about jails, and how criminal justice reform is spreading across the country. We use a variety of methods—such as polling, media scans, and surveys—to determine if implemented reforms are reducing jail populations, particularly for overrepresented populations, like Black, Native Americans, and Brown people.
And we follow contextual measures, such as public safety and county expenditures, to give us the information we need to identify the consequence of changes to criminal justice systems at our sites and nationally.
We know that the measures selected will not be perfect and have limitations, so we revisit a them periodically to determine if it is providing us useful and actionable information. This increases the likelihood that the decisions we reach are well informed and align with our values and goals.