Evaluation Tools to Guide Outcome-Based Thinking and Decisionmaking
Over the past decade, “outcomes” has gone from a buzzword in the nonprofit, government, and foundation worlds to a full-fledged movement. As the outcomes movement and outcome-based decisionmaking have grown, many models and frameworks for applying this thinking have emerged.
Though evaluators and practitioners have benefited greatly from the development of various tools to guide outcomes thinking, understanding the unique advantages of each model and how to select the right one can be challenging. Responding to this challenge, the Rensselaerville Institute for Outcomes published Outcome Frameworks: An Overview for Practitioners in 2004. This book offers insights into which model tool might be appropriate for the particular needs of a program at a given time. The tools described in Outcome Frameworks fall into three main categories:
- Program planning and management: Program planning and management tools are outcome models that support an effort’s proposal, funding, and implementation phases. They illustrate the logic, theory of change, and anticipated flow of an intervention, providing markers against which both incremental and ultimate progress may be measured.
- Program and resource alignment: Program and resource alignment tools ensure that resources and efforts are expended in support of organizational goals.
- Program reporting: Program reporting tools allow organizations to capture and communicate the fullness of the results they have achieved.
The matrix below provides an overview of major outcome-based tools in use, with information about which one might be appropriate to the particular needs of a program at a given time and how they may be used in an evaluation. Understanding the unique advantages of each tool and what it is well-suited for will help state leaders and evaluators determine which tool to use in evaluations.
Table 1. Overview of Evaluation Tools for State and Territory Leaders[a]
Program Planning and Management
Logic Model: diagram representation of a program showing what it is supposed to do, with whom, and why
| Key Concepts | Strong Points | Uses |
|---|---|---|
| Inputs, outputs, outcomes; arrows show relationships between elements in the model | Easy to use; provides an easily understood representation of a program’s theory of change | Program overview; presentations; program and evaluation planning |
Outcome funding framework: key management focus on the achievement of specific, sequential results for customers of services; emphasis on results, not activity
| Key Concepts | Strong Points | Uses |
|---|---|---|
| Investor return, results, customers, milestones, performance targets, outcome statements | Highly disciplined approach that serves both program investors and implementers; web-based software has strengthened usability | Government and philanthropic grantmaking; program and organization management |
Results-based accountability: real-time approach that describes what desired results look like, defines results in measurable terms, and uses measures to drive action plans for improvement
| Key Concepts | Strong Points | Uses |
|---|---|---|
| Results, experience, indicators, baselines, strategy, action plan and budget, accountability | Thorough system for planning community-change efforts and improvements in program, agency, or system performance; uses lay language; provides direct link to budgeting; useful for integrating different outcome systems | Project planning and start-up; development of community report cards; program and agency improvement plans and budgets; grantmaking and evaluation design |
Targeting outcomes of programs: tracking progress toward achievement of targets; evaluating degree to which programs impact targeted conditions
| Key Concepts | Strong Points | Uses |
|---|---|---|
| Knowledge, attitude, skills, aspiration; process, outcome, and impact evaluation | Fairly easy to use; helps integrate program development and evaluation; implementers and managers can use same concepts | Program design and evaluation |
Collective-impact evaluation[b]: not focused on assessing programmatic impact but on complex initiatives working in complex environments where progress is not linear or predictable.
| Key Concepts | Strong Points | Uses |
|---|---|---|
| Nine propositions to help evaluators navigate the unique characteristics of complex systems, improve their practice, and serve the social sector | Seeks to understand and describe the whole system; focuses attention on context and being responsive to changes as they occur | Evaluating systems initiatives and complex, multifaceted efforts |
Program and Resource Alignment
Balanced scorecard: business-based model designed to provide integrated management and accounting for multiple variables impacting organization performance by connecting them to a set of performance indicators
| Key Concepts | Strong Points | Uses |
|---|---|---|
| Strategy, alignment, short- and long-term objectives, financial and nonfinancial measures, lagging and leading indicators, performance measures and drivers, internal and external indices of success | Allows for a graphic assessment of the degree to which an organization’s resources and efforts support its goals | Monitoring either a single program with several associated initiatives or multiple programs within an organization; analyzing alignment of resources and initiatives to strategic targets |
Program Reporting
Scales and ladders: graphic tool that centers around a series of scales and their placement within a matrix designed to illustrate progress along a continuum of stages
| Key Concepts | Strong Points | Uses |
|---|---|---|
| Scales; mutually exclusive, multiple, and floating indicators | Places a client, community, or program on a continuum; shows incremental and relative progress, stabilization, or decline; individual data together tell a complete story; behaviorally anchored description of levels of change | Demonstrating aggregate progress; measuring concepts that are not easily quantified |
Results mapping: outcome-based evaluation tool designed to systematically capture otherwise nonquantifiable anecdotal evidence
| Key Concepts | Strong Points | Uses |
|---|---|---|
| Causal and synchronistic attribution; levels and milestones | Way to systemize, standardize, gather, and use lessons embedded in anecdotal information | Turning anecdotal information into a useful tool for program presentation, evaluation, and assessment |
Program results story: uses stories to capture organizations’ achievements and present them in a results-based format
| Key Concepts | Strong Points | Uses |
|---|---|---|
| Results, stories, anecdotal evidence | Easily understood approach for presenting results; brings outcomes to human-interest level; captures and conveys richness of information | Presenting program and results to multiple audiences |
[a] Penna, R. Elena, & Phillips, W.. (2005). Outcome Models. The Evaluation Exchange: Evaluation Methodology: Promising Practices. Newsletter, Volume XI, 2, p.5. Retrieved from https://www.alnap.org/system/files/content/resource/files/main/evaluation-exchange.pdf
[b] Preskill, H., & Gopal, S. (2014). Evaluating complexity: Propositions for improving practice. Boston: FSG.