Evaluation and Improvement

Evaluations require money. Money spent on evaluation is an investment in your program and its participants. Do not think of money spent on evaluation as a diversion of funds that could be available for participants. Evaluation is essential if you want to know whether your program is benefiting participants.

It is challenging to specify exactly how much your evaluation will cost because of considerable variation in evaluation purpose and type. Some of these varying factors may include what aspects of your program you decide to evaluate, the size of the program (that is, the number of staff members, participants, components, and services), the number of outcomes you want to assess, who conducts the evaluation, and your agency’s available evaluation-related resources. Costs also vary depending on economic differences in communities and geographic locations.

Sometimes funders establish a specific amount of grant money to be set aside for an evaluation. As a rule of thumb, estimated costs should range from 15 to 20 percent of the total funds allocated for the program. If the amount of money to be set aside for an evaluation is not specified by a funding agency, you may want to talk with other state leaders or partners in your community who have conducted evaluations. They may be able to help you estimate evaluation costs.

Here are some general guidelines to help you think about what information you may be able to get at different evaluation cost levels:

  • Lowest-cost evaluations: If you spend only a minimal amount of money, you will be able to obtain numerical counts of participants, services, or products and information about participant characteristics. You also may be able to find out how satisfied participants are with services or training. However, this is only the foundation for an evaluation. This information will not tell you whether you have been successful in attaining your participant-outcome objectives. Also, at this cost level, you will not have the in-depth information about program implementation and operations needed to understand whether your program was implemented as intended and, if not, what changes were made and why they were made.
  • Low-moderate-cost evaluations: If you increase your evaluation budget slightly, you will be able to assess whether there has been a change in your participants’ knowledge, attitudes, or behaviors and collect in-depth information about your program’s implementation. However, this is only the framework of an evaluation. At this cost level, you may not be able to attribute participant changes specifically to your program because you will not have similar information on a comparison or control group.
  • Moderate-high-cost evaluations: Adding more money to your evaluation budget will allow you to use a comparison or control group and therefore attribute any changes in participants to the program itself. At this cost level, however, your information on participant outcomes may be limited to short-term changes—that is, those that occurred during or immediately after participation in the program.
  • Highest-cost evaluations: At the highest cost level, you will be able to obtain all the information available from the other cost options as well as longer-term outcome information on program participants. The high cost of this type of evaluation is due to the necessity of tracking or contacting program participants after they have left the program. Although follow-up activities are often expensive, longer-term outcome information is important because it assesses whether the changes in knowledge, attitudes, or behaviors that your participants experienced initially are maintained over time.

As you increase your evaluation budget, you gain a corresponding increase in knowledge about your success in attaining program objectives. In many situations, the lowest-cost evaluations may not be worth the expense, and, realistically, the highest-cost evaluations may be beyond the scope of most agencies’ financial resources. As a general rule, the more money you are willing to invest in an evaluation, the more useful the information obtained about your program’s effectiveness will be, and the more useful these results will be in helping you advocate for your program.[6]

 

[6] Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. (2010). The program manager’s guide to evaluation (2nd ed.). Washington, DC: Author.