Which group is most commonly the primary audience for a process evaluation?

A process evaluation focuses on the implementation process and attempts to determine how successfully the project followed the strategy laid out in the logic model. As opposed to outcome or impact evaluations, a process evaluation focuses on the first three segments of the logic model (inputs, activities, and outputs) and how they work together. Process evaluations allow evaluators to make the important distinction between implementation failure and theory failure. Implementation failure is the lack of expected results due to poor implementation practices, such as unmet targets due to an insufficient number of trained community health workers or breakdowns in transportation of medication to the clinic. Theory failure is when program activities are implemented to the standards of the program design strategy but expected outcomes are not found, meaning the theory that linked the activities to expected outcomes is incorrect. For instance, the proper implementation of caretaker training on oral rehydration salt administration, with the aim of reducing patient acute diarrhea, coupled with unchanged incidence of acute diarrhea, would be classified as theory failure.

Although traditionally emphasis has been placed on impact and outcome evaluations because of their importance in the funding streams of NGOs and development agencies, there are significant benefits to conducting a process evaluation. As a checkpoint for program implementation, process evaluation ensures that the program is delivered according to design. If any standards are not met, findings at this stage of evaluation can save subsequent time and funding. In addition, a process evaluation creates a feedback loop by including routine assessments such as a documentation of resources used, measurement of output indicators, and tracking of project reach among the target population. Because these activities are done routinely throughout the timeline of the project, data collection and analysis may reveal early challenges of the program. This allows implementers to alter program activities accordingly and, hopefully, improve the chances of positive outcomes. Lastly, process evaluation allows evaluators and program developers to pinpoint strengths and weaknesses within the program design and improve upon the program in future scale-up efforts.

Measuring every detail of program implementation is not feasible or necessary, especially in large-scale projects. Process indicators should be developed based on the priorities of the stakeholders, clients, and evaluators, while relating back to the original evaluation question and the key processes highlighted in the logic model. For example, if the program outcomes are highly dependent upon the establishment of a supply chain, it may be beneficial for evaluators to appraise the implementation of the supply chain. In this framework, indicators may include timeliness, utilization, and frequency of stock outs in the clinic.

For the purpose of this explanation, it will be beneficial to use an example of a pre-existing logic model. This example is a World Health Organization (WHO) logic model for a program seeking to improve the nutrition of the target audience in relation to the Millennium Development Goals. (Source, WHO, 2011): http://www.who.int/vmnis/toolkit/WHO-CDC_Logic_Model_en.pdf.

Measurement in Process Evaluation

Developing Process Evaluation Indicators

There are four general concepts that should be measured for a process evaluation: the types of inputs, activities, and outputs, and the integration of these components. For each target, evaluators will develop a number of indicators. Indicators are used as measurement guidelines, flowing directly from the logic model.

To demonstrate the process of developing indicators for process evaluation, it is helpful to use a sample logic model (refer to the image above and for further context follow the link). In this example, the inputs and outputs are clearly labeled, and one  of the ‘Delivery’ activities is to develop a delivery system. It can be assumed that many of the inputs listed will be required to run this activity, such as financial resources, management, and staff. The expected output of this activity is “access to or presence of the intervention in the target communities or facilities.” As such, a basic indicator to measure the success of the output would be a binary (yes or no) indicator for the existence of an intervention in the target community, at a time specified in the program timeline. This is a quality indicator, meeting all of the requirements in specificity, measurability, attainability, relevance, and timeliness. The existence of a functional delivery system also serves as a process indicator. If evaluators are interested in whether the system of delivery is developed according to the program timeline or human personnel are properly trained, corresponding indicators would include “timeliness of system completion” and “ratio of staff citing preparedness to total number of staff trained.” Again, most of process evaluation planning depends on the stakeholder and client priorities which, ideally, are discussed prior to the planning of the evaluation.

Determining Baseline

Once process indicators have been defined, evaluators may collect data prior to the start of the intervention to determine the baseline for each indicator. This data will show the current situation in the area of interest (e.g. current existence of delivery systems, quality of supply chains, number of trained staff). This is important because it allows for the program staff to create realistic target indicators. Also, measuring indicators only after the intervention has been completed does not allow evaluators to show what the program activities accomplished, reducing the likelihood of scale-up or continued funding. For more information on baseline and pre-intervention measurements, refer to the Evaluation Study Designs module.

Target Indicators

After all data on relevant indicators have been collected post-intervention, evaluators must compare the values to a target, goal, or standard indicator in order to give meaning to evaluation. Additionally, it is important for the effectiveness of the model that evaluators have distinct experiences and perspectives and are familiar with the context of the community in which the intervention is taking place. Target indicators are established in coordination with the program development team, depending on baseline indicator information. A common oversight in this step is to set indicator targets without referring first to baseline information. When this happens, extremely high or low indicator targets may overemphasize the quality of the program, creating misleading information. The program development team may have pre-determined targets for activities and outputs required for the program to be considered successful. Target indicators can be used as comparison tools throughout the monitoring and evaluation process for mid- and post-intervention measurements. This gives the evaluators an opportunity to convey progress to clients and stakeholders throughout the implementation period.

Footnotes

Weiss, C.H. (1997). Theory-based evaluation: Past, present, and future. New Directions for Evaluation, 76:41-55.

Saunders, R.P., Evans, M.H., Joshi, P. (2005). Developing a process-evaluation plan for assessing health promotion program implementation: A how-to guide. Health Promotion Practice, 6(2):134-147.

Issel, L.M. (2009). Health program planning and evaluation: A practical, systematic approach for community health (2nd ed.)  Sudbury, MA: Jones and Bartlett Publishers.

Bamberger, M., Rugh, J., and Mabry, L. (2006). Real world evaluation: Working under budget, time, data, and political constraints. Thousand Oaks, CA: Sage Publications, Inc.

Which type of method is commonly used in process evaluations?

What are the four primary types of program evaluation?

What are process evaluations used for?

What are the main types of evaluation?