Introduction to Strategic Evaluation:

Section on Evaluation Approaches, Purposes, Methods and Designs

Contact:paul@parkerduignan.com

Evaluation is an activity undertaken within many different disciplines. In addition, evaluation can be conceputalised in a number of different ways. This has lead to a diversity of terms used to describe aspects of evaluation. For those attempting to find out about evaluation, and even in some discussions amongst evaluation professionals and their clients, there can be considerable confusion when people attempt to compare terms which relate to very different aspects of evaluation.

A useful way of looking at evaluation is to distinguish between approaches, purposes, methods and designs.  It can become particularly confusing if people having a discussion about evaluation are talking at different conceptual levels. For example, a discussion about what should be done in an evaluation where one participant is talking about an approach, e.g. empowerment evaluation; another about a purpose of evaluation, e.g. outcome evaluation; a third about a method – e.g. key informant interviews; and a fourth about a design, e.g. a quasi-experimental design.  

The following table lists some of the terminology used under the four headings: approaches, purposes, methods and designs:

Evaluation Approaches

 An evaluation approach is a general way of looking at or conceptualising evaluation, which often incorporates a philosophy and a set of values.

 The main evaluation approaches are:

• Utilisation-focused evaluation – determines methods on the basis of what is going to be most useful to different audiences (Patton 1986).

• Empowerment evaluation – emphasises that the evaluation process and methods should be empowering to those who are being evaluation (Fetterman, Kaftarian et al. 1996).

• Stakeholder evaluation – looks at the differing perspectives of all of a programme’s stakeholders (those who have an interest in it) (Greene 1988).

• Goal-free evaluation – in which the evaluator’s task is to examine all of the outcomes of a programme, not just its formal outcomes as identified in its objectives (Scriven 1972).

• Naturalistic or 4th generation evaluation – emphasises the qualitative uniqueness of programmes and is a reaction against the limitation of quantitative evaluation approaches (Guba and Lincoln 1989).

• Theory Based Evaluation – puts an emphasis on detailing the assumptions on which a programme is based (intervention logic) and follows those steps to see if they occur (Birckmayer and Weiss 2000).

• Kaupapa Māori research /evaluation – an evolving approach to research and evaluation which is based on Māori values, perspectives and research methods (Smith 2000; Moewaka Barnes In Press)

• Strategic evaluation – emphasises that evaluation design decisions should be driven by the strategic value of the information they will provide for solving social problems (Duignan 1997).

The least controversial approach which is currently in vogue, because it is pragmatically based, is Utilisation-focused evaluation – which is simply evaluation focused on meeting the needs of particular stakeholder groups.  

Evaluation as a discipline has not been immune from the immense debates in the social sciences over philosophy of science and values issues.  This means that evaluation approaches will have considerable differences in terms of what methods are deemed appropriate, and how it views the relationship between evaluators and those involved in the programmes and policies being evaluated.  This can lead to very interesting debates about what evaluation is appropriate in what settings.

There are various ways of describing various purposes of evaluation activity, e.g. design, developmental, formative, implementation, process, impact, outcome and summative. The evaluation purpose is best understood as identifying what evaluation activity is going to be used for. Recent years have seen evaluation move to develop types of evaluation that are of use right across a programme lifecycle. It should be noted that any particular evaluation activity can have more than one purpose.

 The range of evaluation terms are used in various ways in the evaluation literature. One way of defining them is as follows: 

• Design, developmental, formative, implementation – evaluative activity designed to improve the design, development, formation and implementation of a programme.

• Process – evaluation to describe the process of a programme. Because the term process could conceivably cover all of a programme from its inception to its outcomes, it is conceptually useful to limit the term process evaluation to activity describing the programme during the course of the programme, i.e. once it has been initially implemented.

• Impact, outcome and summative – looking at the impact and outcome of a programme, and (in the case of summative, making an overall evaluative judgment about the worth of a programme).

One useful way of distinguishing between the purposes of evaluation is to group them into the following three overall types: 

Formative Evaluation

Formative Evaluation includes the terms - design/developmental/formative/implementation evaluation – it is any evaluation activity directed at improving a programme's design, planning, development and implementation. Formative evaluation is a disciplined approach to ensuring that a programme is well developed. It has been developed relatively recently in the history of the evaluation profession (McClintock 1986). There are various models for how it can be undertaken, but it is directed at ensuring independent constructively critical input into programme development.  For instance, a separately funded independent formative evaluation team can work alongside programme planners.  This team critically assesses the decisions that are being made and can provide regular, formal feedback to programme planners and programme funders.  Formative evaluation may use both quantitative and qualitative techniques.

Process Evaluation

Process Evaluation is any evaluative activity directed at describing or documenting what actually happened in the contest or course of a programme.  Process evaluation is sometimes seen to include formative evaluation, but it is conceptually useful to separate formative evaluation out as a specific type of evaluation. Process evaluation can provide extremely useful information about what actually happened in a programme. It can be crucial for communicating best practice to others who want to replicate elements of a successful programme.  For instance, taking as an example a Programme X that has been shown to be effective after an outcome evaluation. Just knowing that Programme X is effective is, in itself, insufficient for someone elsewhere to replicate the programme. Process evaluation gives someone who wants to replicate a programme detailed information on what was done, what problems arose and the what solutions were adopted.  

A second use of process evaluation is in the interpretation of outcome evaluation results. For instance, a programme may not have proved successful on outcome evaluation. However, when looking at the process evaluation from this programme, it will be clear that the negative outcome was a result of specific events that derailed the programme. In the light of this one should not dismiss the possibility that this type of programme, if implemented as planned, could be effective.

A third use of process evaluation is when it is used to examine the context of a programme and the decision making leading up to that programme being introduced. For instance there may be “problem definition creep” in the early decision making phase about the programme objectives and what type of programme should be run. This may lead to a programme being designed which is "easy to implement" rather than one which is more difficult to implement but which is more likely to achieve success with the substantive problem being addressed (Duignan, 1989).

Outcome Evaluation

Outcome evaluation also includes the terms - intermediate outcome evaluation and impact evaluation - it is any evaluative activity directed at determining the positive or negative intermediate or longer-term outcomes of a programme.  It is sometimes also referred to as summative evaluation, which also includes the aspect of making an overall assessment of a programme. Outcome evaluation looks at whether a programme has achieved the outcomes it is seeking. Where this can be done, this is very useful information for stakeholders, particularly if it is in a form in which the effectiveness of the programme being evaluated can be compared with other strategies for achieving the same outcomes. In real world programmes the final outcomes being sought by a programme may take a number of years to achieve and may be outside the timeframe of the measurements being undertaken in an evaluation. Given this, there is a way in which evaluation designers can give outcome-type results earlier within time frames that are more useful for policy decision-making. This entails the development of an “outcome hierarchy” for a programme or policy. This is a set of outcomes that range from immediate outcomes of the programme or policy through to intermediate and then final outcomes. An argument needs to be mounted that each step in the outcome hierarchy is likely to imply that the next step will occur. If this argument is sound then intermediate outcomes can be measured at an earlier level, within reasonable time frames, and the assumption made that there is a good chance that the later steps in the outcome hierarchy will also take place in due course.

In the past, outcome evaluation in a number of sectors, has tended to be largely quantitative. This notion is now being challenged and there is discussion of the argument for using qualitative outcome evaluation in addition to the use of qualitative evaluation for other evaluation purposes (formative and process) (Mohr, 1999).

The three purposes for evaluation can be related to three stages in the programme life-cycle, the start, middle and finish of the programme.

Diagram 8 illustrates the relationship of different purposes of evaluation to programme stages.  

 Diagram 8: Evaluation “Purposes” or "Types"

These three purposes of evaluation are obviously linked – information arising out of a formative evaluation can be used for both improving a programme and also as part of process evaluation (documenting what happened in a programme).  Formative, process and outcome evaluation from one programme can be used to feed into the formative evaluation of a new programme.  

Evaluation Methods

In addition to evaluation purposes there are evaluation methods. These are the specific research and related methods which evaluators use in their day-to-day work. Evaluators will draw on any method that can assist in answering the questions that are being asked in an evaluation.  In the past these tended to be methods originating from the physical sciences, but now they also draw extensively on methods developed in the social and organisational sciences and the humanities. The main evaluation methods include: 

• consultation

• literature review (prospective evaluation synthesis)

• evaluative review of lessons from other existing programmes

• evaluative goal and objective setting critique

• evaluative implementation (programme) logic critique

• formative evaluation workshops for centrally funded community programmes

• evaluation hui

• stakeholder workshops

• pretesting for programme resource development

• piloting

• archival, administrative /routine records collection

• evaluation specific records collection

• observation and environmental audit

• participant observation

• photos, video and audio

• document analysis

• interviews: key informant / participant

• surveys, questionnaires, feedback sheets

• focus groups

These methods of collecting data can then be subject to various methods of analysis: 

• expert assessment

• statistical analysis

• qualitative analysis

• economic evaluation.

Evaluation designs are the way in which the evaluation ingredients – approach, purposes and methods – are put together into the final evaluation in an attempt to answer a set of evaluation questions. Evaluation design will indicate the overall evaluation approach, the mix of formative, process and outcome evaluation it is planned to carry out in the evaluation and the methods and analysis to be used. If there is to be an outcome evaluation within the evaluation, the design will specify which of the many types of outcome design are to be used.  

There are many different types of outcome design which can be used and which often require considerable statistical and other analytical sophistication to deal with. Designs include: non-intervention control group design with pre-test and post-test; non-equivalent dependent variables design; removed-intervention design with pre-test and post-test; repeated intervention design; reversed-intervention non-equivalent control group design with pre-test and post-test; cohort designs in formal and informal institutions with cyclical turnover; post-test-only design with predicted higher-order interaction; and regression-discontinuity design. The classic reference to these designs is Cook and Campbell (1979).  

Version 2.1 21-1-02