Introduction 

One of the big questions that face a development program is "How effective are our programs?". This question is often driven by the marketing team in response to client's queries or by the programs team looking to improve their programs. Program evaluation for many has a reputation of being a black art, often in the realm of academics and statisticians, out of the reach of practitioners. As a result many schools struggle with the concept and often don't benefit from the value of a well structured evaluation process.  This article aims to give an overview of the steps involve in setting up an effective evaluation process and hopefully encourage more schools to look closely at their programs.

What do we want to measure?

Putting down in words what happens through an intense experience can be hard to articulate. The aims of the program are an excellent place to start. Some of the common aims are - developing teamwork or leadership, encouraging self confidence and awareness or improving initiative.  Custom programs should look to client needs when defining possible measurements. Some programs might have multiple client groups with differing needs. For example a school program has the parents, the students and the school itself as a client.

How do we measure this?

Establishing the dimensions and the method of measurement is probably the hardest part of the process. Dimensions need to be established and a scale developed. A decision needs to be made whether to measure quantitatively or qualitatively. Both methods have advantages and disadvantages. This is where assistance from professional academics can help in establishing a valid instrument with well worded questions. An instrument that has been well constructed is important in getting results that reflect the outcomes of the programs in an unbiased fashion.

How do we make it easy to manage?

While a 100 question instrument may give you an excellent insight into the workings of your program, getting it filled in correctly and the data entry will prove a nightmare. Web based questionnaires have made it easier to collect information for groups that are spread out, but rely on having appropriate access to technology. It has the added benefit of completing the data entry at the same time. A paper based evaluation system can be effective in smaller measurements or with groups without access to technology.

Collecting evaluations on the last day of course makes it easier to get a more complete sample size. Leaving it to the participants to send it back after the course can result in having only 20% of the data returned.

Data entry needs to happen as soon as possible to make use of the information while it is fresh. Processing the data a year after the program may be fine for a Thesis, but presents little value to the program designers in program improvement. Having a system that processes the data quickly and easily is better than manually calculating it each time. It also reduces the chance of errors in calculations.

How do we present the data?

A well designed report that presents the findings from the data is an easily understood format is very important.  The measure needs to be explained as well as a benchmark for reference, so that those reading the report know if the score is better than average, excellent or not. A visual representation of the results can make it easier to understand and see at a glance the results of the evaluation.

How do we actually make a difference to the program from our results?

This is the most important part of the process, feeding back the results into the design process. Specific activities can be mapped to the dimensions that are measured. Looking at how much those dimensions have changed can lead to changing the emphasis on certain activities or processes in the course. This feedback has to happen soon after the program. Ideally a program would finish, the data entered and processed, a report produced and discussed the course debrief. Ideas on how to improve the program can be planned for the next program. This can give guidance to the experiments in program design, and a chance to evaluate its effectiveness.

This represents a very brief overview of the process of program evaluation. While we are often very good helping our participants learn from their experiences, we need to focus on learning from our own experiences in running the programs to improve them. There are many excellent examples of this happening on programs around the world. Sharing ideas between schools can help improve all of our programs.