Interest in the evaluation of international humanitarian interventions has never been higher, but has the dramatic growth of the evaluation community led to improved learning, performance or accountability? How are evaluations undertaken? Are the right questions being asked? What influences their findings? What makes for a good evaluation?

A book from the Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP) provides reflections from consultants and academics who have been involved in some of the major humanitarian evaluations of recent years. Case studies are drawn from four continents, embrace conflict-induced and natural disasters, look at the context in which donors, the UN, the Red Cross and NGOs undertake evaluations and suggest how they might be improved.

The evaluations, described with candour and self-criticism, differ in scope but suggest a number of commonalities. Expectations of the outcome of evaluations are often exaggerated. They are often ‘quick-and-dirty’ and insufficiently funded. Terms of Reference (TORs) may be unrealistic, excessively long and fail to differentiate between project and policy level questions. Disproportionately male evaluation teams are hurriedly thrown together and given a rushed briefing before flying off to the field. Though team members rarely know each other prior to the mission, they generally rub along well, but can get into confrontational relationships with field and headquarter staff of commissioning agencies. Evaluators’ contacts with beneficiaries are often inadequate.

Methodology is generally given insufficient attention in the various forms that it is required. In particular, the methods used to get from field findings to recommendations are unclear and need more attention. Debriefing is often perfunctory and writing up rushed. Frequently it is only the team leader who prepares recommendations. Changes in the reports demanded by commissioning agencies may cast doubt on the extent to which evaluation reports are really ‘independent’. Unless evaluations get published there is a risk that agencies will forget them as new crises and priorities emerge and the pressures that led to the evaluations subside.

The book notes that:

• The personalities of team members, whether they are optimists or pessimists, often have a major bearing on findings.
• A well-resourced evaluation on the massive multi-disciplinary scale of the landmark 1996 joint evaluation of emergency assistance to Rwanda has not been repeated, perhaps due to the inter-agency tensions and UNHCR shortcomings it revealed.
• Where evaluation contracts are put out to tender, it is very difficult for evaluators to suggests changes to TORs.

Positive examples of good practice in evaluation are also highlighted. Among the key recommendations emerging from the case studies are:

• Evaluators should not have personal or professional interests in the findings, must avoid points scoring and also remember they are investigating juries rather than sentencing judges.
• Evaluation teams should be involved in preparing TORs and given scope to renegotiate them after initial assessments which should involve a preliminary field visit.
• Evaluators need to be better briefed and supplied with relevant documentation at the outset.
• Reports should be more specific about conceptual and methodological frameworks, how data was obtained and conclusions and recommendations reached.
• Evaluators should be allowed to look upwards at the practices of agency headquarters, not just outwards to the performance of their field staff.
• More evaluation results should be in the public domain and feed into follow-up publications, workshops and training programmes.

Source(s):
‘Evaluating international humanitarian action: reflections from practitioners’, Zed Books. Edited by Adrian Wood, Raymond Apthorpe and John Borton, 2001 Full document.

Funded by: AusAID

id21 Research Highlight: 27 June 2002

Further Information:
Kate Robertson
Deputy Co-ordinator
ALNAP at ODI
111 Westminster Bridge Road
London SE1 7JD
UK

Tel: +44 (0)20 7922 0305
Fax: +44 (0)20 7922 0399
Contact the contributor: k.robertson@odi.org.uk

Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP)

Other related links:
'Is humanitarian aid failing?'

'Aid without accountability? North Korea challenges humanitarian principles'

Reality of Aid is an independent review of development assistance

Among attempts to regulate humanitarian action in conflict zones are those of HAP

PARC supports efforts to improve performance assessment in development practice

The Red Cross Code of Conduct attempts to monitor quality and accountability

The Sphere Project also attempts to regulate humanitarian action in conflict zones