| Main wiki page |
Recent additions |
Recent changes |
What links here |
Categories |
Category cloud How-to guides | Organisation profiles | Project profiles | ||||||||||
BP55: Theme Editorial - Monitoring and Evaluation by Wendy Annecke
Through its focus on Monitoring and Evaluation, this issue of Boiling Point offers some resolution to the questions, challenges and skills that we seek. There is sage advice and insights from experienced M&E practitioners, a useful toolkit and articles, arguments and guidelines to consider. There is also an innovative feature aimed at including a wider body of readers and expertise: A case study scenario requiring the development of an M&E system was presented to a number of international experts to comment upon, and the rich results reveal how differently M&E may be approached according to the implementer’s perspective and on-the-ground conditions. While the scenario was fairly formidable, there was ready reassurance from the experts that it could be broken down into ‘manageable chunks’ and that assistance in the form of guides and toolkits is readily available. Jonathan Rouse, Dana Charron and David Pennise, for example, advise on a range of instruments that can be used to measure technical and socio-economic changes and raise the issue of unexpected impacts. M&E has become a central element of the work of the Gaia Association who note in their response to the scenario that ‘by evaluating the impact of our work, we build confidence with our donors and most importantly, we maintain a dialogue with our target communities to ensure that we continue to serve their needs as best we can’. Although this issue focuses on improved stoves, M&E is of course crucial to all development projects. Risø’s Development and Energy in Africa (DEA) project (see the GVEP International news pages) illustrates the usefulness of M&E to rural electrification through solar and grid connections, solar water pumping, renewable energy for women and sustainable forestry. Kavita Rai (2005) developed an M&E kit specifically for renewable energy programmes, while my article discusses evaluation in the context of an electricity-to-LPG transition. In this edition although each article approaches M&E from a different angle, most of the contributions and the responses to the scenario agree on a few critical components of systematic monitoring and evaluation. The first is to have the objectives of the project and each of the stakeholders clear and aligned. Secondly baseline information is necessary in order to have a standard against which to measure change. Many projects begin without such knowledge and have to reconstruct it after the intervention (which is never entirely satisfactory!). The question of control groups, which may be useful with regard to accuracy in measuring change and making comparisons, is raised by Dutta and Jagoe, Bromley and Bruce as an issue of ethics as well as expense, and deserves further discussion. Thirdly the systematic collection of data; having a plan, allocating tasks and taking responsibility for analysing and reporting is emphasised. Data collection is an art in itself. Should it be qualitative or quantitative? How much, how often and how large a sample size? These are questions that plague evaluators. Some years ago M&E manuals gave the impression that the only way to measure outcomes was in numbers: percentages of reductions (in say wood users) or increases in the number of improved stoves manufactured. Patton (2002:13) suggests that a pragmatic approach generally works best: if the quantities of wood or electricity households are using has to be measured, use a scale or a meter. If you want to know the calorific value of the wood versus the electricity, perform the necessary laboratory calculations, if you want to know what using wood and/or electricity meansto households, how it affects them how they think about it and what they do about it, ask questions, listen to their stories, find out about the conditions and experiences. What has become clear over time is that numbers and stories may be equally subject to distortion and interpretation – neither is inherently objective, both can be usefully collected to test and prove a point. As Karabi Dutta explains in her article in this issue, Monitoring and Evaluation: Experiences from the field, the end result of an intervention is as much a function of user preferences and behaviour as the technical design of the improved cook stove. Collecting information and writing reports should not be an end in itself. The purpose of M&E is as a tool for communicating what is happening in the project and, if necessary, deciding how to change it. The challenge is often to know to whom the information should be communicated. Different stakeholders have different interests in the project, and the power to alter plans, budgets or even to acknowledge that the project has not achieved its objectives, does not belong equally to all stakeholders. In addition energy for poverty reduction is rarely integrated into development policies, in part because it has been difficult to provide concrete evidence of the effect of access to modern energy. Monitoring and evaluating reports may provide such evidence, but to date what is lacking in our work is how to use monitoring and impact studies to fulfil their purpose. Even with improved M&E techniques, energy projects do not have impressive track records for being sustainable nor, ultimately, for reducing energy poverty at the required scale to meet the MDGs. But progress is being made and the articles in this issue demonstrate improved methods for reflecting on and doing monitoring. One of the fields in which there is growing experience is in monitoring efficacy and emissions from improved cook stoves. In So You Finally Bought a Combustion Analyser! Crispin Pemberton-Pigott from Swaziland takes the reader through the steps necessary to use a combustion analyser to its best advantage, while Ilse Ruiz-Mercado, Nick Lam, Eduardo Canuz, Gilberto Davila and Kirk Smith, in their article Low cost temperature loggers as stove use monitors (SUMs), introduce small, rugged, commercially available equipment that could be of significant interest to those involved with stove M&E. One method that has been designed for use in monitoring development projects is Results Based Management (RBM), a key element of which is the results chain -a causal sequence for an intervention that stipulates the necessary steps to achieve the desired objectives. This issue’s toolkit, Six steps to Results Based Monitoring, provides readers with useful advice on how to set up an RBM system by identifying all stakeholders, formulating the assumptions on which strategy is based, analysing risks and side effects, choosing observation fields, specifying indicators for measurement and finally implementation. Expanding on the theory of RBM is Verena Brinkmann’s article on the application of the method, Results based monitoring in GTZ cooking energy interventions: A Burden or a Benefit? Here she highlights some of GTZ’s experiences with the system, including positives and negatives, in a number of their stove programmes. In their thoughtful article, Monitoring and evaluation of health and socio-economic impacts: Key lessons learnt from the Household Energy and Health Project, Kirstie Jagoe, Helen Bromley and Nigel Bruce advance the principle of training and collaboration in country teams as one of their key findings. The participants in the DEA workshop in Tanzania (see the GVEP International news pages) also identified building capacity to conduct monitoring and evaluation in the energy sector as a priority. A feasibility study has been conducted and the development of M&E skills will be one of the functions of the M&E facility to be established in Southern Africa with support from GVEP International and other partners. But M&E should not end with project staff. There is a further challenge that Stephen Gitonga raises in his response to the scenario in this issue: M&E capacity should be developed among all stakeholders including the participants. The kind of capacity that would be valuable to communities would include understanding the demands and expectations of the implementers, and the ability to establish channels of communication with the implementers and funders during the planning and monitoring of the project. Ideally participants should develop their own set of targets to be monitored during the project and should define resulting downstream impacts. These deliberations would provide the basis for reflection on the project itself, and the chance for the participants to answer a key question: was the project well designed to alleviate poverty as they experience it? This may be a time consuming exercise: opportunities for such learning have to be created and the time and resources to do so are seldom available but such a process would contribute towards development and ownership of development in ways which current processes often do not (Annecke 2008). This is a topic we could take up in a special interest group. [top] [end]Notes and references
[top] [end]@HEDON
[top] [end]Download the original article Theme Editorial - Monitoring and Evaluation by Wendy Annecke (140 KB)[top] [end]Contents: Boiling Point 55 - Monitoring and Evaluation
Categories: Boiling Point 55| Monitoring and Evaluation | ||||||||||
Page created:
23 June 2008; Last edited:
01 October 2008; Version: 6 | ||||||||||
Pagename: BP55:ThemeEditorial @HEDON: WTJA | ||||||||||
