| Main knowledge bank page |
Recent additions |
Recent changes |
What links here |
Categories |
Category cloud How-to guides | Organisation profiles | Project profiles | ||||||||||||||||||||||||
M&E of health and socio-economic impacts: Key lessons learnt from a Project by K. Jagoe et al.
The Liverpool team were responsible for the health, social and time-activity impacts evaluation using quantitative and qualitative methods, while the UCB team focused on the effects on indoor air quality and stove performance. The ultimate aim was for the Liverpool and UCB teams to use the experience gained from this work to develop and make widely available, standardised guidelines and protocols for the monitoring and evaluation of household energy programmes. The findings from the evaluation studies have been reported to the Shell Foundation, and are currently being prepared for publication. In this article, the team from the University of Liverpool discusses the key lessons learnt and issues raised during this project (See Box 1). This is followed by a brief introduction to recently developed guidelines designed to address these issues and to assist organisations in planning evaluation studies, their role in the work, and in acquiring the skills, knowledge and tools to evaluate the impacts of household energy programmes.
[top] [end]What is the place of ‘off the shelf’ evaluation packages?For the health and socio economic components of the evaluation of household energy interventions, the goal of developing a standard ‘off the shelf’ impact evaluation package seems to be neither appropriate nor realistic - for two main reasons.Firstly, examining the impacts of a household energy intervention on health, women’s lives, environment and income generation, etc., is a complex task that requires an in-depth knowledge of the community involved, including features such as culture, climate and environment. This creates a situation where, for example, a questionnaire that has been developed for use in rural India is unlikely to be useful in that form in an African or Latin American country until it has undergone considerable adaptation to the local setting. In this project, adaptations had to be made to questionnaires and focus group discussion (FGD) topic guides when they were used at different sites even within rural India to allow for differences in, for example, fuel types and the nature of food cooked each day. Secondly, the aims of the organisation carrying out the monitoring and evaluation are typically diverse and require very different individual projects in order to achieve them (See Box 2). A standardised package would not provide the flexibility to allow for this range of aims.
However, it is also very important to use tried and tested methods where possible, and that there is also merit in retaining common features across studies where relevant to help with comparing the results of work in different countries. Therefore the study design should seek a balance between local relevance and the collection of data that is reliable and comparable with work carried out elsewhere. [top] [end]What role should the development organisation have in the evaluation?Development organisations involved in household energy work vary greatly in their aspirations and monitoring and evaluation expertise. However with realistic aims, appropriate skills as well as support with monitoring and evaluation, the majority can make an important contribution to the implementation of useful evaluation studies.Unless the development organisation has extensive experience carrying out monitoring and evaluation they should seek to establish collaboration with a team, ideally in their own country, who have the appropriate experience and expertise. There are many advantages to collaboration with an organisation from the country where the study is taking place including, having a good understanding of the issues that influence household energy in that area as well as ease of access to the study site thereby keeping costs to a minimum. The benefits associated with the development organisation taking a lead role in their own evaluation work (with research and planning support), as opposed to an outside research organisation carrying out the whole project, are related to the privileged relationship many development organisations have established with the communities that they work in. This allows them a level of access to homes, people and information that others may not be permitted (Figure 1). This relationship can work either way of course, since the relationship could also be a barrier to hearing people’s true feelings or responses may unduly reflect messages that have been a core element of the development organisation’s own promotional activity. However, the possible problems associated with this close relationship should not obscure the potential it offers for trust and honesty,norlead to an assumption that an outside organisation would always achieve greater objectivity and accuracy in data collection. [top] [end]Evaluation study planning must reflect progress with intervention developmentWhat the development organisation hopes to achieve from their evaluation should be strongly determined by the current stage of development of the intervention, and the approach to delivery and adoption. There is potential for wasting valuable resources and time, as well as the goodwill of the communities involved, if an extensive evaluation is carried out only to find that the adopters are not using the intervention or have adapted it so radically to suit their needs that the intervention no longer does was it was designed to do. Therefore detailed evaluation studies should not be carried out until there is good evidence to show that the intervention meets the needs of a majority of prospective purchasers, and that they will be able to use it in the manner intended for reducing indoor air pollution and improving fuel efficiency. As part of the initial planning of evaluation studies, it is important to assess what is known – and not known -about the acceptability and use in practice of the intervention. By way of example, Box 3 sets out the stages of development and evaluation that should have been completed before undertaking a substantive evaluation of health, social and economic impacts of a medium to large scale sustainable stove programme.
[top] [end]Issues in the design of evaluation studiesPossibly one of the greatest challenges when building an evaluation study around a household energy development programme is to align the design and timing of the evaluation work to the timescale and geographical spread of the intervention adoption process.This requires careful planning at an early stage.
The randomised allocation of homes to an intervention such as new stoves (Figure 2) and control (continued use of traditional stoves) is, from a scientific perspective, the most powerful method for studying the impacts of the new stove, but adds another problematic dimension. The disadvantage is that randomisation typically has to be very actively and closely managed, so that in practice it is very difficult to align this study design with the goal of evaluating a market-based programme, where adoption occurs (over time and geography) in a manner determined by the market and various other factors, such as promotional activity and credit facilities, designed to stimulate that market. When planning an evaluation study, it is very tempting to ‘arrange’ the initial adoption for the convenience of the study logistics, and probably to an extent this is inevitable. It is however an issue that needs to be considered carefully at the planning stage, with acknowledgments about the consequences of the resulting decision. Thus, on the one hand, the study needs to be feasible and practical within a reasonable budget and timescale, so it may not be realistic to relinquish completely control over the rate and location of adoption. On the other hand, ‘fixing’ the delivery, pricing, etc., of the intervention in such a way as to ensure the study is relatively easy to carry out may easily lead to the results having limited relevance to programmes where the goal is widespread adoption through financially sustainable mechanisms. [top] [end]Approaches to the evaluation of impacts on health and wellbeingThe mechanisms and pathways involved in the household energy impacts on health are wide ranging. These include clearly defined issues such as the effects of high levels of incomplete combustion products on the lungs, burns to young children from open fires, but also much less easily defined health consequences. An example of the latter would include the ways in which a cleaner, better lit environment for a family might increase opportunities for income generation and education, and thereby improve health in both the short and longer terms. As a result, different scientific perspectives that encompass epidemiological and qualitative research methods are useful in understanding these links and consequences. It was found useful to approach this apparently complex set of issues by considering the evaluation of health and wellbeing under four headings. These are shown in Table 1, together with a summary of the most appropriate research methods for each, and some of the implications for the expertise, costs and settings required.Table 1 Approaches to assessment of health outcomes of household energy interventions, methods and implications for evaluation studies
The linkages between the varied effects of household energy interventions on health are complex by virtue of the many inter-relationships, variable timescales and influences which work in both directions, and so too are the methodological considerations involved in designing studies to demonstrate such impacts. Although the evaluation of health and wellbeing is demanding, it is important that this topic is discussed thoroughly in the planning stages of the study, so that appropriate and realistic objectives are set, suitable techniques chosen, and sufficient resources identified. [top] [end]ConclusionsThe evaluation of household energy programmes is certainly a complex and demanding task, but also very important and worthwhile. Evaluation requires careful planning in advance, taking account of the stage of development of the technology and approaches to promoting adoption, consideration of the information requirements of prospective audiences, and of other factors including local and national trends in fuel availability and use, and policy on energy and development. Development organisations should be encouraged to consider what role they wish to adopt in the evaluation study - whether to take a lead role, take a facilitative role with a collaborating partner from a research group leading the work, or whether to commission the work to an external agency and manage the contract. For some, there is much that can be gained for taking a lead on the development and coordination of the study, particularly if there is a desire to develop capacity for future evaluation work.On the other hand, the demands and challenges involved must be recognised. The decision should be an informed one and arrangements should be in place for whatever level of research support is required, before embarking on the study. Finally, it is critically important that the evaluation work be appropriate to the stage of development of the intervention and only carried out with prior evidence of efficacy and acceptability. Larger-scale evaluation studies should not be planned until these preliminary assessments have been carried out and the technology and means of dissemination shown to be capable of meeting the needs and circumstances of the target population. [top] [end]GuidelinesGuidelines written by the University of Liverpool team have been developed to incorporate the lessons learned from this work. They recommend the development of plans for evaluation work in close partnership with a support organisation that has experience of evaluation research in similar settings. The guidelines seek to adapt established research study designs and data collection methods to the particular circumstances and needs of the project to ensure they are appropriate to the experience, culture and expectations of the people concerned.The guidelines may be downloaded via the @HEDON link below. [top] [end]AcknowledgmentsWe would like to thank all of the managerial, office and field staff at ARTI (Pune, India), Development Alternatives (Jhansi India) and GIRA (Michoacan, Mexico). Our appreciation and thanks also go to the women and their families who participated in these evaluation studies -our understanding of this complex field has increased due to their generosity and patience. This work was funded by the Shell Foundation, London.[top] [end]@HEDON
[top] [end]Download the original article Monitoring and evaluation of health and socio-economic impacts: Key lessons learnt from a Household Energy & Health Project (174 KB)[top] [end]Contents: Boiling Point 55 - Monitoring and Evaluation
Categories: Boiling Point 55| Monitoring and Evaluation | ||||||||||||||||||||||||
Page created:
23 June 2008; Last edited:
15 August 2008; Version: 9 | ||||||||||||||||||||||||
Pagename: BP55:MandEOfHealthAndSocio-economicImpacts-KeyLessonsLearnt @HEDON: GUJA | ||||||||||||||||||||||||
