| Main knowledge bank page |
Recent additions |
Recent changes |
What links here |
Categories |
Category cloud How-to guides | Organisation profiles | Project profiles | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Boiling Point 55: Monitoring and evaluation case study scenario
Each Boiling Point case study presents a fictional household energy dilemma with advice from international experts [top] [end]The ScenarioThe meeting earlier that week had brought some very good news for Savita and the rest of the BALA team. Having spent much of the last year assessing whether a new type of efficient wood burning stove was suitable for use in their area, they had just managed to secure some significant funding with which to scale up the project. But now she was sitting at her desk wondering what to do next, with the scale of the task ahead suddenly becoming apparent…The funders had been quite specific; they would give support for an initial 3 year period with the requirement that 5000 stoves were produced in the first year, 10,000 the next and 20,000 in the third year. The money was coming from a variety of sources, a local Government agency, an international NGO and a private company and each one had a different agenda. They had all specified what they wanted from the programme and she had a list of targets and indicators on a variety of health, social, environmental, technical and economic issues. The whole point of the project was to see if an increase in numbers was possible, in terms of both demand and capacity (including manufacturing, engagement of local financial institutions, support and distribution networks, etc). If successful, the scheme would then receive increased funding to scale up further and roll out the programme to other areas of the country. Last year’s pilot project had gone well, with over 200 households taking part in a field study as well as the stove undergoing numerous performance and safety tests. The stove design needed a bit more work to make it acceptable to users, and the manufacturers seemed capable of producing the quantities they needed, but these weren’t Savita’s main concerns. She would need to work closely with her own project team as well as other local organisations, and then she also had to satisfy the many demands of the funders as well as her own organisation’s management.How was she going to design and implement a programme of this size? With all the day-to-day issues she would face, how would she monitor overall progress and also check that the work was going as planned? How were they going to tell what users thought of the stove and how often they used it, and what about marketing and aftersales –she has been involved with many of these issues before but never all at once! Savita knew she had to develop a Monitoring and Evaluation system but wasn’t sure where to start. In previous work she had tried to develop one, but being honest this had always been a last minute thing and now she was beginning to feel out of her depth… So in terms of M&E, how should Savita run the various stages of the programme so that everyone is kept happy and how does she prove that the various objectives of the project are being delivered? Disclaimer: The story presented in this case study is fictitious and as such any characters and organisations within it are not based on real life.
[top] [end]Case study response by Jonathan Rouse et alCongratulations to Savita and her team at BALA for getting this exciting opportunity to expand their work. With these new funds comes the challenge of satisfying the monitoring and evaluation requirements of new partners. Help is at hand from many quarters: no need to panic.Savita is right to consider both how she will monitor ongoing progress, as well as evaluate outcomes and impacts. Monitoring and evaluation are two slightly different concepts, though they are often related and complimentary.
Donors are often particularly interested in evaluation results, though will wish to see that you have carefully considered monitoring for the integrity of the project. Many of the tools used for monitoring and evaluation are the same, and findings from monitoring can often contribute to evaluation. There is not space to go into detail on all of the M&E areas that Savita will need to address, so we will focus on assessing socio-economic impacts and monitoring indoor air pollution and fuel-use.
[top] [end]Socio-economic impact assessmentSocio-economic impact assessment is about understanding what this stove and project means to people. Although stove efficiency and reduced indoor air pollution levels are important indicators for the donors, hearing from neighbours about convenience and ease of keeping the stove alight may have been what convinced people to purchase the stoves. Understanding these perceptions and motivations can assist you in developing effective marketing messages, and ensuring you design a stove which people want to use.Assessing socio-economic impacts is not unlike peeling an onion: a skilled investigator will reveal many layers, and by spending time talking to women and watching them cook in smoky kitchens, will likely suffer watering eyes. Time savings and changes in spending habits are important socio-economic impacts, but what are the secondary effects of these: relaxation, income generation, schooling, more food? Or perhaps financial savings have brought no benefit to women and children, and the removal of smoke has filled the house with flies. And what do these mean to women, men and children: better nutrition, better opportunities, more confidence or even empowerment? It is very important for investigators to keep an open mind, because users and householders may raise any number of unexpected impacts.
Specific visual tools such as ‘seasonal charts’ can be a great way to ignite discussion and debate among participants in group discussions. Seasonal charts consist of a table (as in Figure 1) with seasons denoted as columns, and various activities or experiences as rows. Rows could include: cooking location, type of stove used, household income, expenditure on fuel and so on. The group is invited to indicate seasonal changes for each row using beans or pictures. Seasonal charts can also be adapted for use with individuals who cannot read: simply use drawings of seasons and activities. In themselves charts can capture much information, but through further questioning and discussion the facilitator can gain invaluable insight into family life, choices and impacts. Focus group discussions and other open-ended techniques can often reveal much more than just socio-economic issues and impacts, and are an important first step in developing questionnaires. Understanding seasonal variations can also assist in broader M&E planning, for example ensuring IAP monitoring is repeated at comparable times of the year. [top] [end]Indoor air pollution and health monitoringDepending on her funders’ interests and the cooking practices in her region, Savita may want to monitor the impact of the new stoves on indoor air quality. This will give her a metric that is related to changes in human health. According to the WHO, there is strong evidence that exposure to indoor air pollution (IAP) increases the risk of pneumonia, chronic obstructive pulmonary disease (COPD), and lung cancer, and moderate or preliminary evidence that several other diseases may also be associated with the toxic compounds found in biomass smoke. A key to success in IAP monitoring is matching the study approach and design to the project phase.As the technology that the programme will promote still needs some design adjustments, we recommend that Savita wait to do a formal field assessment (effectiveness test) until the stove design has been finalised, manufacturing and distribution of the product is well established, and she has fully characterised her audiences. She can then perform a population-based study in homes with stoves that will have long-term relevance. At the current programme stage, we recommend that Savita conduct an efficacy test using a Before-After study design (monitor IAP levels first while traditional stoves are still being used and then again after the new stoves are installed). An efficacy test will focus specifically on the ability of the new stove technology to reduce indoor air pollutant levels in real-world homes under somewhat controlled conditions. By limiting variation in both the meals cooked and other major factors that can affect IAP levels during the tests, Savita’s study will require a relatively small sample size and fewer resources. Of course, she will not be able to draw conclusions about the extent to which the new stoves are used or whether the benefits are scaleable across populations: this will be assessed at a later stage. A range of instruments can be used to measure indoor air pollution. Typically, particulate matter (PM) and carbon monoxide (CO) are measured as they are considered the most harmful to health, have a long history of being measured and studied (both indoor and outdoor) and are representative of many of the other harmful air pollutants emitted from biomass cooking stoves. One effective and relatively inexpensive instrument for monitoring PM is the UCB Particle Monitor, which can store minute by minute PM concentrations. Two effective instruments for monitoring CO are the HOBO CO Logger (which has minute by minute datalogging) and CO diffusion tubes (which provide one average concentration).
[top] [end]Fuel useAnother impact that can be assessed relatively easily is fuel savings. Following a simple “kitchen performance” protocol, field workers weigh each household’s daily fuel for several consecutive days, first while traditional stoves are still being used and then again after the new stoves are installed (Before-After approach). The weighing can be done using simple, inexpensive spring scales (see Figure 2). Such an effort provides direct evidence on the fuel saving implications of the new stove. Fuel savings is a particularly valuable assessment, as it provides information that is useful for many audiences. For example, fuel savings is important for understanding the value of the stove to the customer in relation to its purchase price. The same metric is critical for documenting carbon and environmental (resource use, forests) savings.[top] [end]In summaryFuel-use patterns and fuel savings are important household energy success metrics to be considered with others presented in this response, and throughout this edition of Boiling Point. As good consultants, we have told Savita what she already knows: household energy is complex, with a very wide range of impacts. The most nuanced understanding of these layers of impacts comes from a combination of qualitative and quantitative information derived through observation, survey tools and participatory research. We encourage Savita to begin with a broad range of impacts on the table and then narrow the list by considering her funders’ interests, her stakeholders’ information needs, and her resources. We are confident an achievable fit-for-purpose plan will result, and Savita’s monitoring and evaluation activities will be launched.
[top] [end]Case study response by Firehiwot Mengesha et alAlthough the task of satisfying several donors’ requirements may seem daunting, with careful planning Savita and the BALA team can put in place an appropriate and affordable M&E system to meet all of their needs. Thorough M&E has been essential to our work at Gaia Association as it has enhanced our impact on the lives of our beneficiaries and also resulted in increased funding and policy support for our projects. Gaia Association is an Ethiopian NGO working to promote the use of alcohol fuels for household energy in a variety of settings including refugee camps, government institutions, and city housing. We seek to establish the local, sustainable production of both stoves and ethanol fuel to provide low and middle-income urban households in Addis Ababa with a safer and healthier alternative to biomass and kerosene. Since Savita’s project and objectives are similar to ours, elements of our M&E model could be applicable to her project. This basic model involves clearly identifying project objectives, selecting indicators and deciding upon the appropriate means of verification.[top] [end]Define Project ObjectivesBased on the outcomes of her pilot study, Savita should define project objectives which should be in line with her donors’ demands and priorities. With clearly defined objectives, Savita can then establish an appropriate M&E system, and some of these objectives might include the following:
[top] [end]IndicatorsAt least one measurable indicator should be identified for each objective:
[top] [end]Means of verification (MOV)
Monitoring stove sales: If BALA is the stove retailer, they will be in a position to directly access and monitor stove orders, sales figures and stock levels and ensure that the targets are being met. If the stove retailers are independent of BALA, Savita must establish a means of monitoring stove orders and sales in each location. The retailers could report their sales and orders to BALA on a monthly basis so that she can track and forecast the growth of the business. Monitoring IAP: BALA should conduct an IAP study on a sample of households that use the improved stoves. To reduce costs, training in IAP monitoring can be sought from other NGOs conducting similar research in the region and equipment can be borrowed or rented. IAP testing should be conducted both before and after the introduction of the new technology. Gaia Association ran a very successful IAP study in a sample of households in Addis Ababa and in Kebribeyah Refugee Camp. The study demonstrated quantitatively that the ethanol stove technology reduced levels of harmful pollutants to within WHO standards. These positive results led to increased donor funding. Monitoring Fuel Use: Baseline fuel use surveys can be conducted prior to the commercial scale up of the project to ascertain how much fuel the average household uses for their daily cooking needs. This could involve weighing standard fuel bundles to find out how much households are consuming. Follow up surveys can then be conducted in the same homes after they have purchased the improved stoves, to measure the difference in the amount of fuelwood used. This data can be strengthened by conducting controlled cook tests on the relevant stoves to establish how much fuel is required for preparing a typical meal. Gaia Association have conducted baseline and follow up surveys of this type in all of our project sites to determine fuel use patterns and changes following the introduction of the ethanol-fuelled stove. Monitoring Time Saved: Qualitative household surveys can be used to find out how much time has been saved cooking and gathering wood and if this saved time is now used for other activities. Monitoring Stove Use: The M&E team should instruct the surveyors to observe stove use during their household visits and then report any problems or concerns to the M&E team. Monitoring Income Saved: Baseline and follow up surveys can be conducted to determine household expenditure on fuel (where fuel is purchased) and how this changes with use of the new stove. This data can be corroborated by checking fuel prices in the market, and by cross-referencing using the fuel use survey data. Gaia Association used similar techniques to demonstrate income savings accrued at the household level by switching from fuels such as kerosene and charcoal to ethanol for cooking. The importance of having a solid M&E system cannot be overstated. The starting point in the planning process should always be a clearly defined set of objectives which reflect the donors’ interests in the project and for which measurable indicators may be selected. From there, the most appropriate MOV can easily be decided upon. With such a system in place, along with a timeline and a strong M&E team at BALA, Savita is in a good position to track the progress of her project from various angles. M&E findings can be referred to in planning scale ups, predicting obstacles, modifying project design and securing additional donor support. M&E has become a central element of the work of Gaia Association and an area to which we devote significant time, energy and investment. By evaluating the impact of our work, we build confidence with our donors and most importantly, we maintain a dialogue with our target communities to ensure that we continue to serve their needs as best we can.
[top] [end]Case Study response by Jocelyn Roberts
Savita’s contract with her friend’s company would allow her to sell them carbon credits through the voluntary market. Carbon finance has very strict requirements pertaining to the quality of data provided and the frequency with which data is up-dated and analysed. So Savita should use the basic requirements for verification of the carbon credits as the backbone of the monitoring and evaluation (M&E) system she will develop, making sure that the various indicators and reports required by the other funders are included in the new system. A monitoring system is not the only technical aspect of project implementation. Savita should also use a logical framework analysis, or log-frame, as a planning tool to assure the various objectives of all stakeholders are met. The log-frame provides, in one easy-to-read worksheet, the inputs and aims of the project, the indicators used to measure the achievement of aims, and the outcomes and outputs expected from implementation. A good log-frame can provide a technical guideline throughout the life of the project, for the evaluation of implementation as well as to assess if the project has met its initial aims. Paired with a project timeline, these should be the fundamental documents that will guide the execution of the project. Based on the log-frame she develops, the M&E system will allow her staff to show donors and stakeholders that resources were well used throughout the duration of the project. The M&E system could consist of the following parts: a baseline survey; periodic monitoring (collection and analysis of data to determine if the project is meeting geographic, scale, and time targets); and evaluation at key programme milestones, including post intervention to allow for the review of intended impacts. As the monitoring system will require considerable time and resource to implement, Savita should make sure that she has the funding to support it. She will then have to obtain approval from the Government, International NGO and private company who are funding the project before she starts the implementation. After the first year of implementation, the results of the monitoring will have to be verified and certified by an approved independent institution. Savita should start by designing the baseline survey in order to establish the pre-intervention conditions of stove users in the target and control areas. In order to address the needs for social, economic, health, and environmental data as well as the perceptions of stove users, quantitative and qualitative data will have to be collected from a variety of sources. A sample of relevant indicators include: demand, supply, the number of stoves sold, equipment ratio, fuel mix, fuel source, IAP monitors, stove price, household income, stove use data (for how long does a family use a stove?), shelf life (how long between production and use in the home?), user satisfaction, time saving, money saving, etc. The baseline survey will allow them to “see” changes in indicator values from project initiation through to completion and beyond. The baseline survey should have as many “layers” of data collection as there are units of required analysis. For example, a donor may need provincial and district level economic, demographic, and environmental data for the geographic region of the intervention. This data is sometimes available from government statistical surveys. Health, fuel, and technology use data will have to be collected at the city, commune or village level (in the case of Cambodia) in order to be specific to the programme targets. In addition, qualitative data should be collected through various methods of social science research, such as focus group discussions (FGD), interviews, or surveys about preferences and perceptions. Those interested in a complete qualitative baseline survey addressing behaviours and perceptions should plan to implement a knowledge, attitudes, and practices (KAP) survey. Similar to monitoring practice, a KAP survey can be repeated post-intervention to gauge the impact of the intervention (and other influences un-related to the project). It may also be repeated annually thereafter to gauge the sustainability of the knowledge and new practices gained through the intervention. Once the data is collected it has to be both entered and stored in statistical and/or econometric analysis software such as SPSS or STATA. Analysis of the data will provide the most useful information on the pre-intervention conditions in the target area, and the changes that have occurred since the project began. Baseline studies should be easily replicable and monitoring should consist of the repeated, predetermined, collection and analysis of data from the field. As a result of the latter, Savita may have to offer some kind of incentives to households to ensure the timely reporting of the required data. In order to satisfy the stringent needs of the verifier for carbon credits, she will have to provide data on the number of stoves in use from two different points along the distribution chain: from producer, to distributor, to retailer to user. But with an effective M&E system in place, Savita will be able to respond to the reporting needs of all the funders, including carbon finance, and she will be able to better manage the progress of project implementation in the field.
[top] [end]Case study response by Joy ClancySavita would be wise to include gender analysis as part of her approach to monitoring and evaluating her stoves project. She might be surprised to read this because in a household stoves project the target group is usually women. But gender is about men and women, so what has gender got to do with stoves? Well quite a lot actually.There is plenty of evidence to show that when household equipment is bought, even equipment for the kitchen, men are involved in the decision making process (see for example, Dutta 1997). So the men within the household need to be convinced about the benefits of buying the BALA stove. Often men and women will also have different selection criteria for a stove, for example, women might want one that is easy to light and gives a cleaner kitchen whereas men may want a stove that gives quicker meals. So the BALA stove will need to meet both women’s and men’s needs. Another reason for including gender is that it will probably be a requirement of the international NGO, particularly if they are using donor funds. Gender could be included as one of the social indicators Savita has to measure. However, there are also sound practical reasons for paying attention to gender issues. There is a growing acceptance of the fact that ignoring gender in projects is a contributory factor to project failure (Fong and Bhusan, 1996), while paying attention to gender can lead to a better fit of project interventions with the intended beneficiaries and thus create greater management efficiency in terms of delivery (Skutsch, 1998). In other words by including gender analysis in her monitoring and evaluation methodology toolbox, Savita increases the chances of meeting her project target. Help is at hand for Savita. The Department of Technology and Sustainable Development (TSD, University of Twente) and ENERGIA have developed gender analytical tools specifically for use in the energy sector. These tools can easily be combined with existing procedures, in particular, they fit into the project cycle. They differ from other gender analytical tools in two ways. Firstly they make explicit the ‘gender goals’ for a project, i.e. identifying which gender issues will be addressed, and secondly they assess the gender capacity of organisations involved in project delivery (Skutsch 2004).
BALA also needs to assess whether or not, as an organisation, it is equipped to deal with a gender approach to project implementation, for example that staff are gender sensitive to cultural issues in the region where stoves are to be promoted (i.e. are women able to attend training sessions at night or at some distance from home?). ENERGIA’s gender-analysis tools consist of a framework with a number of steps. Within each step there is a set of questions that need to be asked in a logical order, and the data can be gathered by a number of methods, including desk studies and participatory approaches. The questions are not meant to be prescriptive and can be adjusted to suit the context. The data collected is then used to complete a number of tables which can then be used to analyse the data, to aid decision making, and to help identify areas for remedial action (for example, increasing women’s participation in stove design). The tools were designed for the planning phase of energy projects, although they have been shown to work for energy project evaluation (Clancy et al, 2007). ENERGIA members who have used the tools report them as easy to work with. There is an easy to follow manual which BALA can use and it’s free to download via the @HEDON link at the end of this article. The tools provide comprehensive data, although they do need to be adapted for the particular context either to prevent the collection of redundant data, or to ensure the collection of more context specific data. So BALA has some work to do but Table 1 gives some suggestions. Table 1 Gender analysis of BALA Stoves Project
[top] [end]Case Study response by Stephen GitongaSavita needs to prepare a full monitoring and evaluation (M&E) framework that will provide all stakeholders, including programme management, field staff, beneficiaries and donors with information on programme status and results. It should help assess the progress of key programme aspects at each stage and the achievement of the overall objectives. The basic framework would involve four steps.[top] [end]Step one: Constructing the logicThe first step in this process is the careful review of the strategic focus of the programme in light of the interests of donors and beneficiaries, to ensure that the programme satisfies the requirements of both groups. The outcome of this exercise is an overall framework that details M&E at each level and stage of implementation.At the programme level, indicators must be well formulated and explained as well as broad in nature and aimed at showing the achievement of programme goals and outcomes. The project level indicators are more specific and aim at showing the achievement of programme outputs and the accomplishment of activities. However, they should also show the achievement of specific project goals and outputs. The programme level indicators define the overall focus of the programme which follows the logic and requirements of the donors and beneficiaries. The project level indicators assess the process of programme implementation and are critical to the programme management team. The experience of the GEF Small Grants Programme (SGP) shows that global programme M&E should also include process indicators covering, for instance, aspects of project design, approval, implementation and completion. When developing the framework logic, every M&E activity at each stage should be linked in with the overall programme goal and outcomes. [top] [end]Step two: Developing the baseline criteriaThe second step is to prepare the baseline criteria of programme activities based on the strategic focus. Both monitoring and evaluation require information about the current state of the beneficiaries or locality before programme activities begin, and are usually centred on the strategic focus identified in step one.This forms the baseline information from which the assessment of impacts can be made and will be of help to Savita in the identification and construction of indicators. Through these indicators, Savita can be in a position to assess the programme progress and accomplishments within the logic constructed in step one. While the baseline information is mapped, specific milestones should be identified that will be accomplished at the various stages of programme implementation. Because of the need to monitor accomplishments while taking remedial measures, the monitoring and evaluation framework that Savita should prepare needs to be systematic, but at the same time allow for unexpected occurrences and results. [top] [end]Step three: A sustainable system for M&E framework implementationFor an M&E framework to operate effectively, a third step is necessary in the form of a system that ensures the process of activity implementation is being adequately monitored and assessed. At the UNDP GEF SGP a computerised, real-time online system captures M&E activities as they happen in over 100 countries. The system is based on a database which can be operated both online and offline depending on the local situation. The online system is complemented by a reporting system which has obligatory benchmarks and deliverables. The sum total of both the computerised system and the other reporting requirements determines how effectively the M&E framework is being implemented.The system should allow for the extraction of reports to satisfy both donors and programme management. It should also summarise the achievements of a series of benchmarks and deliverables at specific programme stages, directly from the online real-time database. Because the benchmarks and deliverables are based on the identified baselines they are a core element of M&E and their achievement depends on a focussed, thoroughly coordinated and synchronized implementation of different aspects of the programme activities [top] [end]Step four: Ensuring a feedback mechanism for donors and beneficiariesKey to the effectiveness of the M&E system is the ability to give feedback to donors as well as beneficiaries, with the latter often not effectively done. The UNDP GEF SGP has developed a method that intrinsically incorporates a feedback mechanism within the M&E process. It is based on the premise that an effective monitoring and evaluation framework is that which allows the participation of the beneficiaries and gives them the ability to feedback in to the project.It allows the beneficiaries to ask questions and also give answers related to the important aspects of programme activities. It also enables the programme to build the capacity of the beneficiaries and also enhances understanding between the various project groups. Linked to the feedback mechanism is a jointly developed baseline scenario which, using the indicators, helps the beneficiaries to clearly understand the impact of the programme on their lives. [top] [end]Enhancement of accountabilityWhere the beneficiaries are involved in a participatory process, monitoring becomes a continuous activity. It thus ensures that there is both technical and financial accountability during programme/project implementation.The GEF SGP promotes participatory monitoring and assessment in the design and implementation of Country Programme activities as part of a broader approach to M&E. Apart from building capacity, it enhances the involvement of affected beneficiaries and stakeholders alike, and provides for better correction of mistakes during programme implementation, thus ensuring that lessons are articulated and learned by the beneficiaries themselves. It contributes to building consensus, creating a sense of “ownership” of the process and programme approach, and promotes mutual understanding. [top] [end]ConclusionStove monitoring and evaluation activities include capacity-building and public awareness components that are part of the participatory process of establishing an effective M&E system. It is therefore important for Savita and the programme team to have a good sense of the beneficiaries’ perceptions and practices before the programme begins, as they would do for the donors.The process of participatory monitoring and assessment begins at inception. The GEF SGP experience with climate change and energy programmes demonstrates that early consultation in project design with beneficiaries is as important as consultation with the programme donors. The consultation at this stage involves such aspects as defining problems, potential courses of action, available beneficiary resources, the role of external support, construction of the baseline and beneficiary expectations. At the programme level, M&E design strategy requires participatory compilation of baseline information and also agreement on the definition of programme concepts (e.g. to define the focus of activities). The beneficiaries and programme management need to reach a consensus on programme objectives and activities and in the process establish an effective monitoring and evaluation plan that includes their roles and responsibilities. Once these aspects are defined, the M&E system can effectively be established, implemented and feedback provided. [top] [end]@HEDON
[top] [end]Download the original article Monitoring and Evaluation Case Study (401 KB)[top] [end]Contents: Boiling Point 55 - Monitoring and Evaluation
Categories: Boiling Point 55| Monitoring and Evaluation | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Page created:
23 June 2008; Last edited:
04 August 2008; Version: 6 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Pagename: BP55:MonitoringAndEvaluationCaseStudy @HEDON: QUJA | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
