| Main knowledge bank page |
Recent additions |
Recent changes |
What links here |
Categories |
Category cloud How-to guides | Organisation profiles | Project profiles | ||||||||||||||
Monitoring and Evaluation Experiences from the Field by Karabi Dutta
The first IAQ monitoring and evaluation (M&E) project conducted by ARTI was in the year 2000. The project was sponsored by MNES (Govt of India) and in the following eight years the Indoor Air Monitoring team of ARTI has worked extensively in rural Maharashtra and also in the urban slums of Kolkata in India. They have also undertaken M&E on an improved stove programme in rural Bangladesh. At present the team is involved in the monitoring and evaluation of the woodstove being developed by Philips, the consumer products company. A more detailed list of M&E projects conducted by ARTI can be found via the @HEDON link below. The monitoring and evaluation of improved stoves consists of two phases. In the first phase, laboratory based stove performance tests such as the Water Boiling Test (WBT) are conducted during the design-stage of the improved stove. In the second phase, field tests provide feedback on the performance of stoves in the hands of actual users in their own kitchens. This can be extremely useful, particularly at the early stages of stove dissemination. After the improved stove has been in use for some time it is beneficial to observe long-term changes in user behaviour in order to gain an understanding of how the stove performs as it ages. Poor field performance at any time can indicate faults in the construction of the stove and poor communication between improved stove designers and users. [top] [end]Successful study designThere are basically 3 study designs for determining the reduction in IAP in households:
The cross-sectional design requires the simultaneous sampling and monitoring of a large number of houses for both traditional and improved stoves, in the same geographic area. Since sampling needs to be carried out in houses that may not have necessarily received improved stoves, it creates many social and practical problems during monitoring. However, if the project period is limited then this design can be adopted. In all ARTI projects, the study design adopted was the “before and after” pattern, except with the Shell Foundation project where a ‘before and after with control” study design was used. The latter method is very helpful if health parameters are part of the work, as study periods are often short term (1 to 1.5 years) and the stoves are usually installed in the latter half of the study. As health benefits due to an improvement in indoor air can take a long time to become visible, it becomes difficult for a medical practitioner to understand a change in participants’ health within this short span. But if the control group is there (people who were never exposed to an improved stove), then a comparison may be made between the 2 groups that gives a greater confidence in the results. The more simple ‘before and after’ study design will still provide a very good comparison of the improvements (if any) which may have been achieved by the introduction of improved stoves in the rural kitchen. It is an entirely field based study conducted in order to understand the actual performance and acceptance of the stove in the rural community. Compared to the other two designs this method requires a smaller number of households to be monitored but needs a longer sampling period. It is also not very helpful if a health study is part of the research, for reasons given above.
Stove Performance test This is a WBT conducted with 2.5 litres of water as this mimics most of the cooking patterns in South East Asia (E.g. boiling rice or noodles). The aim is to understand the performance of the stove in the field and whether the stove provided by the manufacturer is as per specification. Emission testing Particulates and Carbon Monoxide (CO) are monitored by placing special equipment in the kitchen room for a periodof48 hours. The electrical monitors are installed at a specified distance and height away from the stove in order to capture the actual emissions and effects of ventilation for a stove in normal daily use. Emissions are also measured during the Water Boiling Test. Kitchen Performance test This is a 7 day test where a record of the total fuel used by the cook per meal is recorded, in addition to the food cooked and the total number of men, women and children present for the meal. This provides a more realistic stove fuel consumption and the comparative ‘before and after’ study gives the researcher a clearer picture of the fuel saving achieved by the improved stove. It also makes it easier to convince cooks about the fuel saving benefits of the improved stove. Controlled Cooking test This test can be performed in the laboratory or field, with ARTI preferring the latter. The test involves cooking exactly the same meal, with the same cook, on both the traditional and improved stoves. A record is kept of the time taken, the measured amounts of food and the total quantity of fuel used to cook the meal. This particular test provides a very clear idea of the comparative improvement that has been achieved by the improved stove, in terms of savings in time and fuel as well as the ease with which each item can be cooked on each stove. [top] [end]Methods – past and presentFor ARTI’s first project, two villages were selected about a 2 hour drive from the office by car. In total 8 project staff were involved but with no specific duties for each member, except for the project co-ordinator who directed activities. So each day two people were selected to go to the village and remain with the monitoring instruments as it was too risky to leave them unattended. The project staff were provided with survey questionnaires with which to interview households and data entry and monitor preparation was the project coordinators responsibility. The project did experience some problems as too many people were involved and there was a lack of specific responsibilities, which made it difficult to manage the process well. When combined with a project coordinator who was only able to visit site occasionally, there was a lot of confusion and although the project was completed it was unplanned and chaotic.Based on the lessons learned from the first project and subsequent guidance from by Dr Kirk Smith, Dr Nigel Bruce and their team members under the Shell Foundation project, we have since developed a good monitoring plan. [top] [end]Planning and designingThe first step is to decide on a study design and sample size based on the duration and budget of the project. Only after several visits to the area are villages or study areas selected and the cooperation of the Local Self Governing body of the village is sought before progressing. They are requested to provide a good, reliable field worker who will help with the day-to-day work. After this households are selected and only those who wish to take part are included in the survey. All the households selected for the study should share some basic common criteria. Details of study design and household selection are available in Energy for Sustainable Development, Vol XI No. 2 June 2007 ‘Design Considerations for field studies of changes in indoor air pollution due to improved stoves (pp71-81).
The role of the Project Manager is varied and includes the daily monitoring of staff activities and the on-site supervision of IAP monitoring and stove performance tests. The project manager usually travels to the villages to accompany the field workers 4 days a week, in order to talk to participants and resolve any outstanding problems which they may have. One day a week is kept for desk work, communications, data checking, preparation of reports etc. Other than this at least 1 hour every day is devoted to reviewing the days work and planning that of the next. Data analysis and preparation of the project report are entirely the responsibility of the programme manager but these activities are done after completion of the field work. Daily informal meetings are held with the core staff to plan activities and resolve any problems, depending on feedback from the field staff who are in constant telephone contact. To save time these meetings are often held while travelling. The field visits are planned so that each study area is visited by core staff at least twice a week, helping to quickly resolve smaller problems such as searching for alternate houses and stove damage. A detailed weekly and monthly calendar of activities is always planned in advance with work being assigned to each staff member, keeping in mind all other work, social functions and holidays. Work is planned on a weekly rather than daily basis to allow for the invariable last minute interruptions to be more flexibly integrated into the schedule.
[top] [end]ConclusionsA well-designed improved cook stove programme can provide multiple benefits for end-users. However, simply introducing an improved stove does not guarantee that positive outcomes will be achieved. Ultimately the stoves introduced into people’s kitchens will have to be adopted into their daily cooking practices and as such the end result of an intervention is as much a function of user preferences and behaviour as the technical design of the ICS. This is where the monitoring and evaluation of indoor air quality and stove performance plays such an important role.[top] [end]@HEDON
[top] [end]Download the original article Monitoring and Evaluation: Experiences from the field (479 KB)[top] [end]Contents: Boiling Point 55 - Monitoring and Evaluation
Categories: Boiling Point 55| Monitoring and Evaluation | ||||||||||||||
Page created:
23 June 2008; Last edited:
21 August 2008; Version: 4 | ||||||||||||||
Pagename: BP55:MandEExperiencesFromTheField @HEDON: DUJA | ||||||||||||||
