Monitoring and Evaluation

Table of Contents
1. Issue in brief 2.Trends and History 3. Current Best Practice 4. Areas of Research 5. Resources 6. Organizations and People 7. Web links 8. Related Documents Sources

[top] [end]1. Issue in brief

Globally, indoor air pollution is responsible for approximately 1.6 million deaths every year.

Various interventions are available to reduce indoor air pollution and associated health impacts at the household level. Working chimneys and hoods, increased ventilation and improved combustion can reduce concentrations of indoor air pollutants. Reducing human exposure to these harmful by-products of combustion leads to reduce illness. To implement an intervention Monitoring and Evaluation of Indoor Air Quality, (IAQ) is essential. There is a need for proper design of investigations so that causes of indoor air quality problems and their solutions are clearly understood. Numerous chemical and physical factors influence the indoor concentrations of contaminants. Source characteristics, chemical and physical sinks, rates of air exchange, indoor air flow patterns and occupant activities are some of the factors which need to be considered or measured when monitoring indoor air quality. The multiplicity of these factors makes the task of designing an investigation very complex. Determining the parameters to be measured as well as defining the extent of measurement are critical aspects of designing a study. The process is further complicated by the fact that individuals drawn to such efforts are of diverse professional and educational backgrounds-environmental scientists, chemists, industrial hygienists, social workers, architects, mechanical engineers, public health professionals etc. Thus a common understanding of the basis for conducting indoor air quality measurements is required. Well-conceived designs and protocols form a critical starting point for successful measurement programs.

So the basis of successful M&E program is a good study design and a protocol to conduct the study. Design of a study or investigation can be defined as developing a strategy or approach. This involves a specific statement of goals for the study and translation of these goals into measurable objectives. These objectives need to specify measurement parameters and statistically justifiable sample size. Protocols refer to specific procedures to be followed in conducting a study. To implement a design, a detailed procedure has to be developed to guide sample selection, monitoring, quality assurance, and data analysis. The documentation of such procedural items forms the written protocol of the study.

IAQ monitoring sites in urban areas are usually (i) commercial and office buildings and (ii) residential buildings or houses. The cause of poor air quality in these areas are ventilation, filtration, outdoor sources, household items like old mouldy carpets and linen etc. In urban city buildings, airtight construction, designed to improve energy efficiency, permits less fresh air to enter the home. High efficiency heating and cooling systems constantly re-circulate indoor air so particles and VOCs build up over time.

In the rural areas the chief source of pollution is the cooking fuel and stove, light source in the evening, ventilation etc. In both the areas a statistically firm design for the measurement should be prepared and piloted before implementing the design in the study area. It should include-development of study objectives, data analysis plans and interviewing methods and questionnaire construction, identifying study respondents and determining the required number of such participants. The development of questionnaires relevant to the study area and study objective to explain the cause of variations in indoor air pollutant concentration is an important aspect of monitoring and evaluation.

Instruments needed for monitoring depend on the study parameters and design and of course budget. The monitoring can be laboratory based under simulated conditions or field based. In the Laboratory based study more sophisticated instruments can be used like a gas chromatograph to analyse the air in the room etc, but in a Field study the instruments should be portable and easy to use, preferably powered by battery.

Finally, it is important to demonstrate the sustainability and cultural acceptability of a given intervention. Documenting these impacts will help generate the evidence to convince policy-makers and donors at all levels that household energy interventions work in reducing one of the major global threats to children's and women's health.

[top] [end]2.Trends and History

In the 1980s, many protocols were developed by several organizations both regional and international, to test the performance of biomass cookstoves. Some of these, like the testing protocols, developed and implemented by the National Programme on Improved Chulhas (NPIC), and approved by the Government of India, have occasionally been deployed to disseminate improved cookstoves (ICS). However, little development or formal evaluation of these methods has occurred since. In recent years, there has been growing interest in disseminating ICS to not only improve fuel efficiency but also to reliably improve indoor air quality (IAQ) by use of chimneys, improved combustion, and other techniques. Although measuring IAQ in biomass-using households has been the subject of numerous research studies in the last decades, standard monitoring methods that could be deployed by anyone interested in evaluating ICS programs have not been developed.

Inefficient use of biomass fuel and the high indoor air pollution (IAP) is often created by the traditional biomass cookstoves, hence they have been the target of programs designed to substitute various varieties of improved stoves in rural homes around the world. In the past, a few of these programs have been quite successful in disseminating many millions of stoves, e.g., the Chinese National Programme; while others have had less success, e.g., the Indian National Programme. In recent years, because of the increased knowledge of the health effects of IAP, there has been a resurgence of interest in such programs but tempered with a desire to improve the success rate. One problem with past programs was that there had been little systematic evaluation of the impacts of these stoves in actually reducing IAP once installed so that programs and stoves could be better designed and programs better targeted and evaluated. This has partly been due to a lack of accessible equipment and methods for stove disseminators suited to this task since standard methods used by the air pollution community are not easily adapted to use by non-technical workers without laboratory backup and significant budgets.

To meet this need, the Household Energy and Health Programme of the Shell Foundation funded a study to develop easily usable and relatively inexpensive IAP monitoring kits for use by such non-governmental organizations (NGOs).

In 2004, the Household Energy and Health division of the Center for Entrepreneurship in International Health and Development (CEIHD, pronounced “seed”) received funding from the Shell Foundation to provide technical assistance in monitoring and evaluation to household energy practitioners. CEIHD's M&E activities grew out of a project initiated in 2002 by the Shell Foundation in partnership with the University of California, Berkeley, and the University of Liverpool, to develop appropriate monitoring and evaluation strategies for the foundations first four pilot projects. Through this initiative, a series of collaborative protocols were developed to assist practitioners (mostly NGOs) with monitoring plan design, indoor air pollution measurements, stove performance assessment, and the review of socio-economic and health outcomes. Concurrently, the Shell Foundation supported the development of the UCB particle monitor, which uses photoelectric technology to measure particle concentration, and offers a relatively simple and efficient way to monitor pollution concentrations and exposures in the field. .[ http://ceihd.berkeley.edu/heh.mande.htm#Background CEIHD’s] current monitoring and evaluation program builds directly on this activity.

Since March 2004, [ http://ehs.sph.berkeley.edu/hem/page.asp?id=30 University of California, Berkeley,] have deployed these kits with four NGOs representing the first effort to provide non-research groups running stove projects with similar tools, methodologies, and training to evaluate how much improved stoves lower IAP levels in households compared to levels found in households with traditional stoves. Standard kits have also been developed for evaluating stove performance and changes in symptoms of ill-health in women and children. Development and dissemination of these tools will facilitate a more detailed and usable evaluation protocol for improved stove projects as well as provide data-driven recommendations for the next generation of improved stoves and dissemination methods.

In 2005, WHO organized a series of 5-day training workshops as a step towards building regional capacity in the area of household energy and indoor air pollution monitoring. Workshops were conducted as a contribution to the Partnership for Clean Indoor Air in collaboration with the Pan-American Health Organization, the United States Environmental Protection Agency, the German Technical Cooperation (GTZ), the Center for Entrepreneurship in International Health and Development at the University of California at Berkeley (CEIHD) and the Aprovecho Research Center.

Specific workshop objectives of the workshop can be found here

[top] [end]3. Current Best Practice

Indoor air pollution from household energy ranks as the fourth leading health risk in poor developing countries. Breathing elevated levels of indoor smoke from home cooking and heating practices more than doubles a child’s risk of serious respiratory infection and may also be associated with adverse pregnancy outcomes (e.g., stillbirth and low-weight babies). In response to this challenge, founding governments and organizations launched the Partnership for Clean Indoor Air at the World Summit for Sustainable Development in Johannesburg in September 2002.

The Partnership’s Mission

More than 120 public and private organizations have joined thePartnership for Clean Indoor Air and are contributing their resources and expertise to improve health, livelihood, and quality of life by reducing exposure to indoor air pollution, primarily among women and children, from household energy use. The Partnership is focusing on four priority areas:

Social/Behavioral Barriers
Local Market Development
Technology Design
Health Effects

CEIHD at present is working with several household energy projects in Africa and Asia to provide technical support for monitoring and evaluation activities. Their services include training in monitoring techniques, assistance with monitoring plan development, on-site sampling support, and remote data-analysis consulting.

CEIHD is also the distributor and trainer for the UCB particle monitor, which they are currently offering as part of an IAP monitoring kit that also includes the HOBO CO monitor, software, and peripherals.

Evaluation of an household energy intervention program may be done in two areas.

Impact evaluation tries to assess whether an intervention has been adopted and implemented in the community and whether it has been effective in achieving its intended impacts. Economic evaluation tries to demonstrate the economic return of investments in an intervention and may be used to compare the cost-effectiveness of one intervention against another.

In both cases it is important to carefully document intervention impacts to contribute to the international evidence base. Household health and energy programs may be designed to reduce respiratory health problems among children and women, to improve people's livelihoods or to tackle deforestation pressures or land erosion.

But before designing an evaluation project the following things should be considered:

Deciding what to monitor should be demand-driven,
Information should be collected from the target audience,
Evaluation objectives as well as the thematic priorities of an organization.

Other important considerations are:

Scale and type of intervention being contemplated,
Feasibility of introducing such interventions issues,
Institutional capacity,
Financial resources,
Human resources,
Time

The evaluation project design of one project cannot be copied verbatim for another project. Selection of a project design depends on what we wish to evaluate. The evaluation may be for an improved stoves performance in the field, social acceptance of a new stove in the field, a new ventilation design implemented in the field or socioeconomic impact of an intervention. For each of these the study design will be different. Other important parameters will be local environmental and political climate and availability of resources.

After considering these factors a study may be designed based on these three main design options:

the before-and after design with a control group
the before and after design without a control group and
the cross-sectional design

A study may be based on:

Quantitative methods - including performance testing, indoor air pollution monitoring and questionnaires. By this method one can check and quantify objectively compare one intervention against another.
Qualitative methods - include in-depth, open-ended interviews, focus group discussions, key informant interviews, direct observations of behaviours and participatory methods. This method helps to reveal the perspectives of individuals or communities and provide important contextual data to explain the results of quantitative analyses.So qualitative data lends strong support to the qualitative data.

For any evaluation study design the most critical factor is the Sample size, i.e. the number of individuals, homes or stoves to monitor. If the sample size is too large, time and financial resources are wasted on superfluous data collection. If the sample size is too small, it is impossible to answer the questions asked in relation to the impact of an intervention. The intervention projects are basically for the people. So any study of this type involves working with the people spending time with them asking them questions, placing monitors in their households or on their person. Stove performance tests requires spending long hours in their homes using their time and invading their privacy. Therefore working with people involves ethical considerations. Special care needs to be taken to avoid treating participants as mere research subject.

Questionnaires and topics should be pilot tested and at least one or two local person needs to be involved in the questionnaire designing to deal with local dialects and terminology, cultural practices, taboos and social customs.

[top] [end]Most Significant Changes Method

"The most significant change (MSC) technique is a form of participatory monitoring and evaluation. It is participatory because many project stakeholders are involved both in deciding the sorts of change to be recorded and in analysing the data. It is a form of monitoring because it occurs throughout the program cycle and provides information to help people manage the program. It contributes to evaluation because it provides data on impact and outcomes that can be used to help assess the performance of the program as a whole.

Essentially, the process involves the collection of significant change (SC) stories emanating from the field level, and the systematic selection of the most significant of these stories by panels of designated stakeholders or staff. The designated staff and stakeholders are initially involved by ‘searching’ for project impact. Once changes have been captured, various people sit down together, read the stories aloud and have regular and often in-depth discussions about the value of these reported changes. When the technique is implemented successfully, whole teams of people begin to focus their attention on program impact." (from MSC Guide by Rick Davies and Jess Dart)

Most Significant Changes monitoring is different from common monitoring practice in at least four respects: (a) The focus is on the unexpected, (b) Information about those events is documented using text rather than numbers, (c) Analysis of that information is through the use of explicit value judgements, (d) Aggregation of information and analysis takes place through a structured social process.

For more information on this method see:

[top] [end]Monitoring: Indoor air pollution

Biomass pollution. Two pollutants are of primary interest for both health effects and IAP monitoring: particulate matter (PM) and carbon monoxide (CO).

Smaller particles (PM 2.5 and PM1) are likely to be most harmful, as they penetrate deep into the human lung. Larger particles are more likely to get 'filtered' by the upper respiratory tract. Considering available technologies and the relative cost and ease of monitoring, many organizations are measuring levels of PM 2.5 nowadays.

[top] [end]Indoor air pollution measurement options

There are many options for measuring the Indoor air pollution:

indoor concentrations over a period of 24-48 hours,
personal exposure for 24 hours,
outdoor or total emissions
based on the time factor and instruments available one can decide on what pollutants can be measured.

Measurement duration, seasonality and sampling intervals are important factors in deciding when to measure.

The various methods for CO measurement include:

bag collection and lab analysis,
colour-change diffusion tubes and
electro-chemical monitors.

The various methods for PM measurement include:

gravimetric monitors (pump and filter method) and
light-scattering devices.

The choice of method depends on:

the purpose of the project or programme,
the capacity of staff
available financial and
human resources.

All methods require data management and quality control. Although monitoring is done with instruments, pre-and post monitoring questionnaires also need to be filled by the study participants on solid fuel use, stove use, food cooked and household information like family members and kitchen dimensions etc.

[top] [end]Monitoring: Impacts on health

Most household energy interventions - directly or indirectly - aim to improve health among their target populations. WHO in their document Indoor Air Pollution and Household Energy Monitoring have suggested three ways of assessing changes in health outcomes :

The best-available assessment- a physician-based assessment of pneumonia in children and COPD in women;
The feasible quantitative assessment- a questionnaire based assessment of respiratory disease symptoms; and
The qualitative assessment- which obtains information from interviewees on those symptoms perceived to be associated with indoor air pollution.

Since majority of the indoor air pollution studies involve intervention in the form of introduction of improved stoves in rural households, monitoring the performance of the stoves in the field is very important.

It is often stove performance that determines whether a given intervention is adopted or not, and whether it is used and maintained appropriately. As a minimum, an improved stove must meet the users' needs as well as the fire or stove that was used initially. Beyond this, stoves should decrease the amount of fuel needed and make the cooking task easier.

[top] [end]Stove performance criteria

Six criteria are important for evaluating stove performance:

Burning rate- This is the measure of the rate of fuel consumption while bringing water to the boil.
Thermal Efficiency - is the ratio of the work done by heating and evaporating water to the energy consumed by burning wood or any other fuel. This is the most commonly used criterion for comparing stoves.
Specific Fuel Consumption - Fuel required to produce unit output (boiling water). This is the most useful criterion for determining how much fuel a stove is likely to consume.
Fire power- This is the ratio of the fuel energy consumed by the stove per unit time. It tells the average power output of the stove (in Watts) during high power test.
Turn down ratio- This indicates the degree to which the power output from the stove can be controlled by the user. Stoves with a higher turn down ratio are likely to use less fuel during a real-life cooking task, which involves bringing food to a boil and then cooking it at a simmer for an extended period of time.
Speed of cooking - This is mostly a measure of user friendliness. Speed of cooking is specified as the time it takes to boil or cook a given amount of food, generally per litre. However, cooking time also tends to be the time a cook spends near the stove and thus determines duration of exposure to indoor air pollution.
User satisfaction - This represents a subjective but important criterion, as user satisfaction determines stove adoption and use. Stoves are frequently chosen because they cook well and not because they save fuel or emit less pollution. We gain an idea of user satisfaction by surveying local use of the stove.
Emissions - Standard emissions criteria and methods to assess them are not yet available but in the process of being developed. Out of all the performance criteria, stove emissions are most directly related to indoor air pollution levels and thus health.

M&E of the impacts of a household intervention, such as an ICS, can occur at several levels:

Controlled tests: In laboratory or near laboratory settings with simulated cooking, these are easiest, quickest, and cheapest to conduct, but reveal the technical performance of a stove, not necessarily what it can achieve in real household settings, particularly with devices like stoves that are sensitive to operator behavior.
Efficacy trials: These are conducted in real households cooking real meals, but under controlled settings in which every effort possible is made to make sure the stove is used to its best effect. They reveal what is maximally possible, but not necessarily what is actually achieved.
Effectiveness trials: These are conducted in the course of an actual dissemination effort with real populations cooking normally and give the best indication of real-world changes. Only these can determine actual usage under realistic promotion schemes. Evaluating such trials is more expensive, difficult, and time-consuming, partly because the nature of real households is high variability and thus sample sizes must be fairly large to obtain statistically significant results.

If the aim of Monitoring &Evaluation is to understand the impact on populations, the focus would be on effectiveness, i.e., M&E as much as possible aimed to determine the effects in real populations.

Protocols for conducting Effectiveness trials:

Water Boiling Test: This lab-based test attempts to simulate the most common cooking modes of a stove while keeping other factors constant to make the results as comparable as possible between different projects. The test consists of three phases, each representing a particular cooking situation: (1) bringing water to a boil with a cold stove; (2) bringing water to a boil with a hot stove; and (3) simmering water with a hot stove. The results relate to four of the six performance criteria: efficiency, specific consumption, time to boil and turn down ratio.
Controlled Cooking Test: This lab-based test involves local cooks preparing a local dish. Adding these variables limits comparability of results to a given setting but provides important feedback as to the likely acceptability of a stove by local users. The results relate to specific consumption, speed of cooking and user satisfaction.
Kitchen Performance Test: This is the most difficult and resource-intensive test. It consists of a survey and a fuel consumption test with families using both the traditional and the improved cook stove. The test gives results of user satisfaction and per capita fuel consumption for a given stove. As the test encompasses many different variables, often a large number of tests need to be performed to assure statistical accuracy in the results.

[top] [end]Monitoring socioeconomic impacts

Interventions in the form of change in fuel or stove have a socioeconomic impact. Elizabeth Cecelski ,Technical Director for Research & Advocacy, ENERGIA, International Network on Gender and Sustainable Energy in her paper,[https://practicalaction.org/docs/energy/docs50/bp50-gender.pdf Is gender a key variable in household energy and indoor air pollution interventions?] has commented that, ‘Household energy interventions are generally seen as beneficial to women, affecting many aspects of their lives (Klingshirn, 2000; HEDON, 1995).Many such programmes have involved women as staff and entrepreneurs as well as beneficiaries. Despite this, many more household energy programmes have failed than have succeeded in reducing wood fuel consumption and indoor air pollution. Past research has identified success factors, such as focus on likely adopter groups, financial sustainability, interaction between stove designers, producers and users, mass production, minimal subsidies, and meeting consumer needs (Barnes et al 1992). But A gender disaggregated analysis of household labour time, would lead to the conclusion that it is the availability or otherwise of women’s unpaid labour time that is the crucial factor in determining the extent of wood fuel use, or the extent of economizing on wood fuel use. Further, that even if income were to increase without any reduction in the availability of women’s unpaid labour, then there is not likely to be a reduction in the use of wood fuel, either through using more fuel efficient stoves or through moving onto other fuels.

As per the Indoor Air Pollution workshop resources WHO document,.socioeconomic impacts include:

Time use: An improved cooking stove which consumes less fuel will result in less time spent on fuel collection for those who gather wood. Equally, shorter cooking times (for example when moving from a 1-pot to a 2-pot stove) also free up time. Saved time can have secondary benefits.
Changes in expenditure: In situations where fuel is purchased, fuel savings will result in lower expenditure on fuel. Occasionally, for example when shifting from using gathered wood to liquefied petroleum gas (LPG), expenditure on fuel may increase. However, the increased expenditure is often offset by time savings and other benefits. Purchase and maintenance costs must also be considered. Like time-savings, financial savings may have secondary benefits, such as increased expenditure on food and better nutrition.
Prestige and status: A cleaner house due to less smoke or the prestige of owning a modern stove can result in a perceived rise in the status of users. This can be empowering and may also be an important promotional tool.
Other impacts - including problems: Users often identify benefits not foreseen by the implementing organization. Examples include improved portability of the cooking device, the ease of keeping it alight or a reduction in pot-blackening soot.

More discussions on Why household energy is related to gender can be found here

The socioeconomic impact of a study can be evaluated by the following methods:

‘’’Qualitative methods’’’ using Questionnaires, Focus Group Discussions and Key Informant Interviews and ‘’’Quantitative methods’’’ through measurable questionnaires (time, quantity, cost etc) in a before and after study with or without control groups

[top] [end]4. Areas of Research

Authors / contributers wanted! Do you have knowledge or experience in this topic? If you do, please consider writing something for this page...

[top] [end]5. Resources

Household Energy and Health Project evaluated the impact of three improved cookstove programs on indoor air pollution and fuel consumption, as reported on in a special issue of Energy for Sustainable Development

[top [end]6. Organizations and People

Aprovecho Research Center

[top [end]7. Web links

[top [end]8. Related Documents

IndoorAirPollutionInDevelopingCountries.pdf (1,626 KB)
World Health Organization - Research Tools]
"World Health Organization: Interventions to reduce indoor air pollution"
"Monitoring and Evaluation of Improved Cookstove Programs for Indoor Air Quality and Stove Performance"
What's Cooking? A review of the health impacts of indoor air pollution and technical interventions for its reduction WELL Study produced under Task 512 by Jessica Budds, Adam Biran and Jonathan RouseOctober 2001

[top [end]Sources

edit this page

Page created: 01 March 2007; Last edited: 02 July 2007; Version: 7
Knowledge Bank text is available under the terms of the GNU Free Documentation License.

Pagename: MonitoringAndEvaluation @HEDON: EXDA