There has been a fair bit of hoopla around India being declared open defecation free last month. In the media debates, the measurement of India’s open defecation-free status has come under a good deal of scrutiny. Leaving aside the politics of the debate, there remains an important question: how can latrine use be measured rigorously?
Those working in the sanitation space have known for a long time that the measurement of latrine use is prone to inaccuracy and self-reporting bias. If you ask someone about whether they are using the latrine, social desirability bias may push them into saying, ‘yes’. So when 3ie started the Promoting Latrine Use in Rural India evidence programme, we decided that it was important to focus on the tricky business of measuring latrine use.
The measurement of latrine use is prone to inaccuracy and self-reporting bias. | Photo courtesy: Charlotte Anderson
Related article: Moving beyond just building toilets
The 3ie evidence programme supported four impact evaluations of low-cost, behaviour science-informed interventions to promote latrine use in rural Bihar, Gujarat, Karnataka and Odisha. 3ie, along with our thought partner, the Research Institute for Compassionate Economics and the four research teams from Oxford Policy Management, London School of Hygiene and Tropical Medicine, Eawag-Wateraid and Emory University came together to collectively think about ways we could measure latrine use rigorously. All the teams worked on a standard list of latrine use questions and used a variety of measurement tools. We also had an independent latrine use measurement team, led by IFMR, test two different survey modules amongst a sub-sample of around 2,400 households in study areas.
We have learned a lot from this measurement project and the four impact evaluations. Here are some highlights.
The how and who aspect of measurement
The independent measurement team’s big finding from the baseline survey was that how and who you ask the question influences latrine use estimates. They tested a balanced, individual-level question (that 3ie and research teams developed) with 1,215 households and a household-level question used in the National Family Health Survey (NFHS) with another 1,216 households. The standard NFHS question was, ‘What kind of toilet facility do members of your household usually use?’ and offered various toilet options, with the last one being open space or field. The question in this case and the ordering of responses seemed to favour latrine use.
Individual-level, balanced latrine use question found significantly higher rates of open defecation than the household-level question.
The individual-level question that 3ie and the research teams worked on and tested was: ‘I have seen that some people defecate in the open, and some people use the latrine. Now I want to ask about where you and your family members defecate. The last time (name of household member) defecated, did (name of household member) defecate in the open or use the latrine?’ Unlike the NFHS question, the individual question balanced open defecation and latrine use behaviour in the priming statement and the behaviour question. It also mentioned a clear reference period in the question.
The results show that the individual-level, balanced latrine use question found significantly higher rates of open defecation than the household-level question (see figure below). This finding highlights the need for carefully phrasing the question and capturing intra-household variation to measure latrine use more accurately.
The sequence of questions can also make a difference. The Gujarat impact evaluation team used a physical activity questionnaire where going for open defecation was one of many questions related to different physical activities such as farm work, cattle rearing and so on. Respondents may have felt less social pressure through this tool because they did not perceive its primary purpose to be measuring latrine use behaviour. The physical activity tool found reported toilet use to be 4.4 percentage points lower than toilet use reported through the individual-level questionnaire focussed on latrine use.
When the measurement is done
Asking the question in summer can yield different responses compared to the winter and monsoon seasons. In Karnataka, study participants reported lower latrine use in the summer months due to poor availability of water. The takeaway here is that if data collection occurs in different seasons, it is important to keep in mind factors, such as water availability and preference for using the latrine when it is raining and open defecation is inconvenient.
It’s not just the seasons that can change answers on latrine use, it is also the reference period in the question. In Bihar, 82.5 percent of the study participants reported using the latrine the last time they defecated but only 76.9 percent of households reported latrine use over the last three times. Measurements need to capture both occasional and regular open defecation.
Understanding factors that complicate comparisons and analysis
The behavioural interventions in Karnataka, Gujarat and Odisha had small positive effects on latrine use. But the Indian government’s Swachh Bharat Mission (SBM) activities contributed to a surge in latrine use between baseline and endline across both treatment and control groups in all the study areas. The intensive government activities made it difficult for the impact evaluation to isolate the effect of the intervention. It was also not possible to assess whether SBM was consistently implemented across treatment and control areas. With all of the buzz SBM generated around latrine use, study participants in both arms may also have been primed to give socially desirable answers.
Measuring the effect of the behavioural interventions that focussed on correctly estimating pit filling rates in Bihar became daunting as latrine specifications varied. The study population had pits of varying dimensions that did not meet standard specifications: 28 percent of households reported pits with the depth of 5 feet, 37 per cent reported pits with depths of less than 5 feet, and 35 percent reported pits of depths more than 5 feet. The quality of pit construction was yet another complicating factor. Qualitative research in Bihar showed that some respondents were simultaneously using both pits in their twin-pit construction. This meant that information on the self-emptying aspect of twin pits was therefore irrelevant for such households. These issues highlight the importance of formative testing for generating quality baseline information on latrine specifications.
Triangulation matters
Related article: How to use data to improve decision-making
Using surveys, latrine observations and qualitative research to triangulate and verify measurements is important. A latrine observation index, for instance, can be used to validate the results of a latrine use survey. Such an index may include several indicators of likely use based on the physical state of the toilet, e.g., presence of water container and cleaning supplies in the toilet. In Bihar, observation data showed that latrine use amongst the study population was 90.7 per cent which was in line with the endline survey findings. Latrine observations cannot indicate individual variations within a household. However, the finding from the Bihar study does still highlight that social desirability bias may not have been an issue with the responses to the administered survey.
What have we learned?
Measuring latrine use is indeed quite tricky and is influenced by a host of factors. The phrasing and ordering of questions, the respondents’ perceived intent of the question, the season and the time period over which responses are asked for, can influence estimates. Anyone trying to measure the highly private behaviour of latrine use needs to understand how biases influence responses.
Anyone trying to measure the highly private behaviour of latrine use needs to understand how biases influence responses.
For impact evaluations, formative testing and generation of sound baseline information on sanitation systems and other ongoing activities becomes crucial for quality analysis. Triangulation through multiple methods can help not just with more accurate estimates but also with understanding the mechanisms driving behaviour. If we truly want to achieve sanitation for all, we must begin by bringing in more rigour into the measurement of latrine use.
(With input from Charlotte Lane)
This article was originally published on 3ie’s blog: Evidence Matters.
