Does the gender of the enumerator impact data collection?

Our Motivation

In a day, we have many conversations – some that put us at ease to share our thoughts, and others that make us feel more reserved. In each of these conversations, depending on who we are speaking with, we experience different degrees of comfort in sharing our experiences, opinions, and concerns. 

Primary data collection, simply put, is a tailored 1:1 conversation taking place at scale. The data collected in this process helps not only capture the world as it stands at present, but also influences how it will be designed for the future. Policies, programmes, and systems that are designed using this data have tremendous potential – at best they can eradicate historically entrenched social inequalities and lead us toward a more equitable society or, at worst,  further perpetuate existing inequalities and create new forms of hierarchies. Therefore, the question of what influences the quality of data we collect is an important one that helps us in our journey towards truth, social impact, and equity. 

In our experience at IDinsight, having conducted numerous surveys, we have noticed that the quality of conversation between an enumerator and a respondent can be affected by various factors – the gender of the enumerator being one of them. To formally understand the relationship between enumerator gender and enumerator performance¹, we analyzed data from two of our DataDelta India projects. Our motivation behind undertaking this study has been to also help us understand the value of intentional and proactive efforts for recruiting and retaining female enumerators in the DataDelta team². To that end, through our analysis, we hope to gain insights to make informed decisions about budgeting, hiring, data collection, and training approaches in the future.

Our Learnings

1. Does lowering hiring considerations for prior work experience and educational qualifications have an impact on enumerator performance?

DataDelta follows a rigorous process for hiring local enumerators for its primary data collection projects. As shared in the first blog of the series, in order to hire more gender-inclusive teams, we relaxed some of our hiring requirements. These included relaxing considerations for motorcycle and driving licenses, in favor of public or rented transport; and for post-graduate degree and prior work experience, in favor of longer training with more field pilot sessions – both had proven barriers to women’s advancement in the hiring process. While such relaxations played a crucial role in hiring more representative enumeration teams; we understand that renting private vehicles and extending the number of days for training and pilot sessions comes with budgetary implications. This begs the question – beyond principle-based rationales of fairness, equity, and inclusivity; how do we justify these additional costs? 

At the minimum, we expect that relaxations in the hiring criteria should have a non-negative impact on data quality and survey efficiency for the project. To examine this, we used data from one of our recent survey projects, which included questions on female labor force participation, financial decision-making at home, social norms, and misinformation, among others. The gender of the enumerator was matched with the respondent during survey assignments. To measure efficiency, we compared the number of respondents surveyed by enumerators of both gendered subgroups using indicators of daily attempt rate, daily completion rate, and daily refusal rate. To measure data quality, we compared how each enumerator subgroup performed by looking at the accuracy of the data they collected, the proportion of outlined protocols they followed, the degree to which they made the respondent feel comfortable, and the speed at which they asked questions, protocol adherence rate, comfort score, and speed score³. As seen in Table 1, female enumerators performed at par with their male counterparts on all indicators. This illustrates that there was no negative impact of the revised hiring criteria for female enumerators on the speed at which the data is collected or its quality,⁴ further demonstrating that adequate training can enable candidates with limited opportunities to match the performance of their more experienced and formally qualified counterparts.

2. Are there specific benefits of having female enumerators?

The insights from our first case study prompted us to relax hiring requirements for female candidates. However, considering the budgetary implications of making proactive efforts toward hiring gender-inclusive teams, it becomes vital to move beyond non-negative impact and discern the conditions where hiring a gender-inclusive team can justify the cost by significantly improving the project outcomes, in terms of the efficiency at which the data is collected as well as its quality. 

To investigate this, we analyzed gender-disaggregated data from another recent DataDelta project in India⁵: a large-scale multi-respondent survey involving a variety of topics. In this survey, the modules covered household roster,⁶ demographics, livelihoods, income, general health, and education status were administered to the head of the household (typically male), while those focusing on women’s health, pregnancy care, and child nutrition were targeted toward female respondents of the household. With this survey design, we could assess how female enumerators performed across a broad spectrum of topics and when engaging with respondents of both genders. We analyzed metrics such as completion rate, rate of constraint violations, rate of “Don’t Know” and “Refused to answer” responses, and protocol adherence.⁷ Initially, these indicators were calculated for every enumerator at the level of each question and then aggregated at the module and respondent level. 

We observed that female enumerators performed significantly better than male enumerators across all the relevant indicators when interviewing female respondents on topics typically deemed gender-sensitive (Table 2). Given the hierarchical structure of Indian society, where family and social norms strongly influence women’s behavior (e.g. communication with a man outside the family is often prohibited or simply deemed unacceptable), such a finding isn’t surprising. Imagine being a new mother or pregnant woman who is approached by a male interviewer to talk about pregnancy care, maternal health, or natal care for the child. It’s totally understandable if your first instinct is to decline the survey altogether (i.e. lower consent rate). If you decide to proceed, the likelihood of being left alone with the interviewer is low, and soon the conversation can become uncomfortable.  Consequently, you might resort to saying “Don’t know” or “Refuse to answer” (i.e. lower completion), or you might feel pressured to give socially acceptable answers  (i.e. lower accuracy). But imagine if the interviewer were a female – in that case, societal norms or family prohibitions become less of a concern, making the interactions much smoother and more comfortable for both parties involved, ultimately resulting in higher-quality survey data.

Second, we observe that when interviewing household heads (74% of whom were male and 26% were female), they are generally seen to be doing better across all relevant indicators, but the differences are not always statistically significant meaning they could have occurred just by chance. Put simply, female enumerators are as effective as their male counterparts, if not better, irrespective of the respondent’s gender.  But there’s more to it!

Looking at the module level, we find that female enumerators consistently demonstrate higher completion rates, even in surveys where the questions are not particularly gender-sensitive or targeted to female respondents. Also, we find that female enumerators perform at least as well as their male counterparts for most aspects of data quality, if not all, when questions are non-gender-sensitive⁸ (Table 4). However, it’s important to note that this doesn’t mean female enumerators are inherently superior and should be exclusively preferred for collecting high-quality data on less gender-sensitive topics. Instead, it underscores the importance of recognizing the strengths and limitations that both male and female enumerators bring to the table. While we may not fully understand the mechanism behind the observed differences or lack thereof between the two gender groups, one possible explanation lies in the societal perceptions of women being less intimidating. Given this, one can imagine how it might be easier for women to access their target respondents, gain cooperation, and draw out sensitive and true details. Rather, it’s crucial to acknowledge that these stereotypes and social norms can also have opposing effects. For instance, male respondents may feel pressured to provide socially desirable answers or oversimplify their responses when interacting with female enumerators, assuming they may not grasp technical details as well.⁹

Way Forward

As a data and evidence organization, for us, primary data is about giving a voice to people’s experiences and aspirations in the programs and policies that affect their lives. Since many public sector policies and programmes are targeted at specific—usually, vulnerable—subgroups, it is important that we can design our data collection process in a manner that allows us to accurately reflect these groups’ perspectives. This includes being able to secure their willingness to participate in the survey (leading to higher completion rates), and their open and honest responses (leading to higher data quality). 

Our analysis in India has shown that female enumerators excel in creating these conditions– they collect high-quality data efficiently, on both gender-sensitive questions and others, from both female and male respondents. This is in-part a testament to female enumerators’ acumen for enumeration roles and in part a result of gender perceptions and social norms. As an organization working towards a more equal society, we see our work as contributing towards transforming gender perceptions and social norms in a manner in which people are perceived to truly be equal in their capacities. In the meantime, we owe it to our survey respondents—the bedrock of the policies, programmes, and interventions under study; to whom we ask the most intimate questions about their lives – to create conditions in which they feel safe and comfortable bringing their honest and complete selves to the conversation. Hiring female enumerators, even at relatively higher budgetary costs, is a crucial step towards fulfilling this responsibility towards our respondents.

As DataDelta continues our work to bring high-quality, representative data to social sector decision-makers, we will continue to be more inclusive and representative in our work, be it by gender or other axes of marginalization. As we do so, we will also continue to share our learnings with the sector and look forward to learning from others as well!

Appendix

The table below (Table 5) provides a detailed description of the indicators used in the mentioned research.

Acknowledgments

A big thank you to the DataDelta Gender & Intersectionality Working Group – Sarah Lucas, Krishanu Chakraborty, Priyanka Dua, and Shreya More – for their invaluable support and feedback throughout this study. Also, immense gratitude to the project team members – Shubhi Jindal, Syed Maqbool, Rohan Raj, and Utkarsh Gupta – for providing us access to and contextual understanding of the data. Special thanks to Puneet Kaur, Lenny Tin, and Emily Coppel for their support in refining this blog.

This article was originally published on IDinsight.

—

Footnotes:

1. Enumerator performance, here, includes two components–the quality of the data being collected, and the efficiency with which it is collected.
2. Some of these efforts have been highlighted in Part I of this blog series: Access, Opportunity & Enumerator Hiring in India.
3. Detailed definition of these indicators can be found in Appendix.
4. Due to limited data availability, we were not able to conduct significance tests for case study 1.
5. The two survey projects used for this study were conducted in different contexts, including regions and language. As such, our findings do not consider geographical or individual effects, and therefore, comparisons between the two case studies should be avoided.
6. Household surveys typically begin with the creation of a household roster, which is a list of household members along with their basic details which can include but are not limited to gender, living arrangements, and relation with the household head. A household roster can serve various different purposes for a survey. In our case, it was used to identify every person who will be considered a member of the household and determine which individuals can we consider as respondents for the latter modules.
7. Detailed definition of these indicators can be found in Appendix.
8. Male enumerators are found to be doing better with respect to the Rate of “Don’t Know” for Demographic and Education module. However, the difference in both cases is not statistically significant.
9. Axinn W. G. 1989. Interviewers and data quality in a less developed setting. Journal of Official Statistics 5:265–80