What does it take to design an AI tool that people use? For nonprofits, the answer may have less to do with the sophistication of the underlying technology and more to do with how well it understands the intended user.
ARMMAN and SNEHA, both working in maternal and child health, serve as useful examples. ARMMAN has built a WhatsApp-based AI assistant to help Auxiliary Nurse Midwives (ANMs) access clinical guidance at the point of care. Meanwhile, SNEHA’s chatbot is designed to help pregnant women and mothers access health information and ask health-related questions they may otherwise struggle to get answers to.
Although the two solutions use similar technology, their design choices look quite different. For ARMMAN, the focus has been on fitting into health workers’ existing workflows and providing concise and actionable information. SNEHA has instead focused on building women’s confidence to ask questions, embedding empathy into responses, and accounting for the realities of shared phones and limited digital agency.
Their examples illustrate that user-centred AI design has implications beyond making a tool easier to use. Understanding the user can change the channel, the format and tone of the interaction, the scope of the product, and even what the AI should be allowed to do.
Designing for the user’s actual context
For both organisations, the first design decision was about understanding the user’s lived reality.
ARMMAN recognised that an ANM was already operating in a highly digitised (and digitally burdensome) environment. Through interviews with ANMs in Uttar Pradesh and Telangana, the organisation found that they were already using between six and 15 apps and portals for digital data entry, often involving considerable duplication. Adding another standalone application risked increasing that burden. So, the organisation chose to meet ANMs on WhatsApp, a platform they checked multiple times a day as part of their work.
The chatbot was also designed as an extension of ARMMAN’s existing training programme. The organisation found that while classroom training produced knowledge gains of 10–20 percent, much of that knowledge would dissipate within six to eight months. The chatbot was therefore conceived as a point-of-care resource that could help ANMs apply protocols in real-world situations rather than relying only on periodic training.
SNEHA arrived at WhatsApp through a similar process, but its starting point was different. To understand phone access, the organisation first surveyed women in Bhiwandi, one of its intervention areas. Located in Maharashtra’s Thane district, the city is one of India’s largest power loom centres. However, the economic condition of its population has deteriorated over the years owing to consecutive communal riots and the gradual decline of the power loom industry.
SNEHA found that while 80 percent of households it surveyed had a phone, only 50–60 percent of women had their own smartphone. The organisation then looked at which applications women already used and found that WhatsApp was the one they were most comfortable with. Its previous experience sending health messages over WhatsApp and SMS during the COVID-19 pandemic also served as evidence of the channel’s viability.
As is evident, ARMMAN adopted WhatsApp to reduce friction for a user who was already digitally active. For SNEHA, choosing WhatsApp was a first step towards making digital health information accessible to women who could not access a phone independently or operate it confidently.
Even women who could read and write often struggled to process a block of written text.
The differences became even clearer once the organisations began testing the solutions. ARMMAN used a prototyping technique called ‘Wizard of Oz testing’. They built a scripted version of the chatbot and had ANMs interact with what they believed was an automated system but was actually a human responding. This helped the team test how ANMs wanted to communicate with the chatbot. They found, for instance, that voice functionality allowed ANMs to articulate clinical questions more effectively than typing. They also learned that the responses needed to be concise and actionable rather than elaborate.
When SNEHA tested their chatbot with pregnant women and mothers, it found that the implications of voice were quite different. Even women who could read and write often struggled to process a block of written text. They reported a sense of comfort with voice responses, as it made them feel like the answer was being explained to them. As a result, the organisation decided that women would receive both audio and text responses regardless of whether they asked their question by voice or in writing.
The differences also extended to the content and tone of the responses. ARMMAN found that ANMs wanted crisp and actionable answers that could help them make decisions. SNEHA found that pregnant women and mothers responded better to warm and empathetic messaging, with reassurance built into the interaction.
SNEHA also found that access to technology did not necessarily translate into the confidence or ability to use it. Many struggled even to think of a question to ask the chatbot. This exposed what the organisation’s CEO, Vanessa D’Souza, describes as a “lack of digital agency.” “It’s not just a question of giving a woman an app,” she says. “You have to build her confidence to think of a question and then use the phone to ask it.”
The organisation consequently changed its induction process, instructing frontline workers to demonstrate interaction with the chatbot, prompt women to identify something they want to know, and support their first interactions with it.
Letting users redefine the product
Neither organisation’s understanding of the user stopped at the initial design stage. As they observed user interaction with the chatbots, their conception of what they should offer evolved. ARMMAN discovered that its initial definition of the chatbot’s role was narrower than the ANMs’ understanding of their own work. The tool had been designed to provide guidance on high-risk pregnancy, but when ANMs began using it as a job aid, they expected it to answer questions across the range of issues they encountered in the field. This included routine pregnancy, delivery, postpartum care, child health, and immunisation. As Amrita Mahale, ARMMAN’s Product and Innovation Director, puts it, “Once the ANM views a chatbot as a job aid, she expects it to answer questions related to her job. And her job is not limited to high-risk pregnancies alone.”
In its pilot in Uttar Pradesh, about 35 percent of questions the ANMs asked were initially classified as out of scope for the chatbot. But as the organisation expanded the knowledge base in response, that proportion fell to around 5 percent. ANMs thus not only helped ARMMAN improve the chatbot’s answers but also changed the organisation’s understanding of what the chatbot needed to be.
User feedback also led to new functionality. ANMs began asking ARMMAN for posters and videos that they could show pregnant women during counselling. This suggested that they did not see the chatbot only as a source of information for themselves, but also as a resource they could use in their interactions with mothers. The organisation responded by developing bite-sized multimedia content that ANMs could use during counselling.
Users can therefore reveal needs that an organisation did not anticipate when first defining a tool. But organisations still have to interpret what they are hearing and decide which changes genuinely strengthen the product and fit its purpose.
Turning user feedback into product decisions
The value of user feedback also lies in creating a process for turning it into improvements. Both ARMMAN and SNEHA built multiple feedback mechanisms into their AI solutions, combining what users explicitly say with what their behaviour reveals.
ARMMAN, for instance, combines direct feedback from ANMs with usage data, analysis of the kinds of questions being asked, and expert review. Questions that are classified as out of scope are reviewed by doctors or public health professionals. These reviews can result in changes to the knowledge base or refinements to the model and prompts.
Crucially, these insights can feed back into the wider programme, not just the chatbot. When ARMMAN saw that ANMs were asking many questions about gestational diabetes—despite it not being part of the first phase of training in Uttar Pradesh—it shared the data with state health officials. Information on diabetes was subsequently included in the second phase of training.
Both organisations note that what users say and what they do can reveal different things.
SNEHA similarly brings together several sources of information: user surveys, focus group discussions, chatbot conversations, and programme and engagement data. This combination reveals what any single source might miss. For example, SNEHA initially sent digital nudges to women in Bhiwandi every Friday. The assumption driving this decision was that since the women largely belonged to Muslim households, they would have greater access to smartphones on Fridays as their husbands were more likely to be home. However, discussions with the women revealed that they misunderstood the nudges and believed that they could only interact with the chatbot on Fridays. SNEHA therefore changed the design of the nudge, clearly indicating that they could ask questions whenever they wanted to. Thereafter, women began sending messages on all days of the week.
Both organisations note that what users say and what they do can reveal different things. SNEHA, for example, found that women responded positively when offered videos, but the actual views were low. Focus groups revealed that their limited mobile data packages, rather than a lack of interest in videos, was the problem. The organisation consequently adapted not just the content but the medium through which it was delivered.
ARMMAN’s experience with audio queries illustrates that technical improvements may not always translate into greater adoption either. When its chatbot was first piloted in Uttar Pradesh, audio accounted for around 10 percent of queries. However, this number fell to 2 percent soon after. This was because audio queries had higher error rates, and users also reported that the wait times felt too long. The team therefore improved the model’s performance and reduced latency, but this did not immediately increase audio usage. Further user research revealed the need to educate users about the feature itself. After another round of user education, audio adoption increased to 12–20 percent across languages.
Knowing where AI should (and shouldn’t) be used
Both organisations also note that with an AI solution, it is important to recognise where it adds value and where human judgement needs to remain central.
For ARMMAN, this has meant thinking carefully about how much users should trust its chatbot. During early testing, ANMs sometimes assumed that they were communicating with a person, not an automated chatbot. ARMMAN recognised that this could lead users to be overly trusting of clinical information provided by the chatbot. The organisation now makes it clear that the chatbot is automated, can make mistakes, and should be used alongside the ANM’s own judgement. Complex or unsatisfactory interactions can also be escalated to a human expert.
SNEHA’s example, on the other hand, illustrates why predefined boundaries can be critical, depending on the context in which a tool is deployed. The organisation observed that women began asking the chatbot about sexual health, infertility, and domestic violence. While this revealed genuine information needs, the organisation did not immediately respond by expanding the chatbot’s scope to cover these topics. In some cases, the team felt that a question fell outside what it wanted the chatbot to address, or that it did not have the expertise to answer it responsibly. In the case of domestic violence, the decision not to include such content was driven by the sensitivity of the issue and the risks associated with women potentially using a shared phone. “Though we have all the material related to domestic violence,” says Vanessa, “including it could actually jeopardise the woman if there’s a conversation about it [that is seen by someone else].”
These choices point to an important lesson: Responsible AI design also involves deciding what not to automate. The objective is not to create a system that can answer every question, but one that is useful within clearly understood boundaries—and has human support when those boundaries are reached.
Both organisations’ experiences suggest that building useful AI for social impact is ultimately about developing the discipline to keep learning from the people it is meant for.
SNEHA and ARMMAN had different intended users, which led to very different design choices. They then tested their assumptions, allowed users to challenge the initial scope of their products, and built processes to learn from both successful and unsuccessful interactions.
Just as importantly, they have had to recognise the limits of what an AI system can safely do. For SNEHA, this has meant deliberately moving slowly where the risks are unclear. For ARMMAN, it has meant maintaining human oversight and ensuring that users do not mistake the chatbot for an infallible expert.
There’s no single blueprint for designing an AI solution, but what connects these experiences is the acknowledgment that user-centred AI design is an ongoing process. It involves understanding the user’s context, testing your assumptions, learning from actual use, and shaping the technology around the user, not the user around the technology.
—
Know more
- Read the following case studies on ARMMAN and SNEHA on the Digital Toolbook for Social Impact platform to better understand how they leverage AI in their work.
- Read this article on building and deploying AI in the social sector.
