Effective Tools and Techniques for Conducting Impact Assessment

Every public health programme, whether it tackles maternal nutrition, immunisation, or adolescent reproductive health, eventually faces one defining question: did it actually work? Answering this requires more than counting beneficiaries or tallying expenditure. Impact assessment is the disciplined process of measuring real, attributable change in people’s lives, and the credibility of that assessment depends almost entirely on the tools used to gather evidence. From structured interviews in a village household to participatory mapping exercises with community members, each technique brings a unique lens to the table. Understanding when and how to use these tools is what separates a superficial review from a meaningful evaluation.

Why the choice of tool matters in impact assessment 🔗

Impact assessment sits at the intersection of research, accountability, and policymaking. Numbers alone cannot explain why a programme succeeded in one district and failed in another, and stories alone cannot prove how many people benefited. A robust evaluation therefore blends quantitative reach with qualitative depth, capturing both what changed and why it changed. The World Health Organization notes that this kind of systematic appraisal uses quantitative, qualitative and participatory techniques to judge the effects of a policy or programme on a population, particularly on vulnerable groups.

For population and family health, where outcomes such as fertility decline, infant mortality, or contraceptive prevalence unfold over years, the choice of tool also shapes what counts as evidence. A poorly designed questionnaire can miss the social pressures behind early marriage. A rushed focus group can amplify the loudest voice and silence the marginalised. The sections below walk through three families of tools that, used together, produce a far more honest picture of programme impact than any single method on its own.

Interviews and questionnaires: the backbone of data collection 🔗

Interviews and questionnaires are the most widely used instruments in impact assessment because they can be standardised, scaled, and adapted to almost any topic, from breastfeeding practices to satisfaction with anganwadi services. Their power lies in their flexibility, but that flexibility is also where most evaluations go wrong.

Structured interviews 🔗

A structured interview follows a predetermined schedule of questions, asked in the same order and wording to every respondent. This consistency is what allows evaluators to compare answers across thousands of households without introducing interviewer-specific bias. The National Family Health Survey (NFHS), India’s flagship demographic and health survey, relies heavily on this approach. Trained fieldworkers conduct face-to-face interviews using rigorously translated questionnaires, with field supervisors performing back-checks to verify accuracy. Such interviews are best when the evaluator already knows what to measure, indicators like age at first birth, immunisation coverage, or out-of-pocket health expenditure, and needs comparable data from a large sample.

Semi-structured and in-depth interviews relax these rules. They use a guide of broad topics but allow the interviewer to probe further when something interesting emerges. These are invaluable when evaluating sensitive or complex behaviours, such as menstrual health practices, decision-making within a marriage, or the lived experience of a tuberculosis patient. They produce narrative data that helps explain the patterns seen in the larger survey.

Designing effective questionnaires 🔗

A questionnaire is only as good as the thinking behind each question. Common pitfalls include double-barrelled questions (“Are you satisfied with the quality and timeliness of services?”), leading wording, vague time references, and culturally insensitive phrasing. In India, where surveys are routinely conducted in more than a dozen languages, translation introduces another layer of risk. NFHS, for instance, organises multi-stakeholder workshops to refine sensitive questions on topics like sexual behaviour and domestic violence before finalising the instrument.

Good questionnaire design also balances closed-ended items, which produce quantifiable data, with a few open-ended prompts that capture nuance. A pilot test on a small group similar to the target population is non-negotiable. It exposes confusing phrasing, identifies questions that respondents refuse to answer, and helps estimate interview time. Even seasoned evaluators are sometimes surprised by how a question they thought was clear gets misinterpreted in the field.

The interviewer effect 🔗

One often-overlooked element of impact assessment is the interviewer themselves. Studies of NFHS data have shown that the characteristics, training, and behaviour of fieldworkers can account for around 32 percent of the variation in the reporting of intimate partner violence. This is a striking reminder that even a perfectly designed questionnaire can produce flawed data if the people administering it are not adequately trained, supervised, and matched to the respondent’s context. Gender, language, and age of the interviewer all matter, especially when the questions touch on stigma, shame, or fear.

Focus group discussions and surveys: depth meets breadth 🔗

If interviews and questionnaires are the workhorses of evaluation, focus group discussions and surveys are the strategic pairing that gives an impact assessment its full shape. One offers depth, the other breadth, and the best assessments deliberately combine the two.

What focus group discussions reveal 🔗

A focus group discussion (FGD) brings together a small group of participants, typically six to ten, who share a relevant characteristic, such as adolescent girls in a particular block, or new mothers enrolled in a nutrition scheme. A trained moderator guides them through open-ended questions, allowing group dynamics to surface ideas, disagreements, and shared experiences that individual interviews might miss. As one published review of focus group methodology in health research notes, FGDs are particularly useful for investigating perceptions, attitudes and behaviours, because the group setting helps participants articulate views they may not have fully formed on their own.

For population and family health, FGDs are excellent for understanding why a programme is working or stalling. Why are women in one cluster refusing the IUCD? What do men think about vasectomy camps? How do mothers-in-law shape feeding decisions? These questions rarely yield honest answers on a written form, but they often surface in a well-moderated conversation with peers.

The strengths and limits of surveys 🔗

Surveys, by contrast, are about scale. A well-designed sample survey can produce district-level or even sub-district-level estimates that guide resource allocation across millions of beneficiaries. NFHS-4, conducted in 2015-16, was the first round to produce district-level estimates for indicators ranging from anaemia to institutional delivery, transforming how impact is measured in the country.

Surveys excel at producing comparable, numeric data: percentages, ratios, trends over time. Their weakness is that they capture what people say they do, not necessarily what they actually do, and they struggle to explain causation. A survey can tell you that contraceptive use rose in a block by twelve percentage points, but it cannot reliably tell you whether the rise was due to a specific scheme, a new ASHA cadre, or unrelated socioeconomic change.

Mixing methods for stronger evidence 🔗

The most credible impact assessments treat surveys and FGDs as complementary rather than competing. A typical sequence begins with FGDs to identify locally relevant questions and language, followed by a quantitative survey to measure the prevalence of issues at scale, and concluding with another round of FGDs to interpret puzzling findings. India’s own Demographic and Health Survey programme, of which NFHS is a part, has explicitly invested in qualitative and mixed-method research to strengthen the validity and reliability of its quantitative surveys.

Participatory approaches: putting communities in the driver’s seat 🔗

Traditional impact assessment treats the community as a source of data. Participatory approaches treat the community as a co-evaluator. This shift in stance changes both the kind of evidence collected and the legitimacy of the conclusions drawn.

Participatory Rural Appraisal and Participatory Learning and Action 🔗

Participatory Rural Appraisal (PRA), developed largely through the work of Robert Chambers, is one of the most influential families of participatory tools. It includes transect walks, social and resource maps, seasonal calendars, matrices, and Venn diagrams drawn using locally available materials. Villagers themselves draw maps showing where families live, where water sources are located, and which households are most affected by ill-health. A timeline activity helps the community recall key events, droughts, epidemics, scheme launches, that shape the context of any intervention.

The evolution of PRA into Participatory Learning and Action (PLA) has been particularly valuable in health programmes. As researchers studying community participation in rural health note, PLA approaches help shift villages from passive to active community participation and can be a powerful intervention for addressing health issues at the village level when used logically and patiently.

Examples from Indian impact assessment 🔗

Participatory methods have shown real value in Indian health evaluations. A community mental health study in the upper Yamuna valley of Uttarkashi used 28 in-depth interviews and 10 participatory rural appraisal meetings with 120 community members to map assets, needs, and priorities. The resulting themes, organised around place, people, and practices, gave policymakers a far more grounded understanding than a standard survey could have produced.

Such methods are especially relevant for tribal areas, urban slums, and remote districts where formal data systems are weak and where outsider-led surveys often miss the most marginalised. The stakeholder identification stage of any health impact assessment is itself a participatory exercise, drawing in developers, planners, frontline workers, and the most vulnerable members of the community to ensure their voices shape the analysis.

Strengths and cautions 🔗

Participatory tools build local ownership, surface knowledge that experts simply do not possess, and tend to produce recommendations that are politically and practically feasible. But they are also resource-intensive, hard to scale, and vulnerable to capture by local elites if facilitators are not skilled. Their value is greatest when paired with quantitative methods, not when treated as a replacement for them.

Putting the tools together 🔗

No single tool can answer all the questions an impact assessment must address. A thoughtful evaluator usually begins with secondary data and stakeholder consultation, designs a survey informed by initial focus groups, supplements it with in-depth interviews of key informants, and validates the findings through participatory exercises with the community. This mixed-method architecture is now widely seen as the standard for credible impact evaluation, especially where interventions are complex, contexts are diverse, and consequences can be positive, negative, intended, unintended, direct and indirect.

For students preparing to work in population and family health, the practical takeaway is simple. Master the basic logic of each tool, learn to write questions that are clear and unbiased, practise moderating a conversation without dominating it, and never lose sight of the fact that data is ultimately about people. The best impact assessments are not just technically rigorous; they are respectful, inclusive, and honest about what they can and cannot prove.

What do you think? If you were evaluating a maternal health scheme in your own district, which two tools would you combine first, and why? And how would you protect the voices of the most marginalised respondents from being lost in the larger numbers?

References

1. https://pmc.ncbi.nlm.nih.gov/articles/PMC11963154/
2. https://www.who.int/health-topics/health-impact-assessment
3. https://www.nfhsiips.in/nfhsuser/nfhs3.php
4. https://www.sciencedirect.com/science/article/pii/S2352827323002227
5. https://www.sciencedirect.com/science/article/abs/pii/S0378378219301847
6. https://pmc.ncbi.nlm.nih.gov/articles/PMC9550650/
7. https://www.niti.gov.in/sites/default/files/2023-03/HEALTH%20SURVEYS%20IN%20INDIA%20REVIEW%20AND%20RECOMMENDATIONS.pdf
8. https://www.egyankosh.ac.in/bitstream/123456789/89775/1/Unit-4.pdf
9. https://pmc.ncbi.nlm.nih.gov/articles/PMC9207712/
10. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9278393/
11. https://www.who.int/tools/health-impact-assessments