Halfway through the project, we have almost finished the ethnographic fieldwork. More on that later. The second, explanatory part of the research has now begun, and we would like to give a short update on that. Apologies to the casual reader: this will get a bit technical.
The aim of our impact evaluation is to test and refine theories about how support for community-led conservation contributes to protecting forest cover. The first thing to say is that reduced deforestation is not the only or necessarily most important outcome of custodianship. But impact evaluation is complicated, and we would rather focus on evaluating one impact area as well as we can than assess a holistic set of goals superficially.
The reason for choosing forest cover and not, for example, justice or well-being, is that there is consistent quantitative evidence that supporting community-led conservation helps reduce deforestation (and environmental loss more broadly), but a lack of evidence on how. Most recently, a study on the impact of the COVID-19 pandemic on protected forests in West Kalimantan compared deforestation in villages affiliated with the NGOs Yayasan Planet Indonesia and Alam Sehat Lestari, to unaffiliated villages. Analysis of satellite imagery found that average forest loss in affiliated villages was 52% lower before the pandemic, and the difference increased to 68% during the pandemic. Evidence from household surveys suggested that these effects were related to support for healthcare and livelihoods.
Designing a qualitative impact evaluation
We aim to take a deeper look at those and potential other impact pathways, to strengthen programme design, guide scaling efforts across Indonesia, and contribute to the global evidence base for community-led conservation. To design this study, we had to learn about qualitative impact evaluation. It turns out there are many different methods and approaches, and it was quite a journey to try to understand their respective strengths, weaknesses, and requirements. (Thanks to all the experts who have shared their time and helped us think through the options!)
In the end, we decided to adopt a bricolage design (Aston & Apgar, 2022) combining a Realist Evaluation (RE) framework with Process Tracing (PT) case studies. The combination is motivated by the complementary strengths of each method.
RE (Pawson & Tilley, 1997) conceptualises causation generatively: interventions do not cause outcomes directly but activate “mechanisms”: the reasoning and actions of specific actors in specific contexts. RE is well-suited to generating ‘mid-range’ theories, applicable under specific conditions rather than universally, that can guide context-sensitive scaling decisions. These theories are formalised as Context-Mechanism-Outcome (CMO) configurations, developed and refined iteratively across multiple rounds of evidence. While RE structures our theories and helps identify which mechanisms are likely active where, it offers limited guidance on how to test specific theories.
We are therefore using PT case studies to test a subset of the most promising theories in depth (Collier, 2011; Beach & Pedersen, 2013; Punton & Welle, 2015). PT provides formalised tests to guide the collection and weighing of evidence, which enables us to increase or reduce our confidence in specific theories. To do this, each hypothesised mechanism is divided into its necessary constituent parts, each specified as an actor and an action. Observable manifestations are defined for each part, and each piece of evidence is assigned preliminary inferential weights on two dimensions. Certainty denotes the probability that the evidence would be found if the mechanism operated as theorised. If we look for a piece of evidence assigned “high certainty” and we fail to find it, it means that our theory is probably false. This is sometimes called a “hoop test”. Uniqueness denotes how unlikely the evidence would be if the mechanism were not operating. If we look for a piece of evidence assigned “high uniqueness” and we find it, we can conclude our theory is probably correct, because there is no other plausible explanation for that evidence being there. This is sometimes called a “smoking gun”.
Our PT case studies are of the “theory-testing” rather than “outcome-explaining” kind (Beach & Pedersen, 2013). We ask whether a hypothesised mechanism was present and operated as theorised, not how much of the observed forest change each possible cause accounts for. Systematically listing and eliminating rival explanations is a different strategy, General Elimination Methodology (GEM), which is sometimes conflated with PT. GEM does not seem useful for our study, because we expect that multiple different mechanisms will contribute to forest change dynamics in every case, including mechanisms that are external to the intervention.
We shall nevertheless document these additional mechanisms as they emerge in the research process, because (1) they help evaluate the certainty and uniqueness of evidence for each part of the process, and (2) because they may inform future quantitative studies. One important consideration is whether any explanations for reduced deforestation are not caused by but nevertheless correlated with the partnership. For instance, if villages with conservation-minded leadership are both more likely to establish conservation partnerships and less likely to lose forest, this would be a factor (“confounder”) that future quantitative evaluations should account for.
The months ahead
The study proceeds through five iterative rounds of theory development and testing, followed by synthesis.
Rounds 1–3 (May–August 2026): Theory development. We shall develop an initial set of theories based on organisational documents and scientific literature on community-led conservation (Round 1 -> see Figure 1 for the current set). Then we shall test and refine the theories through semi-structured interviews with YPI senior staff (Round 2, ongoing), and consultation of existing organisational data from programme monitoring and evaluation, participatory impact assessments, and ethnographic studies conducted by the research team in four implementation sites (Round 3). Each round produces an updated set of configurations with associated evidence and preliminary confidence levels.
Rounds 4–5 (August–October 2026): Process tracing case studies. We shall select two sites based on: (a) time elapsed since establishment of partnership; (b) contextual alignment with the strongest CMO configurations, and (c) the existence of ethnographic data for that site. Each case study involves a review of programme MEL data, and semi-structured interviews with YPI implementation staff, government stakeholders, and community members. Interview respondents will be selected based on their ability to provide information that confirms or disconfirms specific parts of the theorised causal mechanism.
Synthesis (November-December 2026). We shall compile a final set of theories, in the form of CMO configurations, with associated evidence and confidence levels. We shall evaluate the implications of our findings for the theory, practice, and evaluation of community-led conservation more generally.
Thank you for making it to the end of this post! Comments and suggestions are always welcome.
References
Aston, Thomas, and Marina Apgar. 2022. The Art and Craft of Bricolage in Evaluation. Institute of Development Studies. doi:10.19088/IDS.2022.068.
Beach, Derek, and Rasmus Brun Pedersen. 2013. Process-Tracing Methods: Foundations and Guidelines. Ann Arbor: The University of Michigan Press.
Collier, David. 2011. ‘Understanding Process Tracing’. PS: Political Science & Politics 44(4): 823–30. doi:10.1017/S1049096511001429.
Hopkins, Skylar R., Ashley Hazel, Julie D. Pourtois, Andrew J. Chamberlin, Zachary Gajewski, Ian Harryman, Susanne H. Sokolow, et al. 2026. ‘Pandemic Impacts on Protected Rainforests and Rural Communities with Variable Health and Livelihood Support in West Kalimantan, Indonesia: A Mixed-Methods Approach Combining Household Surveys and Remote Sensing’. The Lancet Planetary Health 0(0). doi:10.1016/j.lanplh.2026.101455.
Pawson, Ray, and Nick Tilley. 1997. Realistic Evaluation. SAGE.
Punton, M., and K. Welle. 2015. Straws-in-the-Wind, Hoops and Smoking Guns: What Can Process Tracing Offer to Impact Evaluation? The Institute of Development Studies and Partner Organisations.