qualKit Home
Approaches
Techniques and Tools
Inspiration
French

Untangling significance: A simple framework to support descriptions of change

Jess MacArthur and Tamara Megaw

Assessing complex change is difficult and nuanced. One such example of complex change includes gender transformative change – that seeks to transform gender dynamics towards equality by focusing on the systems and structures that perpetuate inequalities. Other forms of complex change include systems change and resilience. Measurement of such complex change often leads researchers and evaluators to participatory, qualitative, and creative methods. However, the interpretation of stories, images, and other qualitative descriptions of change remains challenging. In this blog post, we introduce a simple framework to help unpack descriptions of change and provide two examples of its tailored use.

Social development programs, such as those promoting gender equality, often rely on qualitative forms of evaluation. These qualitative descriptions of change regularly engage stories, observations, and images crafted through open-ended questionnaires, interviews, and focus groups. To move from data to sense-making, these descriptions must be placed in perspective with broader program objectives and goals.

To support exploration of descriptions of transformative change, I (Jess MacArthur) designed and piloted a novel framework. This framework is used in the data collection and interpretation phases of evaluation, helping to illuminate different perspectives between the evaluator and respondents. The framework also aims to honour respondents’ knowledge within their communities, offering an opportunity for co-evaluation. The framework builds on Jabeen’s (2018) framework of unintended outcomes and Waffi’s (2017) creative participatory self-evaluation methodology. The framework can be used by research participants, program staff or both to help unpack significance.

The framework follows the acronym ‘VOICE,’ which conveys a desire to listen to various voices throughout the assessment process. Each letter represents a different aspect relevant to making sense of change, with specific reflections on unintended outcomes. The final letter ‘E’ has two aspects associated with it. Further piloting of the framework has added two optional aspects: ‘setbacks’ and ‘risks’.

Download Framework English [PDF, 372 KB]

We highlight two pilot examples of the framework. First a qualitative mid-line evaluation of a gender mainstreaming component for program staff of a sanitation-focused project in Cambodia (October 2020). And second, to explore transformative social accountability outcomes in an inclusive sanitation-focused project in rural Bangladesh (2022).

1

In Cambodia, the framework was used as part of the photo submission template for a photovoice activity, with program staff of a sanitation program by Jess MacArthur. Participants were asked to respond to each aspect through a simple multiple-choice question and were asked to ‘code’ their submitted photos using a three-point scale. Responses were summarised in a simple dashboard as depicted below.

 

2

In Bangladesh, the framework is being used be used to assess of gender-transformative social accountability outcomes in an inclusive sanitation-focused project in rural Bangladesh (2022). The framework was adapted by Tamara Megaw to reflect that transformative change is a long-term goal and some progress in some domains may be accompanied by setbacks in others (Hillenbrand et al. 2015). In some contexts, when challenging power relations, there can be a risk of backlash and unintended negative consequences that need to be recognised and addressed early. The setbacks and risk questions were added in this phase.

Drawing on these two examples, we have identified four uses for the VOICE(-SR) framework as a starting point.

  1. Conversation starters to unpack descriptions of change during data collection – The framework aspects can be used as probe questions with participants in interviews or focus groups that follow-up on a reported change. This is the arrangement for the Bangladesh example.
  2. Targeted questions to quantify descriptions of change during data collection – The framework can be used to elicit ‘participatory coding’ (using Likert or 0-10 scales) with participants in interviews, focus groups, or in self-administered surveys. This was used in the Cambodia example.
  3. Analysis of themes to unpack stories of change – The framework can be used as an analysis scaffolding for evaluation teams to enrich descriptions of change. 
  4. Illuminating differences between evaluators and respondents – Bridging the first three uses, the framework can also compare how the research teams’ analysis overlaps and diverges from research participants and evaluator views.

Are you interested in using the VOICE(-SR) framework? What other tools have you used to make sense of stories or images of complex change? We would love to hear from you: jessica.macarthur@uts.edu.au.

Explore other techniques and tools

Photovoice

Photovoice

Photovoice   Photovoice is an interactive photography training and participatory approach, where participants are asked to create photos of changes they observe within the program with their own smartphone cameras.  Related Tools Inspiration While focusing...

Back to UTS-ISF's Water for Women Home Page

DISCLAIMER 

This project has been funded by the Australian Government through the Department of Foreign Affairs and Trade. The views expressed in this website are the author's alone and are not necessarily the views of the Australian Government.