Rethinking Nonprofit Evaluation
In this module, we focus on the ways evaluation processes, especially those related to data, implicate equity and how designing evaluations around responsible data practices can help an organization further their mission and increase their impact. We will cover data responsibility and the data lifecycle, and provide an introduction to evaluation through an equity lens.
Questions to ask at each stage of the data lifecycle
Collect
- Who defines the problem and data needed?
- How is collection done and by whom?
- What does consent look like and how is it obtained?
- How will you collect sufficient data to disaggregate by groups and intersections?
- How will you collect data on structural limitations?
Analyze
- Who does the analysis?
- How can you involve participants in the analysis?
- What protocols? Are pre-built tools/measures being used and how are they audited for bias?
- How can you lay out data limitations clearly?
- Frame findings in terms of systemic issues; don’t just collect data on people in the community, include data on availability/lack of community resources
Store
- What types of data need to be redacted, anonymized, or removed?
- Is encryption neededWhat are the protocols and policies for staff/vendor access to the stored data?
Access
- Will users have free access to raw data? To analysis?
- If there is a price, how will it be determined and charged?
- Are there licenses needed?
- Will data be shared with full rights to manipulate and re-share?
- What liability protections are in place regarding external use/analysis
- Are data accessible all the time or only one-time?
Secure
- Are all data sets protected under the same guidelines?
- When are exceptions made?
- What steps are in place to prevent and respond to data breaches?
Destroy
- How do retention/destruction policies align with organizational mission?
- How do retention/destruction policies align with the risks to different stakeholders?
Non-Profit Evaluation Worksheet
Conducting evaluations with an equity lens takes time; it is an ongoing, learning process that may lead to (or require) deep change at your organization.
This worksheet sketches out a process that alone may take two hours to complete. This is not to scare you off. It is to provide a realistic introduction to what is involved.
Additional Resources
The Stanford Digital Civil Society Lab has identified a few key resources to help you think through questions of security, access, encryption and many other topics that may arise as you work through the data lifecycle. In addition to the resources listed here, we have also compiled a more comprehensive list, which we will share with you following the workshop.
1. Leiderman, Sally. “Evaluation with a Racial Equity Lens Slides.” November 2017
This powerpoint by Sally Leiderman explores how evaluation is both a technical and a political exercise. In 13 slides, she breaks down the power dynamics at play in any evaluation process, between funders, evaluations, the community that is being evaluated, the people running an intervention, and the people being served by an intervention.
Her powerpoint contains useful infographics explaining participatory evaluation as well as a list of additional resources for further understanding systemic racism, participatory evaluation, and power (im)balances in evaluation processes.
2. The Equitable Evaluation Initiative and the Framework
https://www.equitableeval.org/framework
The Equitable Evaluation Framework is designed to help foundations and nonprofits, and the consultants and philanthropy serving organizations that support them, explore what it means to be about and for equity by considering how all their assets and efforts align.
The framework is built around three core principles:
- Evaluations work is in service of and contributes to equity
- Evaluative work should be designed and implemented in a way that is commensurate with the values underlying equity work
- Evaluative work can and should answer critical questions about the effect of a strategy on different populations and on the underlying systemic drivers of inequity, and the ways in which history and cultural context are tangled up on the structural conditions and the change initiative itself.
3. OXFAM UK’s Responsible data practices: Oxfam UK Responsible data training pack
https://policy-practice.oxfam.org/resources/responsible-data-management-training-pack-620235/
Oxfam designed this training pack to:
- Help introduce the principles of responsible data management
- Examine how organizations might handle unexpected issues that arise in different contexts
- Introduce the planning processes that can be used. Throughout, Oxfam advocates for collecting data with respect and upholding the rights of respondents whose data is collected.
4. Oxfam UK responsible data policy
This policy written and adapted by Oxfam is a great example of how an organization can take measures to ensure that data is collected and used responsible at all stages of the data life cycle. To this end, this policy includes definitions and requirements for high-, medium- and low-risk data.
5. Using the data lifecycle to manage data responsibly
This leaflet will help anyone handling, sharing or accessing programme data to properly consider responsible data issues throughout the data lifecycle. It focuses mainly on the responsible management of data in humanitarian crises, from the perspective of data collection. However, the principles, tools and approaches are also more widely applicable. The steps outlined here should be seen as a framework for ensuring that data collectors are responsive, uphold accountability and raise voices.
6. Privacy and Data Security under GDPR for Quantitative Impact Evaluation
In this paper, Oxfam discusses privacy and data security from a rights-based perspective, with a focus on quantitative impact evaluation. We situate protocols we have developed within the context of the General Data Protection Regulation (GDPR), sharing relevant debates along the way. This article provides a useful overview for translating principles and regulations into everyday impact evaluation practice.