Empowerment evaluation

Empowerment evaluation (EE) is an evaluation approach designed to help communities monitor and evaluate their own performance. It is used in comprehensive community initiatives as well as small-scale settings and is designed to help groups accomplish their goals. According to David Fetterman, "Empowerment evaluation is the use of evaluation concepts, techniques, and findings to foster improvement and self-determination". An expanded definition is: "Empowerment evaluation is an evaluation approach that aims to increase the likelihood that programs will achieve results by increasing the capacity of program stakeholders to plan, implement, and evaluate their own programs."

Scope

Empowerment evaluation has been used in programs ranging from a fifteen million dollar Hewlett-Packard corporate philanthropy effort to bridge the digital divide in communities of color to accreditation in higher education and from the NASA Jet Propulsion Laboratory’s Mars Mars Rover project to battered women's shelters. Empowerment evaluation has been used by government, foundations, businesses, and non-profits, as well as Native American reservations. It is a global phenomenon, with projects and workshops around the world including Australia, Brazil, Canada, Ethiopia, Finland, India, Israel, Japan, Mexico, Nepal, New Zealand, South Africa, South Korea, Spain, Thailand, the United Kingdom, and the United States. A sample of sponsors and clients includes Casey Family Programs, Centers for Disease Control and Prevention, USAID, Feeding America, Family & Children Services, Health Trust, Knight Foundation, Poynter, Stanford University, State of Arkansas, UNICEF and Volunteers of America.

History and publications

Empowerment evaluation was introduced in 1993 by David Fetterman during his presidential address at the American Evaluation Association’s (AEA) annual meeting.

The approach was initially well received by some researchers who commented on the complementary relationship between EE and community psychology, social work, community development and adult education. They highlighted how it inverted traditional definitions of evaluation, shifting power from the evaluator to program staff and participants. Early supporters positively noted the focus on social justice and self-determination. One colleague, however, expressed concern about the evaluation establishment's tolerance for dissent and compared the empowerment evaluation writings to Martin Luther's 95 Theses.

Empowerment Evaluation: Knowledge and Tools for Self-assessment and Accountability

Empowerment Evaluation: Knowledge and Tools for Self-assessment and Accountability

Empowerment Evaluation: Knowledge and Tools for Self-assessment and Accountability the first empowerment evaluation book, provided an introduction to theory and practice. It highlighted EE's scope, ranging from its use in a national educational reform movement to its endorsement by the W. K. Kellogg Foundation’s Director of Evaluation. The book presented examples in various contexts, including federal, state, and local government, HIV prevention and related health initiatives, African American communities, and battered women’s shelters. This first volume also provided various theoretical and philosophical frameworks as well as workshop and technical assistance tools.

Foundations of Empowerment Evaluation

Foundations of Empowerment Evaluation

Foundations of Empowerment Evaluation, was the second EE book. The book provided steps and cases ranging from hospital settings to summer school programs. It also applied the Standards to empowerment evaluation, e.g. utility, feasibility, propriety, and accuracy. It highlighted the role of the Internet to facilitate and disseminate the approach.

Empowerment Evaluation Principles in Practice

Empowerment Evaluation Principles in Practice

The third book was titled: Empowerment Evaluation Principles in Practice. It provided greater conceptual clarity by making explicit EE's underlying principles, ranging from improvement and inclusion to capacity building and social justice. In addition, it highlighted its commitment to accountability and outcomes, by stating them as an explicit principle and presenting substantive outcome examples. The book also illustrated high, medium, and low levels of empowerment evaluation. Cases described include educational reform, youth development programs, and child abuse prevention programs.

Empowerment Evaluation and the Digital Villages: Hewlett-Packard Packard’s $15 Million Race Toward Social Justice

Empowerment Evaluation in the Digital Villages- Hewlett-Packard's $15 Million Race Toward Social Justice.jpg

Empowerment Evaluation and the Digital Villages: Hewlett-Packard Packard’s $15 Million Race Toward Social Justice, the fourth book, presented three case examples of how empowerment evaluation was used to bridge the digital divide in communities of color. The communities included 18 Native American tribes, a primarily Black east coast community, and a west coast Black, Latinx, and Pacific Islander community. The Tribal Digital Village, composed of 18 Native American tribes, used empowerment evaluation to help build one of the largest unlicensed wireless systems in the country, as well as a digital printing press.

Empowerment Evaluation: Knowledge and Tools for Self-assessment, Evaluation Capacity Building, and Accountability

Empowerment Evaluation: Knowledge and Tools for Self-assessment, Evaluation Capacity Building, and Accountability

The fifth book is titled Empowerment Evaluation: Knowledge and Tools for Self-assessment, Evaluation Capacity Building, and Accountability (2nd edition). It represented a significant and transformative contribution to the literature. It consolidated over 21 years of work, highlighting the theory, principles, concepts, and steps of empowerment evaluation. According to Donaldson (2015), "This book marks the 21st anniversary of empowerment evaluation, an approach that has literally altered the landscape of evaluation." Case examples ranged from a 10-year empowerment evaluation tobacco prevention initiative to an empowerment evaluation conducted by fourth- and fifth-grade students.

Empowerment Evaluation and Social Justice: Confronting the Culture of Silence

Empowerment Evaluation and Social Justice

The sixth empowerment evaluation book was titled Empowerment Evaluation and Social Justice: Confronting the Culture of Silence. It described how empowerment evaluation is grounded in an emancipatory tradition. It is designed to help people learn to confront the status quo by questioning assumptions and prescribed roles, unpacking myths, rejecting dehumanization, and no longer accepting the "truth" about how things are or can be. This book also demonstrated how empowerment evaluation is inexorably bound to the pursuit of social justice. Case examples included a USAID-funded empowerment evaluation effort to eliminate tuberculosis in India and a Feeding America empowerment evaluation designed to fight for food justice. These case examples also highlight how an empowerment evaluation can be applied when there is a pre-existing evaluation or program plan.

Theories

The primary theories guiding empowerment evaluation are process use and theories of use and action.

Process use represents much of the rationale or logic underlying EE in practice, because it cultivates ownership by placing the approach in community and staff members’ hands.

The alignment of theories of use and action explain how empowerment evaluation helps people produce desired results.

Process use

Empowerment evaluation is designed to be used by people. It places evaluation in the hands of community and staff members. The more that people are engaged in conducting their own evaluations the more likely they are to believe in them, because the evaluation findings are theirs. In addition, a byproduct of this experience is that they learn to think evaluatively. This makes them more likely to make decisions and take actions based on their evaluation data. This way of thinking is at the heart of process use.

Principles

Empowerment evaluation is guided by 10 principles. These principles help evaluators and community members align decisions with the larger purpose or goals associated with capacity building and self-determination.

1. Improvement – help people improve program performance
2. Community ownership – value and facilitate community control
3. Inclusion – invite involvement, participation, and diversity
4. Democratic participation – open participation and fair decision making
5. Social justice – address social inequities in society
6. Community knowledge – respect and value community knowledge
7. Evidence-based strategies – respect and use both community and scholarly knowledge
8. Capacity building – enhance stakeholder ability to evaluate and improve planning and implementation
9. Organizational learning – apply data to evaluate and implement practices and inform decision making
10. Accountability – emphasize outcomes and accountability.

Concepts

Key concepts include critical friends, cultures of evidence, cycles of reflection and action, communities of learners, and reflective practitioners. A critical friend, for example, is an evaluator who provides constructive feedback. They help to ensure the evaluation remains organized, rigorous, and honest.

Steps

EE's three-step approach includes:

1. establish their mission;
2. review their current status; and
3. plan for the future.

This approach is popular in part due to its simplicity, effectiveness, and transparency.

A second approach is the 10-step Getting to Outcomes (GTO) . GTO helps participants answer 10 questions using relevant literature, methods, and tools. The 10 accountability questions and literature to address them are:

1. What are the needs and resources? (Needs assessment; resource assessment)
2. What are the goals, target population, and desired outcomes? (Goal setting)
3. How does the intervention incorporate knowledge of science and best practices in this area? (Science and best practices)
4. How does the intervention fit with existing programs? (Collaboration; cultural competence)
5. What capacities do you need to implement a quality program? (Capacity building)
6. How will this intervention be carried out? (Planning)
7. How will the quality of implementation be assessed? (Process evaluation)
8. How well did the intervention work? (Outcome and impact evaluation)
9. How will quality improvement strategies be incorporated? (Total quality management; continuous quality improvement)
10. If the intervention is (or components are) successful, how will the intervention be sustained? (Sustainability and institutionalization)

A manual with worksheets addresses how to answer the questions. While GTO has been used primarily in substance abuse prevention, customized GTOs have been developed for preventing underage drinking and promoting positive youth development. Several books are downloadable. In addition, EE can employ photojournalism, online surveys, virtual conferencing and self-assessments.

Monitoring

Conventional and innovative evaluation tools monitor outcomes, including online surveys, focus groups, and interviews, as well as the use of quasi-experimental designs. In addition, program specific metrics are developed, using baselines, milestones, goals, and actual performance. For example, a minority tobacco prevention program in Arkansas established:

1. Baselines (the number of tobacco users)
2. Goals (the yearly number of subjects helped)
3. Milestones (the monthly number of subjects helped)
4. Performance (the number of subjects who stop smoking)

These metrics help the community monitor implementation, by comparing performance with milestones. It also enables them to make mid-course corrections.

Selected case examples

Stanford University School of Medicine applied the technique to curricular decision-making. EE contributed to improvements in course and clerkship ratings. For example, the average student ratings for required courses improved significantly (P = .04; Student's one-sample t test).

EE guided Hewlett-Packard's $15 million Digital Village Initiative. The initiative was designed to help bridge the digital divide in communities of color. Outcomes ranged from Native American's building one of the largest unlicensed wireless systems in the country to creating a high-resolution digital printing press.

The State of Arkansas used EE in academically distressed schools and tobacco prevention. Outcomes include improving test scores, upgrading school-level performance, and preventing and reducing tobacco consumption.

A school district in South Carolina invested millions of its own dollars to provide each student with a personalized computing device as an educational tool. EE was used to support large-scale implementation of the initiative and monitor outcomes associated with teacher and student behavior change.

USAID used empowerment evaluation to help eliminate tuberculosis in India. Tuberculosis Champions were provided with rights-based training to help them assert themselves and their rights in the health care system. In addition, healthcare providers became more sensitized to the healthcare needs of the community. An empowerment evaluation dashboard was used to monitor progress throughout the engagement.

Feeding America used empowerment evaluation with its network of food banks throughout the United States. The focus of the initiative was to help food banks establish their own goals, assess their performance, and plan for the future. They used an empowerment evaluation dashboard to monitor their progress toward specified milestones and annual goals. This effort also applied a racial equity lens to their work.</ref>

Rationale

Response to critique

EE is conducted by an internal group, not an external individual. Programs are dynamic, not static, and thus require more fluid, responsive, and continual assessment. The evaluator becomes a coach, rather than the expert. Investigating worth and merit is not sufficient. The focus should also be on program improvement. Empowerment evaluation, as a group activity, builds in self-checks on bias. Internal and external forms of evaluation are compatible and reinforcing. However, the Joint Committee's standards were applied and empowerment evaluation was found to be consistent with the spirit of the standards. Empowerment evaluation is not a threat to traditional evaluation. It may instead help to revitalize it.

Empowerment evaluation is part of an emancipatory research stream. Its unique contribution is its focus on fostering self-determination and building capacity. Empowerment evaluation is guided by process use. An additional effort has been made to further distinguish empowerment from collaborative and participatory forms of evaluation. Empowerment evaluation is not limited to disenfranchised populations. Everyone can become more self-determined. Empowerment evaluation is part of a worldwide moment. It has become a part of the evaluation landscape.

Empowerment evaluation needs to continue to focus on the consumer or program participant, as well as staff members. Empowerment evaluations do not believe that bias in evaluation can be removed by distancing oneself from the group or program being assessed. An acccurate and useful evaluation requires immersion and close proximity to program participants and staff members. Both internal and external forms of evaluation are needed to conduct a useful exercise. Empowerment evaluators are most useful when they serve as evaluation consultants.

The definition of empowerment is the same as when the approach was first defined and introduced to the field. However, it has been expanded to further clarify the purpose of the approach. Fetterman and Wandersman agree that empowerment evaluation is part of an emancipatory stream of research. It also relies on process use to guide it. They also believe that a greater effort has been made to distinguish empowerment from other forms of stakeholder-involved approaches. However, empowerment evaluation can be viewed along a continuum from less empowering to more empowering in nature. Empowerment evaluation is designed to help the disenfranchised. However, the boundaries are much broader and inclusive. Everyone can benefit from self-assessment and becoming more self-determined.
Fetterman advocated that evaluation be shared with a broader population.

Debates and controversy

Empowerment evaluation challenged the status quo concerning who is in control of an evaluation and what it means to be an evaluator. Conventionally, evaluations are conducted by a specialist. In EE, the group or community performs the evaluation, guided by an empowerment evaluator or “critical friend.”

First wave of criticism

Shufflebeam claimed that evaluation should be left in the hands of professionals who objectively investigate the worth or merit of an object and that EE violates the (as yet unadopted) Joint Committee's Program Evaluation Standards.

Fetterman and Scriven agreed on the value of both internal and external evaluations. They also agree on a focus on the consumer. However, staff members, sponsors, and policymakers also have important roles to play in evaluation. Scriven however claimed that the evaluator must maintain distance from program participants to avoid bias.

Chelimsky re-framed the discussion between Fetterman, Patton and Scriven, explaining that evaluations serve multiple purposes: (1) accountability; (2) development; and (3) knowledge. Scriven, and to a lesser extent Patton, focused on accountability, while Fetterman focused on development.

Second wave

The second wave of debate and discussion emerged between 2005 and 2007. The primary critiques focused on conceptual and methodological clarity:

Cousins attempted to differentiate between similar approaches, e.g. collaborative, participatory, and empowerment evaluation. Cousins asked whether EE is practical (focusing on decision making), or transformative (focusing on self-determination) and viewed self-evaluation as more likely to have a self-serving bias. This critic noted the variability in attempts at empowerment evaluation. Fetterman, Rodriguez-Campos, Zukoski,and Contributors (2018) clarified the distinctions between stakeholder involvement approaches, as well as similarities.

Miller and Campbell conducted a systematic literature review of empowerment evaluation. They highlighted types or modes of EE, as well as settings, reasons for use, selection process and degree of participation. They highlighted practice variants depending on the size of the evaluation. They suggested that clients were selecting it for appropriate reasons, such as capacity building, self-determination, accountability, cultivating ownership, and institutionalization of evaluations. However, they also found that approximately 25% were empowerment in name only. In addition, they argued for additional conceptual clarity.. Fetterman and Wandersman (2005) responded to this concern by producing Empowerment Evaluation Principles in Practice.

Patton accepted EE as part of the evaluation field and proposed that given its established status, additional clarity distinguishing collaborative, participatory, utilization and empowerment evaluation would be fruitful. (As noted above Fetterman and Wandersman, 2005, responded to this concern.) Patton acknowledged improvements ranging improved definitions and the 10 empowerment evaluation principles. He was concerned that self-determination was not on the list. Patton applauded and recommended process use for empowerment evaluation. He accepted the contributors' commitment to forthrightly describing problems. Patton proposed greater emphasis on outcomes or results in EE.

Scriven believes that self-evaluation is flawed, because it is inherently self-serving, and rejected its use for professional development. He questioned the ability of EE to actually empower people and recommended a neutral evaluator role. He suggested that internal and external evaluations are not compatible. He also suggests that empowerment as well as randomized controls are merely forms of ideology.

Response to critique

Fetterman and Wandersman responded by attempting to enhance conceptual clarity, provide greater methodological specificity and highlight EEs commitment to accountability and outcomes. They acknowledged and applauded Miller and Campbell's systematic review of EE projects, while noting neglected or omitted case examples and questioning some of their methodology.

Fetterman et al produced 10 principles to contribute to conceptual clarity and explained empowerment evaluators do not empower people. People empower themselves. Empowerment evaluators help create an enviropnment conducive to empowerment. Fetterman et al asserted that evaluations are inherently subjective and are shaped by culture and political context, and that EE is committed to honesty and rigor. EE is more inclusive than traditional evaluations, placing cross-checks on data and decisions. Participants often know more about problems than outsiders and have a vested interest in making their programs work. They claimed that internal and external evaluations can operate together effectively as additional cross-checks.

While the similarities among collaborative, participatory and empowerment evaluation were described in the first and second empowerment evaluation books, they recommended Cousins' tool to highlight the differences, focusing on depth of participation and control of evaluation technical decision making In addition, as noted above Fetterman, Rodrigquez-Campos, ukoski, and contributors, produced a volume clarifying these distinctions.

The most significant response to the critiques focused on outcomes. Fetterman & Wandersman argued that outcomes and results were important to EE. They produced a long list of specific project outcomes including:

Outcomes

• CDC funded a study using a quasi-experimental design that demonstrated improved outcomes as a result of empowerment evaluation.
• Empowerment evaluation used in Arkansas distressed schools, increased standardized test scores.
• Native American's built a wireless system and digital printing press supported by empowerment evaluation.
• Stanford University's School of Medicine used EE to prepare for an accreditation site visit. Increases in student course ratings were statistically significant.
• Arkansas save millions in excess medical costs from applying empowerment evaluation to tobacco prevention programs. This resulted in legislation creating the Arkansas Evaluation Center.

Scriven's assessment

Scriven agreed that external evaluators sometimes miss problems obvious to program staff members. He also stated they have less credibility with them than an internal evaluator. As a result, he concluded, it is less likely their recommendations will be implemented.

Scriven agreed that EE contributed to improvements in internal staff program evaluations and that empowerment evaluation could make a contribution to evaluation if combined with third-party evaluation.

Fetterman's Summary of Responses and Q&A Session

Fetterman's book Empowerment Evaluation and Social Justice: Confronting the Culture of Silence (2023) responds to most of the critiques raised throughout the decades. In addition it provides a chapter of frequently asked questions in classrooms, seminars, webinars, workshops, and in the literature.

Professional association affiliation and awards

Empowerment evaluation was a catalyst for the creation of the American Evaluation Association's Collaborative, Participatory, and Empowerment Evaluation topical interest group. Approximately 20% of the American Evaluation Association membership has been affiliated with the topical interest group.SAGE Publications, a social science textbook publisher, cited an empowerment evaluation book as one of their "classic titles in research methods". Four empowerment evaluators received honors from the association: Margret Dugan, David Fetterman, Shakeh Kaftarian, and Abraham Wandersman. The 21st anniversary of empowerment evaluation was celebrated at the American Evaluation Association. The room was packed. There was standing room only and members of the audience filled the hallways. Luminaries in the field praised empowerment evaluation including comments such as:

"Empowerment evaluation: An approach that has literally altered the landscape of evaluation" (Steward Donaldson)

"Empowerment evaluation: Exemplary is its openness to dialogue, reflective practice, and process use" (Michael Patton)

"Empowerment evaluation 21 years later: There is much to admire about empowerment evaluation" (Michael Scriven)

External links

• Empowerment Evaluation Blog
• The Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group
• Collaborative, Participatory, and Empowerment Evaluation Center
• David Fetterman - AEA Ignite Lecture
• Fetterman & Associates
• Global Impact Award for Empowerment Evaluation and Social Justice: Confronting a Culture of Silence