Neither too narrow nor too broad
Mohaqeqmoein,M.H.& Fetterman, D.M.
July 2008
To be useful a definition should be neither too narrow nor too broad, and like many disciplines evaluation suffers from both (Coryn, 2007, 31). Stufflebeam & Shinkfield's taxonomy suffers from being at the "too narrow" and restrictive end of the evaluation definition spectrum; excluding high quality and useful evaluation work. Stufflebeam & Shinkfield in their book" Evaluation Theory, Models, and Applications" present a new taxonomy for evaluation approaches.
1. Pseudo evaluations.(approach 1 to approach 5) include empowerment evaluation approach
2. Questions- and Methods-Oriented Evaluation Approaches (Quasi-Evaluation Studies). (approach 6 to approach 19)
3. Improvement- and Accountability-Oriented Evaluation Approaches. (Approach 20 to approach 22) include CIPP model.
4. Social Agenda and Advocacy Approaches. (approach 23 to approach 25)
5. Eclectic Evaluation Approaches ( approach 26) include Utilization-Focused Evaluation
Putting the intellectual mischief and attempt to incite or provoke a reaction aside, the categories as constructed are telling - but more about the authors than the approaches.An overriding question associated with each of these evaluation approaches is : Who is in control? Who is the evaluation for? Why is the evaluation being conducted? Without answering these questions it is impossible to categorize them accurately or meaningfully. A quick review of a few paragraphs in the book reveal the "soft underbelly" of their logic and the flaws associated with their thinking.
On page 154 we can read:
When an external evaluator's efforts to empower a group to conduct its own evaluations are advanced as external or independent evaluations, they fit our label of empowerment under the guise of evaluation.
Already we see two flaws in their thinking. There is no claim about empowering anyone in empowerment evaluation. Empowerment evaluations can not and do not attempt to empower anyone. People empower themselves. Empowerment evaluators create an environment for people to empower themselves (Fetterman and Wandersman, 2005).
Second, no one claims that empowerment evaluation is an external or independent evaluation. It is explicitly an internal form of evaluation designed to foster self-determination and improvement (and cultivate internal forms of accountability - the kind that lasts long after the formal external evaluation disappears and the authorities shift their attention to other matters).
Such applications give the evaluees the power to write or edit the interim or final reports while claiming or giving the illusion that an independent evaluator prepared and delivered the reports or at least endorsed internal evaluation reports….
Once again we see a straw person argument. No one is making claims of this nature except the authors. The power of empowerment evaluation is process use. The more that people take an active role in conducting their own evaluations the more likely they are to: 1) find the findings credible; and 2) accept and implement the recommendations (because they are theirs). Writing and/or editing is part of cultivating ownership and should be encouraged for accuracy. The internal evaluation may be endorsed or not by an external body but that is not the point of conducting an internal empowerment evaluation. The point is to build internal evaluation capacity.
Objectives of training and empowering a disadvantaged group to conduct evaluations are laudable in their own right. However empowering groups to do their own evaluation is not evaluation (Stufflebeam and Shinkfield, 2007, 154).
Here again we see a failure to understand what empowerment evaluation is or is not before launching into a critique. There is agreement that training "disadvantaged" groups to conduct their own evaluations is laudable. However, that is where the agreement ends. This is where we see an unnecessarily restrictive and intolerant tone. A group that conducts its own evaluation is by definition evaluation. They may not find it a sufficiently credible form of evaluation, they may not find that this form conforms with their understanding or what evaluation is supposed to do, but it is a form of evaluation. They can only see external accountability as a goal of evaluation and in essence define it accordingly. However, there is a whole world out there in which evaluation is used to help develop programs and contribute to knowledge. Accountability is only one of many purposes of evaluation as Chelimsky pointed out many years ago. Empowerment evaluation focuses on helping program develop with contributions to accountability and knowledge construction. Their blindness to the multiple purposes of evaluation blind them to the potential of evaluation itself as a productive force in the world.
On page 330 we continue to read:
Since information empowers those who hold the information, the CIPP model emphasizes the importance of even-handedness in involving and informing program's stakeholders.
A cardinal rule in traditional forms of evaluation is not to promote your own approach as superior in an argument about the validity of various approaches. Following their "rules" and logic it would not be "objective" for them to evaluate their own approach in such a laudatory fashion.
In addition, it stretches credulity to suggest that the CIPP model in particular, which does not advocate for program or participant control, is empowering. Cousins widely-recognized graph compares CIPP or objectivist approaches with collaborative, participatory, and empowerment evaluation approaches and the CIPP model is the furthest removed from anything vaguely resembling an empowering approach (with the lowest level of participant involvement or control).
Moreover, evaluators should strive to reach and involve those most in need and with little access to and influence over services. While evaluators should control the evaluation process to ensure its integrity, CIPP evaluators accord beneficiaries and other stakeholders more than a passive recipient's role. Evaluators are charged to keep stakeholders informed and provide them appropriate opportunities to contribute.
There is a paternalistic tone here "evaluators should strive to...involve those", "evaluators should control the evaluation process. evaluators accord...stakeholders more than a passive role." In addition, "Evaluators are charged to keep the stakeholders informed." In essence the evaluator "holds all the cards" and decides when and how participants can participate in the process. This is often perceived as condescending, demeaning, and authoritarian, while stated as beneficent and generous. It ignores the right and responsibility of people to be independent self-sufficient masters of their own destinies.
Involving all levels of stakeholders is considered ethically responsible because it equitably empowers the disadvantaged as well as the advantaged to help define the appropriate evaluation questions and criteria, provide evaluative input, critique draft reports, and receive, review ,and use evaluation findings. Involving all stakeholder groups is also wise because sustained, consequential involvement positions stakeholders to contribute information and valuable insights and inclines them to study, accept, value and act on evaluation reports (ibid,330).
The spirit behind this statement is simultaneously commendable and abhorred to anyone committed to self-determination. These statements suggest that there is more than a faint hint of recognition about the importance of involving local stakeholders in an evaluation, to help "empower" them and to improve the accuracy of the evaluation. It is important to applaud and recognize how these evaluators are moving the right direction, but not far enough and not from the eyes of participants and staff members who operate the program every day. It is still presumptuous to assume that it is sufficient to "involve" or "inform". The time is long overdue to share control and enter an evaluation as a partnership, not a master throwing a dog a bone.
One cannot help but ask at the end of these excerpts: Why in the CIPP model is involving and empowering stakeholders wise and ethical, while the same in empowerment evaluation, is categorized as "in the guise of evaluation? The answer is a product of the authors' narrow definition of evaluation. The answer lies in who is allowed to control the shape and direction of the evaluation and who's interests are being served in the process.
Reference
Coryn, Chris L. S., (2007), Evaluation of researchers and their research: Toward making the implicit explicit, Western Michigan University, Kalamazoo , Interdisciplinary Doctoral Program in Evaluation, Unpublished doctoral dissertation.
Fetterman, D.M. and Wandersman, A. (2005). Empowerment evaluation principles in practice. New York: Guilford Publications.
Stufflebeam Daniel L. and Anthony J. Shinkfield, (2007), Evaluation Theory, Models, and Applications , Josseybass publication.
1. Pseudo evaluations.(approach 1 to approach 5) include empowerment evaluation approach
2. Questions- and Methods-Oriented Evaluation Approaches (Quasi-Evaluation Studies). (approach 6 to approach 19)
3. Improvement- and Accountability-Oriented Evaluation Approaches. (Approach 20 to approach 22) include CIPP model.
4. Social Agenda and Advocacy Approaches. (approach 23 to approach 25)
5. Eclectic Evaluation Approaches ( approach 26) include Utilization-Focused Evaluation
Putting the intellectual mischief and attempt to incite or provoke a reaction aside, the categories as constructed are telling - but more about the authors than the approaches.An overriding question associated with each of these evaluation approaches is : Who is in control? Who is the evaluation for? Why is the evaluation being conducted? Without answering these questions it is impossible to categorize them accurately or meaningfully. A quick review of a few paragraphs in the book reveal the "soft underbelly" of their logic and the flaws associated with their thinking.
On page 154 we can read:
When an external evaluator's efforts to empower a group to conduct its own evaluations are advanced as external or independent evaluations, they fit our label of empowerment under the guise of evaluation.
Already we see two flaws in their thinking. There is no claim about empowering anyone in empowerment evaluation. Empowerment evaluations can not and do not attempt to empower anyone. People empower themselves. Empowerment evaluators create an environment for people to empower themselves (Fetterman and Wandersman, 2005).
Second, no one claims that empowerment evaluation is an external or independent evaluation. It is explicitly an internal form of evaluation designed to foster self-determination and improvement (and cultivate internal forms of accountability - the kind that lasts long after the formal external evaluation disappears and the authorities shift their attention to other matters).
Such applications give the evaluees the power to write or edit the interim or final reports while claiming or giving the illusion that an independent evaluator prepared and delivered the reports or at least endorsed internal evaluation reports….
Once again we see a straw person argument. No one is making claims of this nature except the authors. The power of empowerment evaluation is process use. The more that people take an active role in conducting their own evaluations the more likely they are to: 1) find the findings credible; and 2) accept and implement the recommendations (because they are theirs). Writing and/or editing is part of cultivating ownership and should be encouraged for accuracy. The internal evaluation may be endorsed or not by an external body but that is not the point of conducting an internal empowerment evaluation. The point is to build internal evaluation capacity.
Objectives of training and empowering a disadvantaged group to conduct evaluations are laudable in their own right. However empowering groups to do their own evaluation is not evaluation (Stufflebeam and Shinkfield, 2007, 154).
Here again we see a failure to understand what empowerment evaluation is or is not before launching into a critique. There is agreement that training "disadvantaged" groups to conduct their own evaluations is laudable. However, that is where the agreement ends. This is where we see an unnecessarily restrictive and intolerant tone. A group that conducts its own evaluation is by definition evaluation. They may not find it a sufficiently credible form of evaluation, they may not find that this form conforms with their understanding or what evaluation is supposed to do, but it is a form of evaluation. They can only see external accountability as a goal of evaluation and in essence define it accordingly. However, there is a whole world out there in which evaluation is used to help develop programs and contribute to knowledge. Accountability is only one of many purposes of evaluation as Chelimsky pointed out many years ago. Empowerment evaluation focuses on helping program develop with contributions to accountability and knowledge construction. Their blindness to the multiple purposes of evaluation blind them to the potential of evaluation itself as a productive force in the world.
On page 330 we continue to read:
Since information empowers those who hold the information, the CIPP model emphasizes the importance of even-handedness in involving and informing program's stakeholders.
A cardinal rule in traditional forms of evaluation is not to promote your own approach as superior in an argument about the validity of various approaches. Following their "rules" and logic it would not be "objective" for them to evaluate their own approach in such a laudatory fashion.
In addition, it stretches credulity to suggest that the CIPP model in particular, which does not advocate for program or participant control, is empowering. Cousins widely-recognized graph compares CIPP or objectivist approaches with collaborative, participatory, and empowerment evaluation approaches and the CIPP model is the furthest removed from anything vaguely resembling an empowering approach (with the lowest level of participant involvement or control).
Moreover, evaluators should strive to reach and involve those most in need and with little access to and influence over services. While evaluators should control the evaluation process to ensure its integrity, CIPP evaluators accord beneficiaries and other stakeholders more than a passive recipient's role. Evaluators are charged to keep stakeholders informed and provide them appropriate opportunities to contribute.
There is a paternalistic tone here "evaluators should strive to...involve those", "evaluators should control the evaluation process. evaluators accord...stakeholders more than a passive role." In addition, "Evaluators are charged to keep the stakeholders informed." In essence the evaluator "holds all the cards" and decides when and how participants can participate in the process. This is often perceived as condescending, demeaning, and authoritarian, while stated as beneficent and generous. It ignores the right and responsibility of people to be independent self-sufficient masters of their own destinies.
Involving all levels of stakeholders is considered ethically responsible because it equitably empowers the disadvantaged as well as the advantaged to help define the appropriate evaluation questions and criteria, provide evaluative input, critique draft reports, and receive, review ,and use evaluation findings. Involving all stakeholder groups is also wise because sustained, consequential involvement positions stakeholders to contribute information and valuable insights and inclines them to study, accept, value and act on evaluation reports (ibid,330).
The spirit behind this statement is simultaneously commendable and abhorred to anyone committed to self-determination. These statements suggest that there is more than a faint hint of recognition about the importance of involving local stakeholders in an evaluation, to help "empower" them and to improve the accuracy of the evaluation. It is important to applaud and recognize how these evaluators are moving the right direction, but not far enough and not from the eyes of participants and staff members who operate the program every day. It is still presumptuous to assume that it is sufficient to "involve" or "inform". The time is long overdue to share control and enter an evaluation as a partnership, not a master throwing a dog a bone.
One cannot help but ask at the end of these excerpts: Why in the CIPP model is involving and empowering stakeholders wise and ethical, while the same in empowerment evaluation, is categorized as "in the guise of evaluation? The answer is a product of the authors' narrow definition of evaluation. The answer lies in who is allowed to control the shape and direction of the evaluation and who's interests are being served in the process.
Reference
Coryn, Chris L. S., (2007), Evaluation of researchers and their research: Toward making the implicit explicit, Western Michigan University, Kalamazoo , Interdisciplinary Doctoral Program in Evaluation, Unpublished doctoral dissertation.
Fetterman, D.M. and Wandersman, A. (2005). Empowerment evaluation principles in practice. New York: Guilford Publications.
Stufflebeam Daniel L. and Anthony J. Shinkfield, (2007), Evaluation Theory, Models, and Applications , Josseybass publication.
