Evaluation Checklists: From Creation to Validation - Prof. Wesley A. Martz, Study notes of Marketing Research

Download Evaluation Checklists: From Creation to Validation - Prof. Wesley A. Martz and more Study notes Marketing Research in PDF only on Docsity! Evolution of Evaluation Checklists From Creation To Validation Wes Martz, Ph.D. Daniela C Schröter Ph D . , . . Overview of Session Introduction to Evaluation Checklists1 OEC Development and Validation2 SEC Development and Validation3 4 Lessons Learned Why develop checklists? Advantages Disadvantages • Consolidate vast knowledge in a parsimonious manner1 • Overuse (fatigue) • Unnecessary barriers • Improve task performance2 • Reduce influence of halo • Tunnel vision Inappropriate use and Rorschach effects1 • Reduce resource use3 • 1 Scriven, 2007; 2 Hale et al., 2007; Wolff et al., 2004; 3 Persaud, 2007 Validating Checklists • Limited guidance available for validating evaluation checklists • Valid checklists are important to validity and credibilit of e al ati e concl sionsy v u v u ed Validation Methodology ORGANIZATIONAL EFFECTIVENESS EVALUATION CHECKLIST (OEC)  OEC Validation Study PHASE 1: EXPERT PANEL REVIEW ‘besa Expert Panel Overview • Study participants – Subject matter experts (organizational and evaluation theorists) T t d ( f i l l t– arge e users pro ess ona eva ua ors, organizational consultants, managers) • Review OEC for providing critical feedback • Identify strengths and weaknesses • Complete the critical feedback survey • Write comments directly on the checklist OEC Validation Process Performance Criteria • Applicability to full range of intended uses • Clarity • Comprehensiveness • Concreteness • Ease of use • Fairness • Parsimony • Pertinence to the content area D. L. Stufflebeam (2000). Guidelines for developing evaluation checklists: The checklist development checklist. Available at http://www.wmich.edu/evalctr/checklists/guidelines_cdc.pdf. Net Strength Score vs. Mean 7.5 7 Concreteness PertinenceFairness Clarity 6 6.5 M ea n Sc or e ComprehensivenessApplicability 5.5 M Ease of Use Parsimony 5 -10 -8 -6 -4 -2 0 2 4 6 8 10 Net Strength PHASE 2: FIELD TEST OEC Validation Study Field Test Overview • Evaluation Client – For-profit organization based in the U.S. – Specializes in storytelling using electronic media – Stand-alone entity (e.g., not a division or program) • Conduct evaluation using revised OEC • Strictly follow OEC to ensure fidelity • Post evaluation semi structured client interview- - • A formative metaevaluation to detect and correct deficiencies in the process OEC Validation Method Strengths & Weaknesses • Strengths – Relatively quick validation process – Based on relevant evaluative criteria – Features a real-world application • Weaknesses – Single-case field study – Selection of the case study – Selection of the expert panel members arv1 ’ = Validation Methodology SUSTAINABILITY EVALUATION CHECKLIST (SEC) Overview of the SEC Purpose & Users • Purpose: Planning and designing project and program evaluations of sustainability for sustainability in development contexts – Remember evaluation tasks – Consider criteria of sustainability – Reduce errors of omission – Increase evaluation usefulness – Generate ideas and discussion among stakeholders – Support proposal writing – Think about existing sustainability evaluations • Users: – Evaluators and researchers – Program planners, funders, managers – Recipients or participants SEC Development &Validation • Development – Literature-based development imbued with iterative feedback from experts and practitioners C ti– on nuous process • Validation – Nonexperimental, exploratory, mixed method study – Systematic feedback on checklist accuracy and tilit iu y v a • Self-administered Web-based questionnaire (N=106) Expert interviews (N=10)• • Email (N=5+) SEC Study Participants • 111 pre-identified experts and practitioners in – Evaluation S t i bl d l t– us a na e eve opmen – International development • Sampling – Purposeful maximum heterogeneity sampling – Expert recruitment via direct contact – Practitioner recruitment via Listservs Participants: Organizational Affiliation Ed tiuca on, 16% Other 15% Profit, 9% Government, 10% , National non-profit, 15% International non-profit, 31% Multi-lateral, 4% SEC Validity Study: Analytic Framework (4) Respondents by Response Medium SEC Validity Rating Scales Qualitative Results @) Item Analyses Cross-Item Analysis (3) Weaknesses CF Strengths > — oo Recommendations Cross-Case Analysis Descriptive Statistics Cross-Section Analysis —— Triangulation Summary and Implications © © SEC Validity Measures • Measures 25 it i h kli t d tilit– ems measur ng c ec s accuracy an u y – Five point agreement scale from 0 – not at all to 4 - completely • Item analyses – 7 items were removed – Convergent and discriminant validity of remaining items – Internal consistency estimates/reliability • Accuracy scale: α = .87 Utilit l 89• y sca e: α = . SEC Accuracy Planning and designing sustainability … Identifying criteria of specific relevance Considering all aspects of … Comprehensive (i.e., complete) Total Accuracy Evaluating sustainability evaluations Considering those is need Identifying information needs of … Considering ethics in sustainability … Identifying defensible information … 0% 20% 40% 60% 80% 100% Perceived Percentage of SEC Accuracy Cross Case Analysis Codes UniqueItem Question Responses across responses themes Omissions What is missing? 85 85 13 Commissions What components or checkpoints are not necessary? 58 52 9 Confusions What areas are confusing? 71 68 8 Errors/ problems Are there any other errors or problems that need to be dd d? 55 50 11 a resse Strengths What, if any, are the strengths of the checklist? 79 105 12 How can the checklist beSuggestions improved? 75 94 10 Example “Omissions” Application , 1% Clarification, 4% Clarity, 4% Glossary , 1% Supplementary Tensions, 11% Guidance, 2% New (sub)category Structure, 1% materials, 6% , 8% None, 7% Strength, 6% Reconsiderations , 1%Specifications, 48% Example “Strengths” Detail, 2% Tensions 1% Generality, 4% Holistic, 3% Use, 3% , Comprehensive, 32%Food for thought, 6% Concept 9% Clarity, 7% Structure, 13% , Content, 11% Contribution, 10% SEC Validation Method Strengths & Weaknesses Strengths Weaknesses • Relatively quick validation • Development limited by time process, but long development process and resource constraints • Validation limited to content • Validation based on relevant evaluative criteria and open to critical feedback and perceived utility, though evidence of use exists • Feedback from a highly diverse group of experts and • SEC application limited to volunteers, resulting in lack of systematic information about practitioners application LESSONS LEARNED Development and Validation Lessons Learned • Checklist development should address unique attributes of the evaluand • Sampling frame is critical • Checklist validation should be grounded in theory practice and use, , • Mixed method approach provides increased confidence in validation conclusions • All checklists are a “work-in-process”