Evaluation in the Face of Uncertainty

Adding “Surprise” to the Mix

Historical Roots: Evaluation, Planning, and System Behavior

From Explaining Surprise to Dealing with It

Development Path of This Book

Guiding Principles

How to Read This Book

In Sum

2. Structure of the Unexpected

Where Does Surprise Come From?

Beyond Simple Distinctions

In Sum

3. Placing Surprise in the Evaluation Landscape

When Is the Probability of Surprise High?

When Is Surprise Disruptive to Evaluation?

In Sum

4. Minimizing Foreseeable Surprise

Theory: Using Explanatory Power and Simplified Relationships

Exploiting Past Experience: Capitalizing on What We Already Know

Limiting Time Frames to Minimize the Opportunity for Surprise

In Sum

5. Shifting from Advance Planning to Early Detection

Leading Indicators

System-Based Logic Modeling

In Sum

6. Agile Evaluation

Data

Agile Methodology

Retooling Program Theory

Agility and Stakeholder Needs

In Sum

7. How Much Is Too Much?: Appreciating Trade-Offs and Managing the Balance

A Framework for Appreciating Design Trade-Offs

Maximizing Choice, Minimizing Risk

Evaluation Design

In Sum

8. Applying the Examples to Categories of Cases: The Life Cycle View

“Unintended Consequences”: Unity across Programs and Their Evaluations

Interpreting Cases through a Life Cycle Perspective

In Sum

9. Applying the Examples to Categories of Cases: The Social/Organizational View

Navigating through the Cases

Placement of Cases on the Social/Organizational Map

Categorizations Derived from the Data

In Sum

10. Lessons from Individual Cases: Tactics for Anticipating Surprise

In Sum

11. Lessons from Individual Cases: Responding to Surprise

The Middle

Leading Indicators and Agile Evaluation

In Sum

12. Unanticipated Program Outcomes

Case Descriptions

Applying the Cases to Unintended Program Outcomes

Comparing the Cases

Predicting the Need for Agile Evaluation

In Sum

13. Concluding Thoughts

Cases

Case 1. Grasping at Straws and Discovering a Different Program Theory: An Exercise in Reengineering Analysis Logic in a Child Care Evaluation Setting, Dennis P. Affholter

Case 2. Shifting Sands in a Training Evaluation Context, James W. Altschuld and Phyllis M. Thomas

Case 3. Evaluating Programs Aimed at Promoting Child Well-Being: The Case of Local Social Welfare Agencies in Jerusalem, Anat Zeira

Case 4. Assessing the Impact of Providing Laptop Computers to Students, J. Dan Strahl, Deborah L. Lowther, and Steven M. Ross

Case 5. Quasi-Experimental Strategies When Randomization Fails: Propensity Score Matching and Sensitivity Analysis in Whole-School Reform, Gary L. Bowen, Roderick A. Rose, and Shenyang Guo

Case 6. Unexpected Changes in Program Delivery: The Perils of Overlooking Process Data When Evaluating HIV Prevention, Bryce D. Smith

Case 7. Evaluating Costs and Benefits of Consumer-Operated Services: Unexpected Resistance, Unanticipated Insights, and Déjà Vu All Over Again, Brian T. Yates

Case 8. Keep Up with the Program!: Adapting the Evaluation Focus to Align with a College Transition Program’s Changing Goals, Kristine L. Chadwick and Jennifer Conner Blatz

Case 9. Assumptions about School Staff’s Competencies and Likely Program Impacts, Laura Hassler Lang, Christine E. Johnson, and Shana Goldwyn

Case 10. Mixed Method Evaluation of a Support Project for Nonprofit Organizations, Riki Savaya and Mark Waysman

Case 11. Evaluating the Health Impacts of Central Heating, Jeremy Walker, Richard Mitchell, Stephen Platt, and Mark Petticrew

Case 12. Recruiting Target Audience: When All Else Fails, Use the Indirect Approach for Evaluating Substance Abuse Prevention, Molly Engle

Case 13. Unintended Consequences of Changing Funder Requirements Midproject on Outcome Evaluation Design and Results in HIV Outreach Services, Lena Lundgren, Therese Fitzgerald, and Deborah Chassler

Case 14. Generating and Using Evaluation Feedback for Providing Countywide Family Support Services, Deborah L. Wasserman

Case 15. Trauma and Posttraumatic Stress Disorder among Female Clients in Methadone Maintenance Treatment in Israel: From Simple Assessment to Complex Intervention

Miriam Schiff and Shabtay Levit

Case 16. From Unintended to Undesirable Effects of Health Intervention: The Case of User Fees Abolition in Niger, West Africa, Valéry Ridde and Aissa Diarra

Case 17. Unintended Consequences and Adapting Evaluation: Katrina Aid Today National Case Management Consortium, Amanda Janis and Kelly M. Stiefel

Case 18. Evaluation of the Integrated Services Pilot Program from Western Australia, Peter Hancock, Trudi Cooper, and Susanne Therese Bahn