Principles for Public-Sector AI Evaluation
Drawing on lessons from research and practice, this session presents a practical framework for evaluating AI in government. Participants will leave with a set of principles, methods, and questions for assessing AI effectiveness, measuring public value, monitoring performance over time, and making informed decisions about adoption, scaling, redesign, or retirement.
By the end of this workshop, participants will be able to:
- Apply a practical framework for evaluating AI effectiveness and public value in government settings.
- Use evidence from testing and ongoing monitoring to assess whether AI systems continue to meet agency needs and performance expectations.
- Make informed decisions about when to adopt, scale, redesign, or retire an AI system based on performance, risks, and public outcomes.
Practical Approaches to Evaluating AI for Pub…Try Before and After You Buy
This session explores how agencies can monitor performance over time, detect emerging problems, and understand when an initially successful implementation may require adjustment or reevaluation.
Practical Approaches to Evaluating AI for Pub…Comparing Humans, AI, and Human-AI Teams
Explore methods for comparing human-only, AI-only, and human-plus-AI workflows and identifying where AI adds value, where it creates risks, and where hybrid approaches perform best.
Practical Approaches to Evaluating AI for Pub…Measuring What Matters
This session examines how agencies can move beyond productivity measures to define success in terms of service quality, accuracy, equity, public trust, resident and worker experience, and public outcomes.
Practical Approaches to Evaluating AI for Pub…Why Measuring AI Is Hard
Explore the limitations of common AI benchmarks, the differences between predictive and generative AI systems, and why performance in laboratory tests often fails to predict success in real-world settings.