Try Before and After You Buy
How can agencies assess whether an AI tool is likely to work before committing significant resources? This session covers practical methods for testing AI systems prior to procurement or deployment, including baseline comparisons, pilot design, task-based evaluation, and the identification of acceptable and unacceptable risks. Also, performance can change as models evolve, workflows shift, staff adapt their practices, and systems encounter new contexts and use cases. This session explores how agencies can monitor performance over time, detect emerging problems, and understand when an initially successful implementation may require adjustment or reevaluation.
Practical Approaches to Evaluating AI for Pub…Coming UpPrinciples for Public-Sector AI Evaluation
Drawing on lessons from research and practice, this session presents a practical framework for evaluating AI in government.
Practical Approaches to Evaluating AI for Pub…Comparing Humans, AI, and Human-AI Teams
Explore methods for comparing human-only, AI-only, and human-plus-AI workflows and identifying where AI adds value, where it creates risks, and where hybrid approaches perform best.
Practical Approaches to Evaluating AI for Pub…Measuring What Matters
This session examines how agencies can move beyond productivity measures to define success in terms of service quality, accuracy, equity, public trust, resident and worker experience, and public outcomes.
Practical Approaches to Evaluating AI for Pub…Why Measuring AI Is Hard
Explore the limitations of common AI benchmarks, the differences between predictive and generative AI systems, and why performance in laboratory tests often fails to predict success in real-world settings.