How Not to Buy Stupid AI: A Practical Guide to Evaluating AI Products and Tools

Agencies face increasing pressure to “use AI,” but not every automation or predictive tool is appropriate, and not every product delivers what it promises. Building on lessons learned from how researchers evaluate the effectiveness and fairness of AI tools, this session offers practical strategies for government. Participants will discuss how to interrogate vendor claims, what to demand of those selling us AI, work with partners in academia and industry to measure outcomes, and avoid paying for systems that don’t perform in practice or, worse, cause more harm than good. We discuss the limitations of AI auditing and practical strategies for getting smarter about what we buy.