---
id: "workshop:how-not-to-buy-stupid-ai-a-practical-guide-to-evaluating-ai-products-and-tools"
type: "InnovateUS workshop"
title: "How Not to Buy Stupid AI: A Practical Guide to Evaluating AI Products and Tools"
url: "https://innovate-us.org/how-not-to-buy-stupid-ai-a-practical-guide-to-evaluating-ai-products-and-tools"
date: "2026-03-02"
duration: "60 minutes"
series: "Prediction, Automation, and Decision-Making with AI: Risks and Opportunities"
instructors: ["Santiago Garces","Aleksandra Korolova"]
recording: "https://www.youtube.com/watch?v=KTWpAWBLino"
transcript: "automatic, from the recording"
source: "InnovateUS (innovate-us.org)"
licence: "CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org"
---

# How Not to Buy Stupid AI: A Practical Guide to Evaluating AI Products and Tools

InnovateUS workshop · 2 March 2026 · 60 minutes

Agencies face increasing pressure to “use AI,” but not every automation or predictive tool is appropriate, and not every product delivers what it promises. Building on lessons learned from how researchers evaluate the effectiveness and fairness of AI tools, this session offers practical strategies for government. Participants will discuss how to interrogate vendor claims, what to demand of those selling us AI, work with partners in academia and industry to measure outcomes, and avoid paying for systems that don’t perform in practice or, worse, cause more harm than good. We discuss the limitations of AI auditing and practical strategies for getting smarter about what we buy.

## Learning goals

- Use evidence-based methods for evaluating vendor claims and proposals.

- Identify red flags such as black-box models and unverifiable performance metrics.

- Come away with a “buyer’s guide” grounded in scientific rigor

## Speakers

- **Santiago Garces**, Former Chief Information Officer, City of Boston
- **Aleksandra Korolova**, Assistant Professor of Computer Science and Public Affairs, Princeton University

Part of the series [Prediction, Automation, and Decision-Making with AI: Risks and Opportunities](https://innovate-us.org/workshop-series/risks-and-opportunities).

Recording: https://www.youtube.com/watch?v=KTWpAWBLino

## Transcript

Automatic transcript of the recording: names and terms can be misheard, and the times can be a few seconds off. Each time links to that moment of the video.

**Aleksandra Korolova**

[0:00](https://youtu.be/KTWpAWBLino?t=0) And so nice to be here to speak to this audience. As was already mentioned, what I do is I work on understanding the societal impacts of AI. And I work on developing new technologies that can respect societal values, such as privacy and fairness. And the way I got originally into evaluating AI was through studying the meta ad platforms and the privacy and non-discrimination promises that meta used to make on those platforms. And now I am more focused on generative AI and issues such as bias, reliability, safety, and functionality. And what I'm gonna do today is I'm gonna give you my perspective on AI evaluation coming very much from the academic standpoint.

[1:03](https://youtu.be/KTWpAWBLino?t=63) So trying to summarize what we know about what AI can and cannot do academically speaking, and what lessons one can take from that for procurement decisions in practice. And the first thing that I often think about when thinking about AI evaluation is whether the AI is actually needed in the particular context. So for example, there was recently news that Burger King will install a chat bot in its headphones in order to evaluate whether the employees are sufficiently polite and friendly. And the first question that comes to mind in this case is why is it needed to have this complex AI chat bot when perhaps the same thing could be achieved with a simple button that every customer could simply click.

[2:09](https://youtu.be/KTWpAWBLino?t=129) The next question that I often ask is whether it is plausible that AI can do the task at hand. And to give you an illustration when it may not be so plausible, let me tell you a story of Retorio, which used to be a startup that claimed that it would be able to identify personality traits relative to employment, relevant to employment based on a short two or three minute video. And when you say something like that, the first question is, is it really possible to reliably identify such personality traits? And what the researchers have done is to show that actually in Retorio's case, their performance was completely non-robust to any sort of changes in what the person was wearing.

[3:09](https://youtu.be/KTWpAWBLino?t=189) So what they did is they had the same person speak exactly the same things in the video, but she was either wearing glasses or a head covering. And the results of the analysis were completely different. And that's an example of a demonstration when it's not even clear whether the AI can plausibly work in this context. The next question we are often interested in is whether the AI can work better than a simple alternative. And when we talk about simple alternatives, we talk about both humans performing the task, and also we talk about kind of like the simple old school statistics, such as running a few variable regressions.

[4:11](https://youtu.be/KTWpAWBLino?t=251) And it turns out that there are a number of tasks where even the most advanced AI does not outperform the humans or like a very old school, much more interpretable technique than AI. And one evidence for this that I think you may find relevant is for recidivism prediction. So researchers back in 2018 compared the performance of Compass, which is a quite widely used tool, an AI tool for recidivism prediction, to the performance of humans on the same task and to the performance of a linear classifier that just depended on two to seven features. And what they found is that indeed the humans and the simple classifier were no worse than the supposedly very advanced proprietary AI technology, at least on this task.

[5:22](https://youtu.be/KTWpAWBLino?t=322) This brings me to the next class of tasks that are considered in scientific literature called predictive optimization tasks. This is a classification out of Princeton's Center for Information Technology Policy that says that the predictive optimization tasks are tasks when machine learning is used to make predictions about some future outcome that pertains to individuals. And then those predictions are used to make decisions about them. So a pretrial risk assessment is a one prime example of what would constitute predictive optimization, predicting school dropout rates, credit worthiness, suicide risk, child abuse, et cetera, all fall into this bucket.

[6:27](https://youtu.be/KTWpAWBLino?t=387) And what the science and the extensive evaluation of AI on these kinds of tasks tells us is that in a lot of cases, there needs to be a lot done in order to establish legitimacy of AI for these kinds of tasks. And the folks at CITP have actually developed a rubric that can be used to ask the vendors for these kinds of tasks to establish this legitimacy. And in general, if your use case falls under predictive optimization, what I highly recommend is both using a rubric and also engaging an independent researcher to really assess legitimacy of AI for that use case.

[7:32](https://youtu.be/KTWpAWBLino?t=452) So this has been about kind of like the standard AI. Now let's talk for a second about generative AI, which I imagine many of you are excited about. And the excitement about it is very understandable for a couple of reasons. One, the generative AI is so exciting because it seems to be a very general purpose, right? You are able to apply it to pretty much any task at hand. And it also is unlike many other types of AI, generative AI is very approachable because it works through an interface that both understands and responds in language or images or video or other kind of forms of communications that all of us are familiar with. But it turns out that this general purpose and approachability are double-edged sword.

[8:39](https://youtu.be/KTWpAWBLino?t=519) And I like you to think about, having a general purpose cleaner, right? It works very well, but if you're gonna use a general purpose cleaner on your very fancy furniture, the results may not be exactly what you expect. And so in fact, there's no reason to believe that the general purpose AI will work very well for your use case. And in fact, the performance of this generative AI depends greatly on a number of factors. One of the primary ones is the quality of the training data that this generative AI was trained on and whether this training data represents your use case. It depends on whether the task can be well-specified and well-judged, or whether there's an expert level human guidance and oversight that is available when using this generative AI.

[9:48](https://youtu.be/KTWpAWBLino?t=588) And it also depends on the possibility of automated correctness verification. So for example, the reason we're seeing so many advances that generative AI makes in math is because in math it's possible to automatically verify correctness, which is not the case for many other real world tasks. The way generative AI companies usually talk about performance of their models is through benchmarks. But unfortunately, these benchmarks tell us very little about performance in real world use cases. For example, Chad GPT achieves very high performance on the bar exam, but that actually tells very little about its ability to practice law, which is a much more broader set of tasks than answering multiple choice questions.

[10:51](https://youtu.be/KTWpAWBLino?t=651) And to bring this point home, let me give you like a very simple example from a resume screening case study that we did a few months ago with my group. As many of you know, hiring and resume screening is one of the very popular uses of generative AI right now. And we wanted to check whether it's actually good at that task. And so what we did is we came up with resumes where one resume was clearly much more qualified for the task than another resume. And we asked the models to compare these resumes. And what we found is that some of the models are only at the 50 to 80% range of correctly comparing two resumes that any human would have no problem saying which one is a better performance.

[11:58](https://youtu.be/KTWpAWBLino?t=718) And so if it can't compare even two resumes correctly where the answer is very clear, it really begs the question of what do these models do when the answer is not even so clear. And in fact, this problem of validating AI performance on AI tasks is validating AI performance on hiring tasks is not new. AI has been used in hiring extensively and very few vendors actually have validation information. And there's a great paper that talks about this phenomenon called the fallacy of AI functionality. So when you're considering deploying generative AI, I think this problem of validation is really exacerbated by the general purpose nature of it.

[13:06](https://youtu.be/KTWpAWBLino?t=786) And so my recommendation would be to be relentless in demanding the performance evaluation of this generative AI specifically for your use case. And how to do that will obviously depend on your use case on what you have access to and so forth. But some examples could be using your own past data that has not been shared with a vendor for evaluation, comparing AI vendors training data to your training data, running pilots where you compare performance when using and versus not using AI, requesting the vendor to provide engineers that can be dedicated to customization and evaluation for your specific use case, implementing a continuous feedback and error analysis loop and even getting an independent researcher or domain expert involved in this evaluation.

[14:17](https://youtu.be/KTWpAWBLino?t=857) Most importantly, I urge you not to fall for the hype. The performance on the benchmarks says very, very little about your specific use case. And how this generative AI will perform in your specific use case. And other two concerns that are perhaps new for generative AI, but you've probably all heard about are hallucinations. I like this example from the markup where New York City deployed an AI chatbot that was hallucinating answers actually in the information... that they were claiming to be answering based on. And what I'd like to say is that for generative AI, hallucination is not a solved problem. Scientifically, we do not know how to address it completely.

[15:18](https://youtu.be/KTWpAWBLino?t=918) Yes, one can work on decreasing the rate of hallucinations, but as long as it's generative AI, it will hallucinate. Similarly, bias and explainability in generative AI are also not solved problems. On the right side, you see an example from a paper that's showing a person asking which colleges they should apply to and with or without explicit identity markers, and also asking for an explanation whether those identity markers are being used. And the chatbot is not correct and is not forthcoming about whether those markers are used. And so this is, again, an unsolved problem. Again, there are mitigation techniques, but both the bias and ensuring faithful explainability is not something that we know scientifically how to do for generative AI right now.

[16:31](https://youtu.be/KTWpAWBLino?t=991) So skipping over the traditional views of how to address them using data, I would say that when we're talking about generative AI, the approach that is most constructive is to putting the burden of risks of things such as hallucinations, bias, and poor explainability on the vendor. The common approach that vendors take is they say, well, of course, you should verify. Your users should be the ones verifying the answers. But it's against human nature. Humans are not perfectly vigilant. And they cannot continuously verify everything that they treat. So my suggestion, again, depending on what the use case is, this may be implemented differently. But looking for ways that vendors can share responsibility for failures, provide mechanisms for their reporting, and analyze and continuously work on their mitigation would be a good step forward specifically for generative AI.

[17:49](https://youtu.be/KTWpAWBLino?t=1069) And if your use case is such that it can tolerate these kinds of errors at all, then probably it is not ready for full deployment of generative AI. The final thing that I want to touch upon is cost. Generative AI is and will continue to be costly. I would urge you to consider all competitive offers because I think the integration these days of new technical tools is relatively easy. So you don't need to pick a chatbot from the same company with which you do email and things like that. And the one leverage that you typically have over the generative AI vendors is your data. And so to the extent that you can own and not share your data for the benefit of the vendor and only share it for the benefit of yourself, this would help you be more bulletproof against future price increases.

[19:08](https://youtu.be/KTWpAWBLino?t=1148) Let me stop here. I'd like to say that in the academic community, there is a lot of conversation about the fact that it is people who are deploying the AI and the ones who are going to be shaping the development of it and shaping what kind of questions get addressed. So I'm looking forward to learning from all of you how to best shape it for the common good. I'm also very happy to talk about these questions at any time. And many of my colleagues and also Marv Hickok have done a lot of work on this topic that may be of interest for you to read further. Thank you very much.

**Eileen Twiggs**

[20:00](https://youtu.be/KTWpAWBLino?t=1200) Thank you so much, Alexandra, for that very informative presentation. I'm going to go ahead and switch it on over to Santi, who's going to follow up with his perspective from the city of Boston. Over to you, Santi.

**Santiago Garces**

[20:14](https://youtu.be/KTWpAWBLino?t=1214) Hello, welcome. Thank you for having me here. And thanks for Alexandra for sharing her thoughts. I'm going to share my screen. I have some slides as well to share. So again, I'm Santi Garces. I'm the chief information officer for the city of Boston. So I work closely with a number of academics, but my job is to buy technology that hopefully makes a better experience for constituents and for our employees at the city. So I'll give you some of our perspectives from my experience at the city. So I tried to, again, in government, we're usually busy. We usually don't have a lot of money and time to figure out what is the right thing to do. So I was trying to, based on our experience, doing a lot of work with AI, particularly with generative AI, how I tend to think about buying and evaluating the tools that we're going to buy.

[21:17](https://youtu.be/KTWpAWBLino?t=1277) So first, my experience focusing on the goal is really important. And we'll dig into that. Understanding what the alternative is. I think that one thing that we have to understand is that if we're going to use AI, even the status quo, like if we're doing something like understanding what the current state looks like so that we can make a comparison about a new approach versus a current approach. And then the last piece is about, like I said, that we learn and manage risk. I think that right now we're at a time that a lot of people are doing pilots, and I think that pilots are helpful. But I think that if you're not learning how would you scale something if the pilot does succeed is equally a failure of, like if you found something that works and you don't have a way of driving scale, that's also a problem. So foremost, I think it is really important to evaluate use cases more than tools.

[22:19](https://youtu.be/KTWpAWBLino?t=1339) So AI, like, you know, like any other technology is too broad to be thought and to be able to evaluate, to be able to understand whether it works or not. You know, like what we put on the website as a technology is, you know, like there's ways in which we can put things that are on the website in ways that are effective and efficient and things that we know that we wouldn't want to put on the website. So we don't want to put confidential information from our lawyers or from, you know, but those are the kinds of things that we've become accustomed because we're so used to working in some of these other technologies. But I think like going back to understanding what is the use case that we want to give to the tools is really helpful and important because the tool enough itself is not going to tell you whether it works, whether you're actually able to solve a problem. That's more when we think about how this applying the tool impact.

[23:19](https://youtu.be/KTWpAWBLino?t=1399) So what do we mean by focusing on the goal? So we need to understand who are the users that are going to be impacted by the solution? Like who's the person whose problem we're going to try to solve? What is the problem that they experience? Why is this a problem worth solving? Like, I think that in government, we have so many different things that we're responsible for. The understanding if it is worth prior, it's not that a problem might not be worth solving, but there might be other problems that have a higher impact than we should be focusing on those problems. And lastly, I think that being able to understand how is it that we recognize that we have succeeded if we've reached this solution? Like, how do we know that this actually worked or didn't work? So usually I like to think about like three area, like three types of metrics when thinking about technology. So like value, like how well does a tool help us solve a specific problem? Cost, how much money does it cost?

[24:20](https://youtu.be/KTWpAWBLino?t=1460) And it can be money and time as well. You know, it could be the time that it takes to build or maintain risk. So understanding how often does something fail and also understanding how does it fail, what is the mechanism by which it fails and what is the context in which it fails? So when we are trying to understand if something works, and I think that this is an area right now, it seems we're at a time where it seems like we have either too many people trying to do things with AI that we think that maybe they shouldn't do with AI, or we can be of the thought that maybe there's too few things that are using AI and we think that more people, that we should be using more of it, more AI. And for me, I think that it is important to understand what is the counterfactual, what is the thing? If we were not going to use this tool, how does it work? Or as Alexander pointed out, a lot of times you might want to understand what a different path, like using a regression analysis versus using kind of like one of these like transformer models.

[25:30](https://youtu.be/KTWpAWBLino?t=1530) So I think that the questions that we ask ourselves when we're thinking about this, like how is this problem currently solved? Like what if, and I'm going to walk you through an example for like end-to-end through these steps that we've gone through recently. Secondly, what are the risks and the costs of the current approach? So one thing that we also need to keep in mind is there's a lot of things that we do in government that don't work very well. And it's not because we don't try hard, our work is difficult, but understanding that maybe the status quo might be very imperfect and actually like some marginal improvement might actually be quite good. Understanding where those potential opportunities for improvement fit. And again, being able to, so that we can do a comparison that is a little bit more apples to apples when we go and start innovating. So some thoughts around again, like starting small is good because it helps us understand like is an approach feasible?

[26:36](https://youtu.be/KTWpAWBLino?t=1596) Is it reliable? Like working through some of the concerns that Alexandra had shared. But I think that pilots for the sake of pilots for like a press release are actually not very useful and they can be frustrating, especially if you've, they can be frustrating because they could feel like you're either wasting time because you're trying a lot of things that might not work or there might be a waste of time because you're trying to do things that do work, but you don't have a way of scaling them. Generally speaking, I find that like chatbots are helpful for prototyping, like trying something out, but they're actually not very, usually they're not the right solution at scale. They're not the right interface because they tend to be less reliable. It is actually hard. You need to make sure that someone's asking the right question to get the right kind of answer. But most people don't know what is the type of questions that they should be asking when they encounter a chatbot. And the last piece would be, I think that being really explicit about what are the things that you want to learn in each phase upfront, makes it clear for you to know whether something's working and you should invest more time and money to do something.

[27:44](https://youtu.be/KTWpAWBLino?t=1664) So in Boston, we follow kind of this like three-phase approach that we think about prototypes, pilots, and production. And why we break it into the three is that it gives us the opportunity of having a mindset where we're not trying to solve everything at the same time. We're trying to solve this specific kinds of questions at different phases. So at the prototype stage, we're thinking is the thing that we're trying to do feasible. So for instance, if we want to do with our recent permitting work, if we want to build a taxonomy, we know that people request permits to do things that are not the way that we think about permits in government. So like most people don't want an electrical permit. They want to remodel their kitchen. So when we start thinking about like, hey, could we have AI help us build kind of this taxonomy or how could AI help us build a taxonomy of experiences, trying out how we could get like AI to process text to build like the type of experience that someone's intending might be a good way just with a chatbot might be a good way of understanding if it is possible at all to do this.

[28:55](https://youtu.be/KTWpAWBLino?t=1735) Starting to define who would use the product, who would use this tool and why would they benefit. In our case, again, like so much of our permitting process has been built to help us administer the process, but it hasn't been built to make it clear for people about what they need to do. In our case, like we thought, yeah, we're solving a problem for our constituents. Starting to think about like, how does the product integrate into the current process and systems? Very rarely are you gonna buy something that is going to live in isolation. You're like, you bought something that's gonna solve all of your problems and vendors are usually probably telling you like, buy this thing and all of your permitting rules are gonna go away if you buy this one solution. But we know that that'd be nice, but that's not usually how government works. Everything that we buy and we put in place functions in the context of a broader set of systems and problems that we have. And how does this fail? Like understanding, and particularly with generative AI, we know that a lot of these tools are heuristic.

[29:58](https://youtu.be/KTWpAWBLino?t=1798) They're like work on this like... So, if you answer some of these questions to your contender, you're like, oh, it seems like we have a feasible approach. We understand roughly how it would fit. Then we can start thinking about piloting. And piloting is an opportunity to deploy something with some level of scale so that we can start getting more data around how well it performs. At that point, they're starting to refine the technical approach. We're starting to refine where does it integrate? We know, for instance, what are the processes that we have to make up for the fact that it might not work all the time? How do we catch errors? We start really mastering the unit economics. We know if this is the system that we're going to use at scale, how could we afford it and how much does it cost? And again, it helps us understand in government, we don't usually have more money to pay for something. So, usually you have to replace something when you're buying something new. So, you start understanding what would you replace? What would need to be true for you to replace? And finally, if all of those things seem good, you go to production and say, all right, this is a new system. And it's still important to continue to evaluate. It's important to keep understanding how is it that you're going to measure whether this thing's working.

[31:36](https://youtu.be/KTWpAWBLino?t=1896) It might start being a good opportunity to think if a particular solution was helpful at solving a particular type of problem, are there other problems that we have that would be useful, that we could use this with? And again, this is a really good way of scaling a solution across the organization because you're not starting from scratch. You might already have a technical approach. You have a good understanding of the cost model and how you're going to scale it. You're just trying to translate those learnings into other contexts. And again, understanding the limitations of the system. So, let me give you an example. So, search on Boston.gov. Why is it important? 67% of people tell us that they trust the content that they find in Boston.gov, and that is the place where they would go to find information about the city. Search on Boston.gov. We use Drupal. We have a content management system to manage tens of thousands of pages that we have. It costs us about $20,000 a year to pay for the search tools that allow us to do search on Boston.gov. Risk? Unfortunately, 9 out of 10 people that use search, the feedback that they give us is negative, and they tell us that they can't find the thing that they were looking for. Anecdotally, talking to users and myself being a user of our website, I usually just would go to Google to try to find the right page, unless I knew the right page. So, it's a thing that doesn't cost us a lot of money. It's really important, but generally, the current state, not very good. So, we started going through this process of prototyping, piloting, and then going in production. So, we started first by enabling AI search on Boston.gov in only one page. So, what does AI search mean? It means two things. Actually, let me see. Yes, I have here explaining what it is. So, AI search means that we're searching over the website in a

[33:40](https://youtu.be/KTWpAWBLino?t=2020) way that is represented as these word vectors. So, we are able to find semantic information. We're doing semantic search. So, for instance, the word cat is related to the word pet, and I might have an article about pet registration, but I might not have something about cats, but it is relevant to the thing that you're looking for, and this vectorized search is able to retrieve it. And the second piece that we're able to do is create these summaries. So, it's not a chatbot, but it's like pulling, because we have a lot of contents that are, again, the authoritative thing that we tell residents, but the information that you're looking for might exist across a couple of different articles. We create a summary out of the articles that might help you get the information you're looking for. It costs us roughly the same amount of money that it costs us to do the traditional search, but it is almost 3.4 times more valuable. Like, people are reporting positive satisfaction 3.4 times a little bit higher than they did before. The number changes, by the way. So, what I'm showing you is, like, so, when we started prototyping, we first were trying to figure out, is this something that would even work? So, we spent a little bit of time.

[34:57](https://youtu.be/KTWpAWBLino?t=2097) My team was like, hey, can we spend some time building the integration between Drupal and we use Google Vertex AI to do this. I said, yeah, let's give it a try. We built the integration, and we started to learn about, like, it was very messy at the beginning. We started restricting the content only to certain types of content, and we found out that when we actually enabled broader search across all the content, it actually was more accurate, which was a little counterintuitive. We thought that by restricting, we would get more accurate answers, but our website's a little bit messy, and we have content sometimes that is misclassified, and then broaden up the search. When we felt that we were comfortable, it's like, this seems like it works, and it seems valuable, we expanded it to 50 pages on the website, and then all of a sudden, over the period of over six months, we got about, like, 8,000 pieces of feedback that this is the information that's driving this dashboard. So, all of a sudden, we had a fair amount of confidence around how much more reliable AI search was than the traditional search, and it got us to the point that by the time that we had to decide whether we're going to renew the traditional search or not, we said, let's just go with the new search, because it just seems to work better, and it costs us roughly the same amount of money.

[36:19](https://youtu.be/KTWpAWBLino?t=2179) It fails in different ways, like, particularly with the summaries, if you are doing traditional search, you might not be able to find the article that you're looking for. With an AI-generated summary, at times, it can hallucinate. So, at times, it can give you inaccurate information, and we gave people the ability of submitting and complaining if they thought that the information was a hallucination. It just, in our estimation, is still overall a more valuable tool, because not being able to find any information is worse than finding information. People are smart, and then we're providing the links where, like, it seemed to fit the right risk-reward analysis. So, some parting thoughts, and a quick lesson learned on the Boston.gov thing. So, other people might be using existing tools in unexpected ways. So, one of the things that we found out as soon as we went to AI Search, AI Search should work better for constituents, like, it does seem to work better for constituents, because they don't know what they're looking for. But when we went live with AI Search in Boston.gov, we found out all of the city employees were coming to us, like, hey, I can no longer find the article, because I'm used to typing a keyword, and I know what to find.

[37:39](https://youtu.be/KTWpAWBLino?t=2259) So, we started to learn, hey, you know, like, the users for Boston.gov are not only our constituents, they're our employees. But we might want to have different search experiences for both, because the reality is we would never want to deprecate and have a worse search for the public than for our employees, because it wouldn't make sense. But it's a little bit unexpected. Building something that works but is hard to maintain is still something that would be, there's a lot of people that get nervous about, like, should I build it or should I buy it? Well, building it sometimes helps you understand that there's something that you're willing to pay money for, that because it is reliable and useful and valuable, and then you just now know how much money you're willing to pay for. So, the third piece is, like, architecting for change. So, like, there's some, like, so much of what's happening is changing, like, there's new models, there's new technology providers, there's new ways of incorporating stuff. So, like, it might not be initially convenience of buying something that is really tight from an ecosystem perspective might feel good. But I think that you want to buy stuff that gives you flexibility later down the road. And then the last piece is you start evaluating. I realized at the end that we hadn't even said how to evaluate. And if we have time, I can show you, like, an actual evaluation.

[38:57](https://youtu.be/KTWpAWBLino?t=2337) You have to understand if the system's generating answers in a way that is dynamic, like, is it always changing? Or does it generate things once? So, I'll give you an example. One that is, like, static, we create summaries for Boston.gov for what the council voted on. That's a static generation. It generates it once, and then the summary stays there forever. Or, like, when we do translation for websites on Boston.gov, it does it once. So, if I know, relatively speaking, how often the translation or the summary doesn't work, that number, I'm, like, I can make a decision about whether I want to switch my approach. But I generally have, like, a static number in mind. Whereas if the system's dynamic, that's going to be actually a little bit harder to evaluate. And the last piece is, again, I think it's really important to understand the context in which the tools are being used. Is this a context in which there's some supervision? This is where bureaucracy actually can be quite helpful. Like, if we have a process by which we revise the outputs of something, and we have a review, we have something, understanding what those controls exist are really helpful in knowing what level of certainty you need to have before you're willing to move forward with a system. So, if we have time in the questions, I'm happy to show you, like, an actual sheet. So, like, the last piece would be, when we talk about evaluation, I would say you want to try a large number of scenarios, like, 100 or more. And you want to try scenarios that might be not very common. Particularly scenarios that you think that might have, like, an equity implication. Like, things that you're, like, if it failed for this particular group of people, it would be bad. So, just creating hundreds or at least 100 scenarios that are, that where you could go back and see if the tool for that particular use case worked in a particular way, that would be my advice. Like, it's not just trying five things, like, into 100, so that you can start getting a real sense of the, kind of, like, the statistical

[41:01](https://youtu.be/KTWpAWBLino?t=2461) validity. And as Alexandra mentioned, robustness, it can get complicated. You could go and find help from smart people like Alexandra. But at times, try it at least 100 times. And then take it from there. But, all right. That's the end of my presentation. Thank you, Santi. I'm going to go

**Eileen Twiggs**

[41:23](https://youtu.be/KTWpAWBLino?t=2483) ahead and spotlight Alexandra here so we can have a little Q&A. We do have some questions. We've been pulling them out of the chat. So, thank you so much for those of you who have been labeling your questions as questions. Santi, I'm going to pose one to you first, only because you just finished your presentation. So, we have a question. At what stage do you start considering the organizational aspects of implementing this into the agency? Can you repeat that, Sally? Like,

**Santiago Garces**

[41:54](https://youtu.be/KTWpAWBLino?t=2514) at what point did we start involving, like, people from the departments or agencies?

**Eileen Twiggs**

[41:58](https://youtu.be/KTWpAWBLino?t=2518) Yeah, the organizational aspects of implementation into the agency.

**Santiago Garces**

[42:06](https://youtu.be/KTWpAWBLino?t=2526) I will say this is one that we've learned that we can do better. So, for instance, we got a lot of negative feedback once we went live with Boston.gov with AI Search because we actually hadn't told people. Because in our mind, again, it was a public-facing tool. So, we could have probably done a better job of reaching out, like, the folks in 311, which used Boston.gov, kind of, like, as their knowledge base. But generally speaking, we would want to do it really early in the process. Because to some extent, like, when we think about evaluating AI tools in a particular context, we think you need to be more of, think about it as a social scientist more than a computer scientist. Because whether the outputs are valid and useful, a lot of times there's not, like, as Alexandra mentioned, if it's math, you can test that. But with making a summary of the counselor, that's a little bit hard to, like, mathematically go and prove how good should a summary be. So, I would say, like, you want to get some of those subject matter experts early on. And in a lot of cases, you want the people from the agencies and these human beings to be the ones that are setting that criteria of how good does something need to be, how much better does it need to be for you to be willing to accept the risk. And again, it's not straightforward.

[43:25](https://youtu.be/KTWpAWBLino?t=2605) You're having to make trade-offs. So, I think that having that buy-in early on from the people that are going to have to live with the trade-offs is really, really important.

**Eileen Twiggs**

[43:36](https://youtu.be/KTWpAWBLino?t=2616) By the way, Alexandra, any question that I pose to Santi, you can feel free to jump in with perspective and vice versa as well. So, we have a couple of questions here that are pretty tool-specific. So, let's see, well, that are more focused on tools. So, which LLM has the lowest occurrence of hallucinations? And I just sort of want to layer in there some comments from earlier that, you know, sort of, like, bias and hallucinations can be mitigated by good prompting. So, with sort of that lens in mind of sort of the role of good prompting in eliminating bias or significantly mitigating bias and elimination, do you have a recommendation on a tool that you think is sort of better than others for hallucinations?

**Aleksandra Korolova**

[44:31](https://youtu.be/KTWpAWBLino?t=2671) So, I'd say that you can mitigate hallucinations, but you can't perfectly solve it. And sometimes when we're talking about mitigating, it goes from, it hallucinates 30% of the time to now it hallucinates 20% of the time, right? And whether the AI tool hallucinates also really depends. on the kind of question that is being asked, on whether this tool has access to search, whether it has access to some verification tools, what is the model, how much thinking time it has, and so on and so forth. So there isn't really one answer that one can give on what is the best tool. You really have to look at what it is in your specific use case, with your specific technology and data access and mitigation techniques that have been given.

**Eileen Twiggs**

[45:36](https://youtu.be/KTWpAWBLino?t=2736) Santi, we have a related question that is actually directed for you, for the City of Boston. So how did you actually approach the translation work using the prototype pilot and production strategy? So what reviews or controls did you put in place for actually reviewing

**Santiago Garces**

[45:52](https://youtu.be/KTWpAWBLino?t=2752) that translated content? So for whatever's worth, that one would be, it's not in production yet. We're doing a lot of the technical work to even be able to put it in production. We're kind of somewhere between prototyping and piloting. So what we've been doing is, there's some work that happened a couple of summers ago where we started to take speeches and things that the mayor had written and then use AI to translate into different languages and then have native speakers. We had like a professional translation of the document and then we had native speakers give us the evaluations, like how good is the translation or not. Again, Boston has 11 official languages, so we have to translate all the official documentation, but like in Arabic, Russian, Mandarin, Chinese, simplified and unsimplified.

[46:56](https://youtu.be/KTWpAWBLino?t=2816) And some languages are actually, there's less native speakers of those languages and then we want to make sure that those, to Alexandra's point, tend to be where we see more hallucinations or issues. So again, we're at the point that we're trying to figure out when do we, aligning with our language access team, when should we translate what kind of documents and when should we not use AI translation, to some extent. But also going back to the point that I made, currently, if you go to the website, you're using AI translation, you're using Google Translate, the kind of free version of it. So the test that we have made showed that once we start using kind of tuned models or more specific models, you start getting higher quality translation that improve the quality. So that's something to keep in mind.

**Eileen Twiggs**

[48:01](https://youtu.be/KTWpAWBLino?t=2881) All right, thank you. So we're getting lots of questions in now. We're going to do our best to curate as many of them as we can. So this is a question for both of you. So for AI projects, what should we test in prototypes and pilots? What can we standardize? How do we train staff to pilot safely? Do you have any best practices to share on how to bring in X, journal partners? And you, Santi, had a very specific question in here, like about when would you start running scenarios at the proto stage or wait until the pilot stage? So I'm going to sort of lump those two together for you.

**Santiago Garces**

[48:45](https://youtu.be/KTWpAWBLino?t=2925) I'd say, like, you have to figure out if something works or not before you start trying to figure out how reliable it is. And it might take a little bit of work to get it to work for the first time. So I think, like, it's OK. And again, it's to some extent, this is where, like, if you if you're focusing on a problem that's worth solving, like, hopefully it means that it is worth spending a little bit of time trying. This is also where you start learning why it doesn't work. So, for instance, one thing that we have found out is, like, large language models are actually not very good at spatial reasoning. They're, like, they're constructs that relate words to each other. But, like, so, like, if you're trying to do, like, yeah, like, what room is bigger of these? If you have a picture, it might not do a good job of answering that question, like probably is making it up. So I think that that's where I would say, like, kind of my sense, like you are leaving the prototype space, the prototype stage once it's working and you're getting it to work well enough that you're like you want the larger end, you want the larger evaluation set, but you're still able you're still willing to change some things about how it works. It's just like it's just starting to look like it actually works. It's actually going to solve the problem. And for whatever, like, you might decide at that point, hey, this is something that's worth doing. It seems like it works. I'm going to go and try to buy something that you might want to be like. I don't want to continue to be experimental, but you at least know that that is a thing. It's going to be harder for someone. Like, I think the procurement process is not going to save you from buying something that doesn't work.

[50:29](https://youtu.be/KTWpAWBLino?t=3029) You have to try. You have to do the evaluation yourself at some point either. In my sense, like, it's better to do it a little bit upfront because then you know what you want to buy instead of just, like, relying on the vendor to tell you that something works or not.

**Aleksandra Korolova**

[50:45](https://youtu.be/KTWpAWBLino?t=3045) Yeah, I really agree with everything Sansi said. I think if it is a problem worth solving, then you have to really invest your own effort in evaluation and figuring out whether this is the right tool and how it works and what its limitations are. There's kind of, no shortcuts to that. I'd like to address the last part of the question of, like, how to bring in external partners. And with my academic hat on, I'd like to say that pretty much every academic would love to be this external partner and provide their expertise because this is the, you folks are really are the ones deploying the technology in a way that really affects people. So, to the extent we can help with that, that's an exciting thing. So, really approaching any academic from the institution that is perhaps geographically closest to you because then it's easier to do meetings or anyone who's written something on the topic or related topic, I think, would give very positive results. Thank you both. We're going to switch over a

**Eileen Twiggs**

[52:03](https://youtu.be/KTWpAWBLino?t=3123) little bit to a related topic, but more on the risk side. So, Santi, back on one of your metrics to success slides, you had, you know, sort of mentioned risk as a factor. And folks are wondering if you and Alexandra could speak a little more deeply on that subject. How do you actually, like, what kind of data do you collect to assess risks? How do you actually do the risk assessment process with a little more maybe specificity, if you can?

**Santiago Garces**

[52:35](https://youtu.be/KTWpAWBLino?t=3155) Yeah, I can. I can try it first. And let me see if I can pull up. There's a document that, well, maybe we don't have time to do that. Yeah, I was going to say, we only have about two minutes left. Two minutes. So, long story short, like, I have it here at the ready. I'm going to share my screen just really quickly. So, I think that it's, there's probably more sophisticated ways that Alexandra can tell you, but, like, this is one example of a sheet where we evaluated. So, we had the original word, we have the summary, and then it's just a one if it's accurate and a zero if I don't think that it's accurate. And I was the evaluator in this case. So, it's just, like, for me, the overall reliability rate is, like, out of my, and I did a random sample. I took, like, random pieces out of, like, the entire thing that needed to be summarized. And, you know, what percentage of it did it summarize and what percentage it didn't summarize.

[53:35](https://youtu.be/KTWpAWBLino?t=3215) There's some qualitative piece, like, one of the virtues of evaluating is you learn how it fails. So, like, you start seeing that sometimes when you're doing the evaluation by hand, you start seeing that actually it's not consistent. Like, sometimes, like, it will do a really good job of summarizing something that seems really difficult, and then it will fail at a thing that seems simple. So, it's, like, that's helpful because you can't count the just simple, a simpler input text is going to lead to a better summary. And you kind of lose that texture if you're not doing it by hand. So, I'd say, like, my recommendation is, like, you should at least try to go by hand sometimes, especially early on in the project. So, you start getting a sense of what it is. Because that, and you, I'm the subject matter expert that's using these tools, knows if that is appropriate or not. Again, in our case with the summaries, like, if you, if we didn't have the summaries, you'd still have to read, like, the only thing that a resident has is this, like, obscure text. So, it's, like, even giving, like, a not very good summary is better than having no summary at all. We actually ended up with the reliability rate in the first pass was, like, 73%. So, we didn't think that it was good enough. We still, our production version didn't include the summaries. But one thing that we've noticed is that the models have gotten better. The summaries, the rate has gotten better. But I haven't had the time to reevaluate. So, we haven't changed the production. But anyways.

**Eileen Twiggs**

[55:04](https://youtu.be/KTWpAWBLino?t=3304) Aleksandra, one minute to add to that, if you'd like to.

**Aleksandra Korolova**

[55:08](https://youtu.be/KTWpAWBLino?t=3308) No substitute to having a subject matter expert evaluate an actual use case that the subject matter expert knows about. No substitute for that.

**Eileen Twiggs**

[55:20](https://youtu.be/KTWpAWBLino?t=3320) Awesome. Thank you so much, both of you. We are actually at time. So, I know there were questions that we didn't get to. And I apologize for that. But, you know, I do want to mention, for those of you who are really curious about this topic, we are launching a whole new series coming this Wednesday on public AI, which has a very, we're going to dig deeper into the AI procurement decision making process there. So, please mark your calendars on that. Please join me in thanking Aleksandra and Santi for sharing their expertise with us today. And thank you all for your great participation in this webinar. I'm sorry that we did not get a chance to get to all of your questions. And a quick reminder that we, of course, will be sending you a survey. So, please give us some feedback. We read all of it. Thank you so much, everybody. See you soon at one of our upcoming workshops.

**Aleksandra Korolova**

[56:19](https://youtu.be/KTWpAWBLino?t=3379) Thank you.

**Eileen Twiggs**

[56:21](https://youtu.be/KTWpAWBLino?t=3381) Thanks, everyone.

* * *

Original page: https://innovate-us.org/how-not-to-buy-stupid-ai-a-practical-guide-to-evaluating-ai-products-and-tools · Licence: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org