---
id: "workshop:ai-tools-showcase-whats-out-there-and-how-to-evaluate-it"
type: "InnovateUS workshop"
title: "AI Tools Showcase — What's Out There and How to Evaluate It"
url: "https://innovate-us.org/ai-tools-showcase-whats-out-there-and-how-to-evaluate-it"
date: "2026-10-19"
duration: "60 minutes"
series: "AI for Public-Sector Procurement"
instructors: ["Santiago Garces"]
moderators: ["Denise Linn Riedl"]
recording: "https://www.youtube.com/watch?v=hRdjhYdSmi4"
transcript: "automatic, from the recording"
source: "InnovateUS (innovate-us.org)"
licence: "CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org"
---

# AI Tools Showcase — What's Out There and How to Evaluate It

InnovateUS workshop · 19 October 2026 · 60 minutes

See how public procurement teams are evaluating AI tools, including contract analytics, RFP drafting assistants, and AI-enabled ERP features, from people who've actually used them.  Hear how municipalities have selected and scoped a tool for its own procurement needs.

## Learning goals

- Apply a set of criteria for evaluating AI procurement tools beyond vendor claims

- Identify questions to ask vendors about data use, accuracy, and accountability

- Compare tool features against your organization's specific procurement needs and constraints

## Speakers

- **Santiago Garces**, Former Chief Information Officer, City of Boston

## Moderators

- **Denise Linn Riedl**, Head of Public Sector Technology, Partners for Public Good

Part of the series [AI for Public-Sector Procurement](https://innovate-us.org/workshop-series/ai-for-public-sector-procurement-fall-2026).

Recording: https://www.youtube.com/watch?v=hRdjhYdSmi4

## Transcript

Automatic transcript of the recording: names and terms can be misheard, and the times can be a few seconds off. Each time links to that moment of the video. The transcript alone: https://innovate-us.org/ai-tools-showcase-whats-out-there-and-how-to-evaluate-it.txt (plain text) and https://innovate-us.org/ai-tools-showcase-whats-out-there-and-how-to-evaluate-it.json (every segment with its start and end second).

**Audience**

[0:00](https://youtu.be/hRdjhYdSmi4?t=0) I'm Matt Oger. I'm the Chief Learning Officer for NASPO. It's great to see some familiar names in the attendance and in the chat, so it's also great to be with you. So welcome back to the Innovate US Workshop Series, AI for Public Sector Procurement. It's co-hosted by NASPO and Partners for Public Good. This series has equipped procurement and other public sector professionals with the practical skills to leverage generative AI to improve competitiveness, effectiveness, and efficiency in procurement. So it's been designed for hands-on learning, as you'll see today, with each session building on the last to give participants actionable tools they can apply immediately to serve the public better. So I know we have received great feedback from some of our members and some of the attendees on the workshop series so far. I hope you are all getting some great tips and actionable items that you can take back to your office and apply. I also just want to say, and I think Christine will put the link in the chat, if you'd like to check out our NASPO website, we have a number of resources and publications on there related to AI as well as just general procurement topics, so please take a look. Those are free and available for you to use. So today's final session in our series, AI Tools Showcase, what's out there and how to evaluate it. We will look at how public procurement teams can evaluate tools, and we're going to hear from people who've actually used those tools. So now I will turn it over to Denise Lynn Riedel from Partners for Public Good, who is moderating this

**Denise Linn Riedl**

[1:35](https://youtu.be/hRdjhYdSmi4?t=95) session. Denise? Thanks so much, Matt. I'm going to share my screen and introduce myself and also tee up our wonderful guest, Santi Garces. I'm Denise Lynn Riedel. I'm the head of public sector technology at Partners for Public Good. PPG, as we call ourselves, we innovate in the back office of government. So that looks like supporting procurement teams, legal teams, budgeting teams as they modernize, in this case, adopt AI and future proof for evolving needs and challenges in government. I lead the technology team at Partners for Public Good. We're a new function of PPG, and so my team focuses on big technological change, implementing AI in government, measuring what works and what doesn't, and just supporting teams as they experiment and learn, and we like to scale all those insights to the whole field. So I'm very excited today, and Santi Garces and I have intersected quite a few times in our career in lovely ways, and so it's just a lovely reunion to be on with you today. But Santi, his current role, he is the co-founder of the AI Lab for Cities at Harvard University. Before that, he was the chief information officer for the city of Boston and held many roles in several other cities. But Santi, I'd love to pass it over to you to talk more about yourself.

**Santiago Garces**

[3:05](https://youtu.be/hRdjhYdSmi4?t=185) Absolutely. So it is, as Denise said, it is always great to be here with her and then with all of you. So as Denise mentioned, I used to be the chief information officer in Boston, and throughout the course of my career, I've had the opportunity of leading the procurement, the development, and the implementation of technology, including AI, even before we called the thing that we call AI AI, and the thing that we call generative AI came into the picture. So very excited to get into it and figure out how is it that we can work together on what is the best practice and how is it that we can evaluate and make sure that the things that we're buying

**Denise Linn Riedl**

[3:53](https://youtu.be/hRdjhYdSmi4?t=233) or how we go about and build them works. I'll also say as a fun anecdote, Santi and I both had the same job. Before I had, I was the head of public sector technology here at Partners for Public Good and thinking nationally from a nonprofit perspective. I was the chief innovation and information officer for the city of South Bend, Indiana. I would argue one of the best cities in the United States, and Santi also had that role. I got to baton, accept the role from him as he left and I joined. Yeah, you know, public service is a relay race and

**Santiago Garces**

[4:37](https://youtu.be/hRdjhYdSmi4?t=277) it is always good to know that you can upgrade and after having some time in government that greater, smarter people can go and take you farther. That's very sweet, Santi. I don't know

**Denise Linn Riedl**

[4:50](https://youtu.be/hRdjhYdSmi4?t=290) about the upgrade of that, but I certainly had a lovely foundation to build on. It's a blessing to inherit a great foundation that someone has built and then build on and do more innovative work on top of it. So, Santi and I are going to be having an awesome conversation today and just like Matt said, we're building on top of the progress that the last two workshops have made. We've overviewed the procurement workflow, where AI fits into it. We'll do a bit more of that today. What Santi and I will talk about today mostly is we're going to discuss specific types of tools that governments can adopt. We're going to talk about building things, buying things, and trying things because all of those are available to cities, counties, states that are looking to implement AI in their procurement workflow. And we're also going to talk about evaluation. So, one of the most important parts about any AI investment is making sure that you're better off and you're defining success and then you're measuring against that success. So, we're going to have a conversation about what that looks like today in various contexts, and we'll also play a little game just to demonstrate how all of that works. But you might have seen a version of this in a previous workshop from NASFO and Partners for Public Good. This is kind of the way we're thinking about AI across the procurement lifecycle, so across planning, solicitation, vendor engagement, evaluation, and award, and then continuous contract management. AI fits into many parts of this. There are lots of homegrown tools that you can use across each of these use cases. Santi is going to be highlighting a case today where planning and solicitation has been streamlined by something built within the city of Boston to augment their procurement staff.

[6:49](https://youtu.be/hRdjhYdSmi4?t=409) There are also things that are built within e-procurement and ERP systems that support solicitation generation in the second pillar or support contract management in this fifth pillar. And then, of course, there are AI-native tools that can also support these things. One example would be what you might see from a Hazel AI that might support solicitation generation in this second pillar. Something like a Procurated, which supports vendor engagement and contract management. What's interesting about this field is it's always changing. So, as an example, there used to be an AI-native tool called Evisort that I think was a recently acquired by Workday. Now it's a part of Workday's ERP functionality. You'll notice that all of these things change. There's going to be acquisitions and mergers. The impact with the exact tool list is that you might

**Bonnie McGilpin**

[8:05](https://youtu.be/hRdjhYdSmi4?t=485) have lost Denise for a second. Let's see if she can get her audio back.

**Santiago Garces**

[8:22](https://youtu.be/hRdjhYdSmi4?t=502) Well, if I can buy Denise some time while she comes back and finishes her point, I can talk a little bit about, I think that she wanted me to discuss the work that we had done with BidBud. And then I can probably talk a little bit about what it aimed to do and how it worked. Yeah, right now I have to live dangerously without a facilitator. So, as Denise was pointing out, there's one of the things that's interesting about procurement in government is that it is highly regulated. So, it means that following the procedures to make sure that you're buying things in the right way is complex and complicated. And often the people that might know what they need to be able to execute a program or service don't necessarily know how to go and procure it. And then there's this grade asymmetry.

[9:25](https://youtu.be/hRdjhYdSmi4?t=565) So, one of the things that we started to consider in Boston was thinking, you know, AI, and it was kind of inspired by the thought, even just goods and services in the city of Boston are regulated by a state-wide regulation called Chapter 30B. That's just for goods and services. There's like other chapters and other pieces of regulation for other types of procurement. The shorthand book, like what we tell people, like whenever someone's trying to, wait a second. Oh, look at that. Questions for me. Well, I'm going to keep talking. Sorry, I have ADHD and I got distracted by the screen. So, what we started to figure out is like, if we were able to empower our buyers throughout departments, it's a hypothesis, like could AI be a vehicle to get people to ask better questions from the people in procurement and to have better quality drafts earlier of the types of procurements that they were going to do, but seemed to be useful. And again, as I mentioned, like just the shorthand, the quick guide for Chapter 30B is like almost 200 pages long. So, like just hoping that people not only have read that, but remember what are the relevant parts and sections, like what applies and what doesn't apply is a task enough itself.

[11:20](https://youtu.be/hRdjhYdSmi4?t=680) So, we decided to go and build BidBot ourselves. And basically what that meant is that we needed to first identify what is the body of knowledge that we wanted BidBot to reference. And that was and to some extent, like even before that, going in and figuring out who is the people that we're going to determine. Denise might be back. We'll keep talking. So, us in IT partnered with the folks in procurement and said, you are the subject matter experts ultimately. You're like managing and regulating how procurement works. So, you are going to be the ones that curate. Oh, Denise, you're back. I'm going to finish just the point. But basically, so the first piece is even from a procurement standpoint, when we're deciding, should we buy this or should we build this? We said, we're going to try to build this. We said, the people that are going to be determining how well this thing works are the people that are the subject matter experts. In our case, it's a little bit metacognitive. It was procurement. And then throughout the development process, what we were saying is like, it is the people in procurement that are going to assert whether they are going to be evaluating the quality of the tool. How well can BidBot respond to questions related to procurement? And to some extent, what we said is like, look, if this works, and we had some estimation of the cost, and we thought that it would cost us about like $400 a month to run BidBot for the entire city. So, I said, like, there's some underlying costs around maintenance and support. So, they said, if we build it, we'll get a better sense to understand how these things work. At the end of the day, if it does work, we can make the decision, should we buy this if we don't want to maintain it? Like, if we think that we don't have

[13:21](https://youtu.be/hRdjhYdSmi4?t=801) the capacity to maintain it, but at least we'll know that there's a thing that actually works that we want to keep. But let me see if Denise is back in. I can. Oh, you see, we're cut to my BidBot example. So, going back to the measures of success, and how did we decide whether we wanted to, like, how well did it work? We started working actually with some of the folks that, yeah, technology always fails us at the critical time. But anyways, we started to focus on, like, describing, and it took some effort to describe what does a procurement tool need to do? So, we started to say, all right, can it help answer some basic questions around helping people navigate what is the, what regulates, like, what statutes regulate a particular type of procurement? So, kind of, like, knowledge, you know, just, like, kind of, some knowledge reference. The second piece is, like, going and starting to think about, like, can it help us, could it help someone write specifications?

[14:50](https://youtu.be/hRdjhYdSmi4?t=890) Could it help someone identify the right type of contract? Could it help, so, like, there's the It's kind of intentional work, if not just going to say, does AI work for procurement or not? But going in and starting to describe what are the different components, because one of the things that we anticipated is that the tool might perform differently around different dimensions. And if we don't do the work of understanding really how is it that we think that the tool might be helpful, even though I'll preview, people started to use the tool in some ways that were unexpected, not necessarily in a bad way. But us having clarity around what did we think that it might do helped us be able to assess whether it was accomplishing those things. And we also set up front what are things that we knew that we wanted the tool not to be able to do. So we said we don't want the tool to be able to adjudicate and to evaluate solicitations, because that's ultimately the decision around whether something meets our needs or not is something that we think that a public servant should be doing.

[16:03](https://youtu.be/hRdjhYdSmi4?t=963) And then based on that, the folks in procurement started to build literally the kinds of questions that we would want, that we think that users might experience, and then what the answer might be around those specific skills that we laid out. And the nice thing is, as we started to do some testing, we were able to see the types of questions. Not only did we have to anticipate what questions our users would put forth to bid, but we actually saw what questions they were putting, and then that helped us make better evaluations. So to some extent, I think to the framework that Denise had laid out, trying and testing helps you get the data around how people are going to use it, which will help you build a better evaluation that will help you make a better decision around, is the tool working or is it appropriate?

[17:05](https://youtu.be/hRdjhYdSmi4?t=1025) And to some extent, the part that is nice in government is, even if what we decide right now is not optimal, governments don't disappear. Usually they don't go bankrupt like companies. So even if what we were trying to do is not perfect, we get the opportunity to learn and make it better and better as we iterate. I think Denise might be back. Denise, are you back?

**Denise Linn Riedl**

[17:30](https://youtu.be/hRdjhYdSmi4?t=1050) I am. Can you guys hear me? Yes. All right. Man, it's always so sad when that happens. Thank you guys for your patience. And honestly, the most important part about this webinar was Santi talking about bid bots. So I'm glad that we got there a bit earlier. Just to backtrack and reference some of the things Santi mentioned, for when you're considering any kind of intervention on your procurement team, just as Santi said, certainly you could build something like what he did in Boston. You could purchase something outright, whether it's a module onto an existing system that you have or some new AI native tool. You could also try something in Copilot, Enterprise Cloud, Enterprise Chat GPT. There's a lot of small things that you could do as a procurement team to experiment with AI in your workflow. And Santi, I think, has probably already talked a little bit about this while I was offline when he thought about measuring the success of bid bots, or really any AI tool, fall in one of three categories, faster, better, or new work.

[18:49](https://youtu.be/hRdjhYdSmi4?t=1129) How were you able to speed up or streamline something? How were you able to make a process better, like better quality, better checks? And then how were you able to free up, via the freed up work, do new things that you've never done before? These are the three things that we think about. And Santi, I'm not sure if you talked about the spectrum of the menu of evaluation that folks can think about. But I'll tee up Santi in a second to talk a bit about the randomized control trial that he did in Boston. But there's lots of lower weight evaluation methods that you can undertake as a government agency, everything from just self-reported impact to pre-post operational comparison. And then on the more scientific end, and rigorous end, difference in differences, and randomized control trials. Can folks still hear me?

**Santiago Garces**

[19:50](https://youtu.be/hRdjhYdSmi4?t=1190) Yes, we can.

**Denise Linn Riedl**

[19:52](https://youtu.be/hRdjhYdSmi4?t=1192) Okay. So, Santi, did you get a chance to talk about the origin story of BidBot?

**Santiago Garces**

[20:00](https://youtu.be/hRdjhYdSmi4?t=1200) Vaguely. Okay. I wanted to tell people that, again, when it came to solicitations, we had this hypothesis that we thought that it is difficult for people to understand over 200 pages, actually it's like over a thousand pages worth of regulations that they're subject to when they're trying to buy something. And knowing not only what sections, what pieces of that whole regulatory corpus apply, but being able to take action on them, that's hard, even, you know, like we do training, we have like a finance academy, but we thought that AI might be a useful tool for that precisely because it is a really good machine of words. And to some extent, the metacognitive pieces, we actually wanted to be able to evaluate the impact of using AI in government operations, and we thought that procurement was a really useful place to test that. So we chose doing our first test in procurement.

**Denise Linn Riedl**

[21:37](https://youtu.be/hRdjhYdSmi4?t=1297) Hey, Santi, can you hear me? Can folks hear me?

**Santiago Garces**

[21:46](https://youtu.be/hRdjhYdSmi4?t=1306) I can hear you now. Yeah. You were breaking up before, but now.

**Denise Linn Riedl**

[21:49](https://youtu.be/hRdjhYdSmi4?t=1309) Great. Okay. Excellent. This webinar is being powered by the hotspot on my phone, currently. But no, Santi, can you take us through some of the initial steps you took to how you got from idea to your first prototype?

**Santiago Garces**

[22:07](https://youtu.be/hRdjhYdSmi4?t=1327) Yeah. So, again, one of the things that we wanted to do is like, and it's like, we're trying to do it. So like, the first thing that we wanted to be able to test is like the technical feasibility. Could this be an approach like, could the use of AI help us support people in understanding and get access to procurement knowledge easier? So the first prototype was like very low fidelity. So we knew that the documentation around the government regulation, again, there's like chapter 30b handbook is public. So we just got a free version of ChadGPT at the time. I mean, this is long time, I mean, this is like over two years ago, like almost three years ago. So it's probably ChadGPT or CLAWD3 and upload the document and see if, you know, like when we ask a certain set of questions, like could it guide us and was it generally kind of accurate around like procurement thresholds around like the right type of procurement instrument.

[23:18](https://youtu.be/hRdjhYdSmi4?t=1398) So why did we do that is like, we just were trying to figure out, is this a thing that was even remotely feasible before we decided to go on and start building on this. So then the question is like, obviously like we didn't necessarily want to give, like we couldn't afford giving everybody ChadGPT licenses. So once we were able to assert that it looked like it was feasible, what we said is that's giving us enough signal that we think that now is the point that we are going to go and invest and build this solution. We leveraged some work that had been done by Northeastern University and the AI for Impact program where they had built an open source knowledge agent, which is still open source if you go to their GitHub. So they said, let's take that code and try to implement a similar kind of design to what we had just tested with ChadGPT. But now we're in the process of thinking more rigorously about like, what is the corpus?

[24:19](https://youtu.be/hRdjhYdSmi4?t=1459) Like I mentioned, like what are the documents that we need to put in there, refine the prompt and then actually starting to test. Because to some extent, after we're done testing technical feasibility, what we're starting to think then is, can we build a solution that would be reliable enough that if we gave access to a large group of people, especially with AI is like, it's all about statistics. It's not going to be perfect, but is it, are the errors rare enough and are the types of errors not problematic that we feel that we could give access to a broader group of people? So that was like, after we're done with the technical feasibility, that's what we started studying.

**Denise Linn Riedl**

[24:59](https://youtu.be/hRdjhYdSmi4?t=1499) And can you, do you have any anecdotes for some of the first, like you're talking to a group here that I think mostly based on the chat are procurement officers at the state, local, federal level. How did procurement personnel react to using it for the first time? What were some of the first things they tried? What did that look like?

**Santiago Garces**

[25:23](https://youtu.be/hRdjhYdSmi4?t=1523) I mean, again, this was like very early on, so I think that there's a fair amount of trust and anticipation and to some extent, like, again, it is interesting working with people around like where there are some folks were wondering how much money would we spend in the usage of the tool basically, would building BitBot put us over some procurement threshold? You can't, you can't take the procurement person out of the procurement. So like there are some questions like that, again, a little bit of hesitation around like if this worked at the beginning, it's like, if it didn't work, would it not work because we were not doing a good enough job configuring it? Or was it too early in the models, like in the life of the models to get it to work well enough? And then the second piece that started to be more interesting is like, if it did work, like the city had just made a big investment in creating this like internal services department with procurement, like there's an entire team of people that were procurement analysts and that were supporting the agency.

[26:33](https://youtu.be/hRdjhYdSmi4?t=1593) So it's a little bit of like, wait, why are we building this thing? Because the city has just made this investment in people. So like thinking a little bit about like the theory of change of why, why and how could this tool help that team as well?

**Denise Linn Riedl**

[26:49](https://youtu.be/hRdjhYdSmi4?t=1609) OK, and can you have you had to talk maybe while I was in virtual internet purgatory, Santi, were you able at all to talk about the RCT or like the Boston's approach for measuring success on this?

**Santiago Garces**

[27:04](https://youtu.be/hRdjhYdSmi4?t=1624) Not yet. And to some extent, and I think that that's a funny thing, because like to some extent, like partially how we were able, yeah, like we were going through this iterative process of trying to manage the risk around, should we, is this the kind of technology that we should invest in? How much money would we invest in? Because like, what is the level of value that it provides? One of the ways that we were able to get procurement to to to sign up, we said we are trying to when we're building this, we want the level of rigor that's even more than an RFP. You know, an RFP helps us guarantee one, that we have followed the right process. And to some extent, if you're doing a good job, not every RFP by itself does this. But a good RFP and a good procurement process lets you understand, like gives you a strong signal about like, how good of a fit is this tool gonna be, you know, like, how well does it solve a public problem? But we said, we want to go one step ahead.

[28:05](https://youtu.be/hRdjhYdSmi4?t=1685) And we're going to do a RCT. And for those of you who are not familiar, an RCT stands for randomized control trial. So this is where we're going to use very, like a very rigorous approach where we're going to get give access to a tool to, we're going to split people into two groups, randomly assigned, so that there's an equal probability that everybody has to be assigned either to the control group or to the, to the treatment group, as in, there's a random, there's an equal probability that you might end up being able to use the tool for this exercise, or not. And we said, we're going to, I had started to describe that we started to build like an evaluation framework around thinking, what are the types of competencies that we think that AI procurement tool could have. And, and then we build an instrument, we build basically like a quiz. And we said, we're going to do this experiment with our staff to try to see truly what are the effects of having access to this technology, because it is going to help us understand is this a technology that is worth investing on or not, with a very high level of rigor, it is expensive, like it takes a lot of time to do these kinds of things.

[29:18](https://youtu.be/hRdjhYdSmi4?t=1758) A lot of work and preparation, so I wouldn't say like, ever procurement needs an RCT. But I think that in a case where we're trying to invest in a technology that is like experimental, where there's a lot of, you know, public concern around like, is this useful or not useful, and there's a lot of noise, having rigor helps. So that's what, but we wouldn't, and I think that to the point of your slide, we wouldn't have been able to get to the RCT, if we hadn't done kind of a more incremental approach to evaluation, because we would have never wanted to put a tool that was going to be going through an RCT, unless we had a pretty... strong confidence, like we had a good hypothesis that it was going to work, that we thought that we might actually have something that was useful.

**Denise Linn Riedl**

[30:10](https://youtu.be/hRdjhYdSmi4?t=1810) What were the kind of KPIs or like metrics that you were really hanging your head on for BitBot?

**Santiago Garces**

[30:17](https://youtu.be/hRdjhYdSmi4?t=1817) So again, I think there's some outcome metrics that people have told us. I'll just highlight the negative first. I'll highlight what people have told us that we are not measuring yet, but just so that just being upfront, people are saying, well, you're not measuring yet whether the government is getting more public value for less money, aka like, are your procurements leading to more efficient purchases for the city? We said, yeah, like those are long-term studies. It's kind of hard. It's harder to assess because to some extent, you don't only have to assess the procurement itself, but what happened afterwards. So we said that's outside of the scope for the time being. But what we said is, we have this theory of change that a higher quality procurement faster means that we will spend higher time iterating over the procurement or that we can be spending more of our mental effort doing more higher value things. So what we started to realize is that a lot of times, even for fairly large procurements, we were not able to spend enough time on the procurement itself and making sure that it was of high quality. A lot of times we were reusing procurements from previous cycles. So we're just taking basically the last time that we did an RFP and then just making some small changes and then reissuing it. So that's kind of why we highlighted that. So you asked, what are the KPIs? So we mostly focus on speed. So how quickly could we build drafts? How quickly could we revise? What's the time to perform tasks? And accuracy. So that was one of the big concerns that people had, was how accurate this thing is. So we were like, how well can it respond to, what's the quality of the information? And then the last piece is around user satisfaction and user perception. So Boston is full of very smart people. So most people started very skeptical.

[32:19](https://youtu.be/hRdjhYdSmi4?t=1939) They're like, I think that AI is snake oil. So one of the third KPI that we're measuring is, what was the percentage of people that thought that AI was actually useful for government? So speed, quality, and then perception of usefulness by our staff.

**Denise Linn Riedl**

[32:39](https://youtu.be/hRdjhYdSmi4?t=1959) Yeah. And kind of following up on that, Santi, like the perceptions of staff, you mentioned staff actually was kind of sold on the RCT almost, because procurement professionals are a very thorough and process-minded group of people. They probably appreciate. How did they feel about, during the actual RCT process, was there any pushback on that operationally in terms of randomly assigning the technology?

**Santiago Garces**

[33:15](https://youtu.be/hRdjhYdSmi4?t=1995) Oh, absolutely. I mean, there's a number of challenges. The first one is, we had to decide who are the people that we're going to invite to do this. And to some extent, there's people who have procurement in their title, but those are the people that regulate and support how we buy things, but they're not necessarily the people that are originating the procurements. And the people that are originating the procurements are many and varied. So we're inviting people from almost every department. And we had to identify. So we ended up deciding that we're going to invite people who had led or participated in a procurement in the past years. So it's like defining our universe of people that we would invite was important. And then the second piece is, there was a lot of ethical and legal questions around, a lot of these people were represented by unions, were in the civil service.

[34:18](https://youtu.be/hRdjhYdSmi4?t=2058) So to some extent, we worked with our people operations team to make it very clear that the results of the evaluation, we were interested in measuring the impact of the technology. We were not interested in measuring the performance of our employees. That was a type of scope, that's not what this was intended to do. So even though from an experimental standpoint, we couldn't tell people, hey, you're invited to participate and you're in the control group or in the treatment group, because that defeats, in order to get a good evaluation, you want to be able to make both groups as similar as possible. So we couldn't fully disclose what was going on, but we did make it clear for people that they could opt out, that their participation or non-participation in the experiment had no bearing into their employment, that none of the results of the experiment were ever going to be shared with a supervisor, that they would be like, we're waiving all of the city's rights to use the results from any management perspective. So working through that gave us the framework around how to do this stuff.

[35:28](https://youtu.be/hRdjhYdSmi4?t=2128) And then the last piece, and it is not a trivial task, imagine putting an event where you're going to invite 300 people to show up to a meeting where they can't entirely know the purpose of the meeting. So we started to figure out, we decided that we were going to support in-person and virtual participation, which also meant that we needed to do the random assignment, considering who are the people that are going to participate, virtual or in-person. It's called stratified design if you're a nerd like me. And then we have to invite these people, invite them to separate groups, to separate rooms, and including separate virtual rooms. So it's like inviting like a surprise party for 300 people, where we were trying to give them enough clue of what they needed to do, where they're going to take this quiz for an hour. And then at the end of the quiz, we would tell them why we did it. And it was basically so that they could help us understand how valuable was using

**Denise Linn Riedl**

[36:29](https://youtu.be/hRdjhYdSmi4?t=2189) this technology or not. Yeah. Oh, interesting. So Santi, you mentioned this whole project started several years ago. The world was so different back then from an AI perspective. The market was really different. I think a lot of folks on this call, if they signed on, they're probably thinking through what a lot of cities are thinking through now. Cities, counties, states, they're thinking through, do we build? Do we buy? Do we hang tight? The market's changing. If you had not undertaken this two years ago, if you were still the CIO for Boston, and you were kind of in the shoes of a lot of the folks on this call, what would you be thinking about that? Would you still build?

**Santiago Garces**

[37:14](https://youtu.be/hRdjhYdSmi4?t=2234) Oh, yeah. I mean, look, I thought, and I felt incredibly lucky being the CIO of Boston at the time, because I think that these are interesting times. They're troublesome times, but they're interesting times. And my sense is the most important thing that we needed to do is to learn. If at the end of the experiment, we realized that AI was useless and it didn't help us with procurement tasks, that'd be a useful thing to know. We would know that we don't want to buy this. We would never want to buy this or build it, or at least in the current... And to some extent, the technology changes and changes somewhat quickly, but maybe once you do an experiment like this, you have a good sense about... It's not like it completely becomes irrelevant from one day to the other. It'll be relevant for at least a couple of years. And we learned so much. Again, we started to see improvements in the quality of the benchmark. So, again, somewhere between the initial idea and then the RCT, we were working iteratively with our procurement team to make the tool better. And we started to see improvements in performance when we improve either the prompt or we improve the context that it referenced, or there was an improvement to the model. We saw hallucinations go way down when we were able to upgrade from Cloud 3.5 to 3.7, I think that it was. Well, we upgraded from 3 to 3.5 and then to 3.7.

[38:49](https://youtu.be/hRdjhYdSmi4?t=2329) We started to build governance frameworks. So, we started to decide who was going to maintain the body of knowledge. Would it be the same body of knowledge that we already gave? So, what I would say is there's no substitute for doing to learn. And I thought that being able to understand how the technology worked and whatnot... And it puts us in a better place. So, to some extent, the question that we have now... The thinking by the time that I left the city is more, we know that the tool that we built is useful. Again, the results of the experiment are not public yet because we're submitting it for a peer review. Again, we wanted the highest rigor. But we're getting this published in an academic journal, peer reviewed by people that know about econometrics that are going to be challenging our regression. So, very high bar. We put ourselves like... But I'll tell you, it worked and it worked fairly well. It's not like a magic thing, but it did improve speed, quality, and the perception of it had meaningful impact for all three of the things that we set out to measure. And then the question that we have to ask now is how do we make available these kinds of tools? What is the level of infrastructure?

[40:14](https://youtu.be/hRdjhYdSmi4?t=2414) I mean, it's mostly social infrastructure. What I would say is generative AI is really difficult to evaluate how well it is. If you just want to say, is this good AI or not? It is feasible to go and say, how good is AI at procurement? And the more specific that you are, the more evaluative that you get. But to some extent, you have to start thinking about what is the capacity of people to be able to manage the context, the governance. The number one thing that BitBot... I mean, it obviously delivers clear answers, but it goes and says, if you have questions or whatnot, you have to go to procurement. Procurement's regulated. You have to go to... One of the things that help us feel better about the risk of BitBot is you can't post an RFP if procurement hasn't reviewed it. It won't post. We won't let you... Well, you can't put it in the ERP. You can't put a purchase requisition unless procurement has seen it. You can't issue an RFP if law hasn't seen it, if it hasn't been publicly posted in at least two publications. To some extent, we started to value the fact that the bureaucracy actually helps... The existing bureaucracy does have a lot of controls.

[41:33](https://youtu.be/hRdjhYdSmi4?t=2493) And all we're trying to do is if we can move quickly to the control, it's not that... The controls become even more valuable. And we have to do something about, can we just deregulate? But more like the regulation helps us find the right policies to assert quality. And could we spend the extra time producing higher quality procurements? It does seem like that.

**Denise Linn Riedl**

[42:00](https://youtu.be/hRdjhYdSmi4?t=2520) That's a great framing, getting faster to the control. Well, BitBot's just a fascinating case, especially since I think you took your fate in your own hands. I think you learned a ton, and you also embrace this high level of rigor, which may not even be necessary for most jurisdictions, but I think is a really interesting thing to learn about. In the spirit of just being whimsical and fun, the next 10 minutes or so, we're going to leave maybe the last five minutes, five to eight minutes for questions. I know there's some questions in the chat. But we thought it would be fun to put Santi in the hot seat. Not that he kind of already is in the hot seat. I'm just the woman behind the black screen. But we wanted to have you play a little game, Santi. And we're going to invite everybody to participate in this game. Santi, you get to make the choices, but everyone gets to have input and chat and via emojis. I'll preface this by saying my main job, besides being head of public sector technology at PPG, is I am a mom of three beautiful children, the oldest of which is obsessed with choose your own adventure novels right now.

[43:08](https://youtu.be/hRdjhYdSmi4?t=2588) So, Santi, you're going to do a choose your own adventure. And we're all going to step into the role of Bob, who is the chief procurement officer of Cloud City, Indiana. So, Bob, Santi, he's eager to make his mark. He was hired to innovate. You know, he's stepping into a job where every department complains about solicitation drafting. Project managers say procurement is slow. Procurement complains about departments. And then vendors complain that RFPs are inconsistent and unclear. So, this is a pretty tricky, pretty tricky situation that we find ourselves in.

**Bonnie McGilpin**

[43:53](https://youtu.be/hRdjhYdSmi4?t=2633) Sorry, Denise, I think we're losing your audio again.

**Denise Linn Riedl**

[44:09](https://youtu.be/hRdjhYdSmi4?t=2649) Everybody's talking about AI. Can we just use it to speed up procurement? Bob sees this as an opportunity himself as one of the most innovative leaders in the city. So, I ask everybody, yeah? No? Can you hear me? Can you hear me? Okay, great. Wonderful. You guys, this is, everyone's a saint for being on this call still. So, basically, Bob sees an opportunity. He wants to be an innovative leader. And he is getting this challenge from leadership. And I'm asking everybody on this call to ponder, like, how should Bob begin? Santi, you get to decide which choice. But I would invite everyone to use your emojis to influence Santi. You know, should Bob research AI tools? What other governments are doing? Should he research the problem more or should he talk to IT?

**Santiago Garces**

[45:08](https://youtu.be/hRdjhYdSmi4?t=2708) There seems to be a lot of hearts, and now there's exploding hearts because there's so many hearts, so I think that the hearts have it, and I would agree with this in the sense like I think that there's stagnation by way of research, but to some extent I think like really understanding what might be the problem, because to some extent it's like it's moving slow might be a symptom and it might not be the disease, so I go with the hearts first. All right, all right,

**Denise Linn Riedl**

[45:42](https://youtu.be/hRdjhYdSmi4?t=2742) you all have done a good job influencing Santi. So we're at problem research. This is the route we've taken. Bob interviews departments, his staff, and vendors. He dives into data, and then he realizes, you know, speed isn't the biggest issue. It's more about inconsistent quality. So his team spends weeks reviewing past procurements that resulted in fewer questions, fewer change orders, and better outcomes. What should Bob do now? Should he partner with IT to prototype something, or should he write a results-driven RFP?

**Santiago Garces**

[46:18](https://youtu.be/hRdjhYdSmi4?t=2778) I think that in this case, if you're trying to use something that you're not entirely sure, like now you have a better sense of the problem, I think that a little bit of experimentation might be in the works, because to some extent we might have a better sense whether we should be investing the time in RFPs and evaluating an RFP takes time. So I would actually, even though there's a lot of hearts, I'll go with the thumbs this time. All right, excellent. All right, so we're going to

**Denise Linn Riedl**

[46:49](https://youtu.be/hRdjhYdSmi4?t=2809) partner with IT to prototype something. Sorry for all of the results-driven RFP fans out there. I'm one of them. But we're going to build a prototype. So Jenna from IT, Bob's new friend, she uses an enterprise AI solution to build a working prototype for Bob and his team by the end of the week. They had a lot of knowledge to leverage for this build. The tool helps draft solicitations and translate department requirements, and then it also helps train new or less experienced staff. So early feedback looks very encouraging for the team. What should Bob do now? Should he collect baseline data and define success metrics, or should he move quickly and start a wider pilot

**Santiago Garces**

[47:30](https://youtu.be/hRdjhYdSmi4?t=2850) immediately? Oh, we have so many thumbs. I think that we'll go with the thumbs. I think to some extent because collecting the baseline data will help us better understand our ability to evaluate the tool and then also get a baseline sense of, even if we decide to go to market to buy something, get a better sense of what's actually technologically feasible or not. So I think that it'll give us a better scoped process. But I go with the thumbs. All right, thumbs have it.

**Denise Linn Riedl**

[48:03](https://youtu.be/hRdjhYdSmi4?t=2883) We do baseline first. So Bob collects baseline data. The mayor's office agrees on success measures before launch. Half of the team uses the prototype and half continues using existing processes. A year later, the results are undeniable. There's time savings, fewer change orders, and vendor questions, great feedback from vendors and departments. Let's see how this all ends. So Bob is the evidence builder. The pilot succeeded because the city knew exactly what it was trying to gain with AI. Bob is so proud, Innovate US invites him to lead a workshop, Santi. Not unlike the fate that you're in now. Would you like to play again and make different choices?

**Santiago Garces**

[48:46](https://youtu.be/hRdjhYdSmi4?t=2926) I will defer to other folks to play again. Great. I like our collective choices or at least my editorialized version of our

**Denise Linn Riedl**

[48:59](https://youtu.be/hRdjhYdSmi4?t=2939) of our collective intelligence. Absolutely. Yeah. No, and thanks everyone for doing that. And the point of this exercise wasn't necessarily that they're actually several happy outcomes that Bob can take. But honestly, I think that the kind of trade-offs and decisions you make in these build versus buy decisions, as well as in the kind of evaluation decisions with these types of implementations, you can make good decisions with bad outcomes. You can make bad decisions with good outcomes. But thanks for playing along, Santi. It was fun. Should we move

**Bonnie McGilpin**

[49:38](https://youtu.be/hRdjhYdSmi4?t=2978) to our questions from the audience now? Because we have a lot of great ones in the Q&A. Absolutely. Why don't I'm happy to read them because I know Denise is having a few internet issues. But our first one is from Brian. As Denise said, the technology is changing all the time. AI native tools get bought out by bigger companies. ERPs are adding AI to existing tools. Can you give advice on how to manage the rapid change and pick a tool or strategy

**Santiago Garces**

[50:07](https://youtu.be/hRdjhYdSmi4?t=3007) that has some longer term sustainability? I think my advice would be there's no way about it, but you have to embrace kind of a learning attitude. You have to be willing. And to some extent, it's like having a good understanding of what are the problems. That's why I like following Bob's path is like having a good sense of what are the problems that are worth that really, really matter so that you could be keeping an eye out. It's exciting to have a market that's very dynamic because it might be that there is more alternatives to solve the problem that you have and that you might want to evaluate what is out there more frequently. So I would say like having a better, like a strong sense of what are problems that are worth prioritizing, continue to go and look. And I would go and say just because things move forward doesn't mean that there's learnings that don't age well. Again, because there's types of things that AI can do or not. And for instance, like this retrieve augmented generation, this pattern is one that is useful and you can get, it can get better. But most of the times, like what happens is that AI, there's a thing that AI was not very good at. And then all of a sudden it becomes good enough at it.

[51:22](https://youtu.be/hRdjhYdSmi4?t=3082) But often if we've found that there's a thing that is useful, it just keeps getting better. But it's like rarely does it go that it was doing a thing well and then it stops doing it well.

**Bonnie McGilpin**

[51:34](https://youtu.be/hRdjhYdSmi4?t=3094) Yeah. And you have lots of questions about Bitbot. Gideon asked, are the skills you wrote

**Santiago Garces**

[51:39](https://youtu.be/hRdjhYdSmi4?t=3099) for the Bitbot tool publicly available? We intended to. I don't know if my colleagues in Boston did yet, but for whatever it's worth, there's, Bitbot has a cousin called Abe. Just joking, but the AI for Impact team at the Berne Center have a publicly available tool called Abe, which is the assisted virus engine, which Massachusetts used. So you could go and find a thing that works very similar to Bitbot and we can ask our friends at Boston to release

**Bonnie McGilpin**

[52:17](https://youtu.be/hRdjhYdSmi4?t=3137) Bitbot into the world. Yeah. And we can share more about Abe in our follow-up email too, that we send to all participants. Andrew asked, what sort of data did you have to connect to for Bitbot? And assume you had all these aspects already in place for this to happen. So for

**Santiago Garces**

[52:40](https://youtu.be/hRdjhYdSmi4?t=3160) whatever it's worth on that one, we were like, one of the things that we wanted to learn was, people are going to say like, this will work, but if you connect it to all these things, we're like, let's be realistic. So Bitbot, what is the most diplomatic way of saying, Bitbot was built in the most realistic slash resource economic way of doing it. So we kind of just pointed it to a body of documents that will constitute like what we usually give employees as guidance for training. And because we thought that it would be fair, we basically, the knowledge source is the same materials that we give to people in this like finance academy. They're all publicly available documents, by the way. So, and there are just like too many documents for a person to read. Although we tell everybody that they should read and familiarize themselves with those documents. So that's kind of, but there's no like ERP data, there's nothing.

[53:44](https://youtu.be/hRdjhYdSmi4?t=3224) And to some extent, partially like what the learning with Bitbot was, is how do we make sure that as we make changes to that body of knowledge, because the state contracts get updated, like, you know, these master contracting vehicles change. So to some extent, like what we were trying to learn is what is our capacity to maintain that body of knowledge, which is truly like, regardless of what tool we were using, whether we were buying Chajupati or Claude licenses or whatever, that's a thing that's still like work that a person needs to do or supervise.

**Bonnie McGilpin**

[54:17](https://youtu.be/hRdjhYdSmi4?t=3257) So here's a good question from Brendan. Can you speak to the collaboration between academia and the city of Boston and make a recommendation for other cities who'd like to do the same thing?

**Santiago Garces**

[54:31](https://youtu.be/hRdjhYdSmi4?t=3271) I mean, we're incredibly lucky that in Boston, and now that I work at an academic institution, you know, like Boston has a major university by neighborhood, basically, sometimes multiple. So it's amazing. So we're very lucky that we got the support from Northeastern to develop Bitbot. And then we got a lot of support from Harvard to do the evaluation. So we worked with folks from the government performance lab, and then also with the Partnership for Public Good to build the instrument. So we wanted that when we build the, you know, like when we are starting to build this quiz that we're going to give people like this test, evaluate, you know, like, that it was meaningful that it would actually test things that were valid around procurement. And the other thing, when we actually went to do the evaluation, we wanted to have two blind evaluators. So like, have two evaluators that didn't know whether the answer was made with AI or not, because we wanted to see if they agreed or not. So we needed external evaluators. So Harvard and the partnership were like, essential, we wouldn't have been able to get to the level of rigor that we wanted without their help. And then afterwards, like with the data analysis and all this stuff, they're poking holes and helping us write the results. So I think there's amazing academics everywhere. And to some extent, I also know that like, you know, like resources like the folks from Northeastern, and now that I would be remiss, now that I am in my new lab for the AI lab for cities, that's our purpose to work with cities and governments everywhere to try to help them navigate this, because it's tricky, and it takes some extra resources. We're hoping that the lab becomes the kind of help that we wish we had when we set out to do this, so that it's easier to do the next 15 bit bots.

**Bonnie McGilpin**

[56:22](https://youtu.be/hRdjhYdSmi4?t=3382) Great. Okay, here's a question from Amy, considering the direction that AI is rapidly going, do you still believe that AI use in the public sector is worth the risk? I'm thinking AI worms, rogue agents, etc.

**Santiago Garces**

[56:38](https://youtu.be/hRdjhYdSmi4?t=3398) Absolutely. And to some extent, so I'll elaborate. So I think, look, a lot of the risks that exist around like, especially the ones that you mentioned, are risks that existed there, like the security vulnerabilities that AI tools are exposing, or security vulnerabilities that existed in the code, they're not AI security vulnerabilities, they're just security vulnerabilities. And to some extent, like AI is really, really good at reading and producing computer code. So it is really good at exploiting, finding those vulnerabilities and exploiting them. To some extent, my hypothesis is that this is one where it will be very unlikely that we're able to keep up in our ability to manage and support our digital infrastructure without AI. And I just, there, like, there's no rolling it back. It's just like, it's part of the reality. So I think, like, to some extent, like what we need to do is understand, when is it helpful? How can we make sure that it is safe, like it is used safely? I think to some extent, like there are a lot of governance questions around like, to what level should things be automated or not be automated? I think that those are really great questions. And maybe different communities will come up with different answers.

[58:00](https://youtu.be/hRdjhYdSmi4?t=3480) But I think that being able to do it in a way that is transparent, that is traceable, that we can, and where we can change our minds, like maybe we say, like, no, we don't want to do this at this speed. But later, we want to revisit that. I think that that's okay. And I think that you can only do that if you have a willingness to learn. But I think that there's no putting the genie back in the bottle. And the other piece is like, AI can be incredibly helpful. When done right, like we see, I think it's a little bit over, like, it's about half of the people that use BitBud, continue to use BitBud regularly, that like, if you look at AI, like AI usage, and in most tools, that's actually fairly high. And most people, remember, don't buy things all the time. So there's good reasons why they might not be using BitBud, because they're actually not doing any procurement these days. So I'd say, like, my, the general sense is that when we build AI in a way that is responsible and useful, there's a lot of good reason why we should be using it.

**Bonnie McGilpin**

[59:07](https://youtu.be/hRdjhYdSmi4?t=3547) I think that's a great place to end. Thank you so much, Santi, for taking the time to share all of your work and talk to everyone today. Thank you so much, Denise, for moderating. I'm sorry that there were some technology issues. But we had, that was a great session. Thank you so much for everyone who participated, to our partners at NASPO and the Partners for Public Good who have helped us so much with this series. You will be receiving a follow-up email with more resources from Christine after the session. And please take the post-workshop survey. It'll pop up on your screen when the session ends, and it helps us inform future sessions. And Christine will share more info in the chat about upcoming workshops. Please join us for another workshop. And with that, I will end this session. Thanks so much to everyone. Bye-bye.

* * *

Original page: https://innovate-us.org/ai-tools-showcase-whats-out-there-and-how-to-evaluate-it · Licence: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org
