---
id: "workshop:ai-across-the-procurement-lifecycle-where-does-it-actually-fit"
type: "InnovateUS workshop"
title: "AI Across the Procurement Lifecycle — Where Does It Actually Fit?"
url: "https://innovate-us.org/ai-across-the-procurement-lifecycle-where-does-it-actually-fit"
date: "2026-10-05"
duration: "60 minutes"
series: "AI for Public-Sector Procurement"
instructors: ["Angela Shell"]
moderators: ["Emily Royall"]
recording: "https://www.youtube.com/watch?v=2R9_YAW3Cx0"
transcript: "automatic, from the recording"
source: "InnovateUS (innovate-us.org)"
licence: "CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org"
---

# AI Across the Procurement Lifecycle — Where Does It Actually Fit?

InnovateUS workshop · 5 October 2026 · 60 minutes

Explore where AI tools fit across the procurement lifecycle, whether planning, solicitation, evaluation, and contract management. See a large-state perspective on integrating AI into a high-volume procurement operation, including where it adds the most value and where human judgement still leads.

## Learning goals

- Map AI tools and use cases to specific stages of the procurement lifecycle

- Distinguish between tasks well-suited to AI support and those that require continued human judgement

- Evaluate a demonstrated AI tool for potential fit within your own procurement workflow

## Speakers

- **Angela Shell**, Chief Procurement Officer, State of California

## Moderators

- **Emily Royall**, Assistant Director, Partners for Public Good

Part of the series [AI for Public-Sector Procurement](https://innovate-us.org/workshop-series/ai-for-public-sector-procurement-fall-2026).

Recording: https://www.youtube.com/watch?v=2R9_YAW3Cx0

## Transcript

Automatic transcript of the recording: names and terms can be misheard, and the times can be a few seconds off. Each time links to that moment of the video. The transcript alone: https://innovate-us.org/ai-across-the-procurement-lifecycle-where-does-it-actually-fit.txt (plain text) and https://innovate-us.org/ai-across-the-procurement-lifecycle-where-does-it-actually-fit.json (every segment with its start and end second).

**Audience**

[0:00](https://youtu.be/2R9_YAW3Cx0?t=0) And welcome back to the Innovate US webinar series, workshop series, AI for Public Sector Procurement, co-hosted by NASPO and Partners for Public Good. It's great to see several of you back. I know I'm seeing some names I saw in the first one we did last week. So this series will equip procurement and other public sector professionals with the practical skills to leverage generative AI to improve competitiveness, effectiveness, and efficiency in procurement. Designed for hands-on learning, each session builds on the last to give participants actionable tools they can apply immediately to serve the public better. In today's session, AI Across the Procurement Lifecycle — Where Does It Actually Fit?, we're going to feature a large state perspective on integrating AI into a high-volume procurement operation, including where it adds the most value and where human judgment still leads. So with that, I'll turn it over to Emily Royal from Partners for Public Good, who is moderating our session. Emily.

**Emily Royall**

[1:02](https://youtu.be/2R9_YAW3Cx0?t=62) Fantastic. Thank you, Jordan. Just want to thank NASPO and Innovate US for partnering with us and supporting this amazing learning series. Good afternoon, everyone. My name is Emily Royal, and I'm the Assistant Director overseeing the Procurement Excellence Network, which is an initiative of Partners for Public Good. The Procurement Excellence Network, if you haven't heard of us before, is a virtual community of what is now over 3,200 public servants that are all advancing procurement reform. And for our members, AI is increasingly an important topic of discussion, especially when it comes to building the future of procurement. And it's raised a lot of important questions about what it means to build a procurement function that is results-driven, is strategic, is fair, is transparent in the digital era with new tools like artificial intelligence. So our use of AI in procurement has introduced a whole new set of challenges and questions across our membership. And we're hearing questions come up on a regular basis, like how do we ensure that our use of AI is transparent? How do we manage vendors' use of AI in response to our solicitations? Is AI safe to integrate into our operations? Which specific tasks is it safe to integrate? How can we feel confident that at the end of the day, humans are ultimately making the decisions about our solicitations and our awards, rather than turning that over to an automated decision-making process?

[2:35](https://youtu.be/2R9_YAW3Cx0?t=155) This is an evolving space. We may not have all the answers for you today, but we do have the benefit of learning from those who are pioneering this work in the field. And to that end, I am joined by Angela Schell, the California Chief Procurement Officer and Deputy Director of the Procurement Division at the California Department of General Services. And she's going to help us explore a little bit where AI tools fit in the procurement lifecycle, sharing with us some real case studies, some real examples from her own work at the State of California, and really share with us what it looks like when a large, high-volume procurement operation is finding value using AI tools while still maintaining transparency and keeping human judgment front and center in the process. So, join me in welcoming Amanda Schell, I'm sorry, Angela Schell. We're so excited to have you. Thank you, Angela. I'll turn it over to you. Thanks, Emily. And thank you, everybody, for

**Angela Shell**

[3:38](https://youtu.be/2R9_YAW3Cx0?t=218) joining today. I'm super excited to talk a little bit about what we're doing in California. I think, you know, we're hopefully doing some somewhat unique information that we'll all be sharing today. But also, because we're California, we also want to be a little bit cautionary at the same time. So, we chose today to just share a little bit of what we're doing in the AI space around our market research. So, with that, if you can go to the next slide, please.

**Bonnie McGilpin**

[4:16](https://youtu.be/2R9_YAW3Cx0?t=256) Sorry, we're just stuck for a second here.

**Angela Shell**

[4:25](https://youtu.be/2R9_YAW3Cx0?t=265) Technology. It's always our friend, right? There you go. All right. So, this is just about me. Long experience with the state of California, lots of time in procurement. This is a younger version of me. I think I'll keep that picture forever. And then also just talking a little bit about, you know, some of the organizations that we participate in, the National Association of State Procurement Officials. California is on the board California is on the board of directors and has been for a few years now. We also use a lot of the NASPO value point contracts as part of our contracting program in the state of California. And there were also members of the Sustainable Purchasing Leadership Council, also on the board of directors there for many years in California. And what that does is it's a national organization similar to NASPO, but SPLC focuses on sustainability in procurement

**Bonnie McGilpin**

[5:20](https://youtu.be/2R9_YAW3Cx0?t=320) operations. All right. Next slide, please. Sorry, we're having a little issue, but let me

**Angela Shell**

[5:32](https://youtu.be/2R9_YAW3Cx0?t=332) bring it right back up. All right. There we go. Okay. And thank you for sharing the slides, Bonnie. I appreciate that. Okay. So, a little bit about our case study in California. So, we participate in strategic sourcing efforts when we're looking at pricing on what we call leveraged procurement agreements. These are overarching agreements that our state departments procure from in the state. So, we're a little bit of a centralized, we're also decentralized state procurement office. We have 150 plus state departments that we delegate purchasing authority to, and we issue several agreements that these state departments can purchase from in an expedited purchasing fashion so that they don't have to do a full competitive procurement for everything they might buy under the sun. So, California is very focused on efficiency improvements, as I'm sure a lot of other states are, local entities. I'm sure we're all focused on efficiency in our procurement operations. And one of the areas that we started looking at in the last couple of years was really around our pricing in our state leveraged procurement agreements. And part of that was looking at our market research and what we're doing to ensure that we have competitive pricing in the state. And so, part of our market research is around price benchmarking and then vendor reporting. So, we require our vendors to report on purchases off of our leveraged procurement agreements, both at the state level, and then we also allow local government entities to purchase off of our contracts as well.

[7:17](https://youtu.be/2R9_YAW3Cx0?t=437) So, vendors have to report on the contracts that they issue out to our local government entities and, again, our state departments. So, we started looking at, you know, what could we do with our market research? How could we be faster at our market research? And then also, we began to receive a lot of increased price requests. Now, some of it, you know, due to the idea of tariffs, but also just from an economic perspective, we allow price increases on our state leveraged procurement agreements. And so, we were getting a big influx and we needed to know that, you know, these price increases that were coming in, are they really substantiated in terms of what the market would tell us? So, we started looking at our AI tools and started using ChatGPT and Copilot to help us look at price comparisons in a much faster way. So, you know, we're looking at things like our very specific contract requirements in California and we wanted to look at how those costs stack up against other states that are of our similar size, our similar volume, perhaps similar vendor pool. And we wanted to use that, which we've been doing in our strategic sourcing effort for many years. We wanted to look at being able to quickly apply the PPI changes that we know are in the past, but that do give a good measure against other markets, both retail and government comparisons. And so, again, we brought in ChatGPT to help us pull information from the market and summarize it for us. Also, using ChatGPT and Copilot to help us classify our data in our reporting, because as you can imagine, in a somewhat manual fashion, you get all kinds of information that comes in, right? You get garbage data in, which results in garbage data out. So, we used that to be able to classify our data that was

[9:20](https://youtu.be/2R9_YAW3Cx0?t=560) coming in in a more uniform fashion. And then, again, summarize and give us some data points. So, in terms of sort of the scale of this market research, we have over 4,000 leveraged procurement agreements and we have to analyze those at least once per year, if not more often, as price increases come in. Some of my food contracts, it might be every three months, might be every six months, it could be, you know, in a year. Again, approximately 150 users. And we really had to take a look at, you know, where are we at in our pricing? And we have various programs that we use in the state with these leveraged procurement agreements. Some are fully competitive bids that have a single pricing. It is what it is. Some of them are things like, we piggyback off of GSA pricing and allow our vendors to come in. They enter into a contract with the state, but they can use GSA schedules. We also have software licensing agreements and then a large-scale program for master service agreements that some of these contracts have over 350 vendors just in that space for our, you know, things like unarmed security guards, things like that. So, we needed to make sure that, you know, these contracts are still getting good pricing given where the market is in all of these areas. So, back in July, this was the last time we updated this slide deck, we were showing that on roughly 75 or so contracts, mostly my goods contracts, we could show that we had approximately 92 million savings just in that benchmarking against other entities. And so, that means that the state is, by the pricing that we're getting, we are saving our state departments $92 million in those buys. And we had over 900 contract line items that we're evaluating. Again, we have thousands upon thousands of line items. And the contract spend on these contracts, it varies from 100,000 up to 76 million on any of these given contracts. So, our desired

[11:24](https://youtu.be/2R9_YAW3Cx0?t=684) outcome really, and I've kind of touched on this, but it was to really benchmark our pricing against similar markets and determine what this cost savings is, and then evaluate the need. This was a key area for us is evaluate the need for rebidding or renegotiating costs. California does use So, we really needed to be able to see, should we rebid this contract? Are we getting the best pricing we could be? And we did find that in some areas, we're achieving no savings, because it's a very specific brand for a very specific department, or it's the way in which the industry is right now. And so, oh, sorry, it sounds like I cut off for two minutes.

**Bonnie McGilpin**

[12:10](https://youtu.be/2R9_YAW3Cx0?t=730) It was only about 10 seconds. It was very, very brief. So, I think you could keep going, Angela.

**Angela Shell**

[12:15](https://youtu.be/2R9_YAW3Cx0?t=735) Okay. All right. So, let's go to my next slide then. All right. So, what were some of the challenges that we encountered? So, kind of why we went into this project was really because, again, our strategic sourcing efforts were taking up to 180 days just to really get in. And this was on price increases, things like looking at the next time we go out for bid for a similar contract. I mean, it was taking us sometimes up to a year. And so, our pricing was not useful to our contract analysis or our awards. Again, those increase requests, our extensions, our rebids, and the new requests that come in from our state departments. All of our data is publicly posted. Anytime a buy is made in the state of California. So, we have robust data sets to be able to see what the state is, in fact, buying. And we do get a lot of new requests. We also analyze that data to see, should this be a statewide leverage procurement agreement or should we go back and let departments do independent agreements that vary across every department? So, the research time alone was taking up to 150 days. Again, internet searching, separate pages each time, our contract reporting, past contracts, all of that was taking an extensive amount of time. So, through this AI tool that we brought in, and I'll touch on a second phase of this a little bit later in the presentation, but by bringing in ChatGPT and bringing in Copilot to help us synthesize our data in a much better way. Could you go back one slide? There we go. Oh, I'm sorry. Go back.

[13:51](https://youtu.be/2R9_YAW3Cx0?t=831) There you go. All right. So, we were able to reduce our days for the internet research down from 150 days down to seven days just by compiling the information, right? And this is a pretty simple use of AI. It really is. It's compiling data. It's getting information. The team was developing their prompts, right? And because they're using ChatGPT, they were developing their prompts. They were having to refine their prompts. All of this, allowing this type of a reduction, our total cycle time. So, this is looking at our usage, profiling, you know, where it's at, cost analysis, our benchmarking against other areas, and then our market analysis down to 30 days. Capturing our data from our vendor reports for that spend analysis in a way that we hadn't done at all. And then reducing our contract costs for our rebids. In some cases, we actually saw a 75% savings in going out for a rebid because we saw that our market price, it was too high. What we were paying was too high. We hadn't done an analysis in a way that That was this much detail, and we were able to show, especially on some of our food contracts, that we got a 75% savings by going back out to bid.

[15:10](https://youtu.be/2R9_YAW3Cx0?t=910) All right, next slide. Okay, so again, it's an AI market research tool. It's something that we're really focusing on that we're continuing to do. And this really kind of helps if you're looking at how you want to be able to justify use of AI, especially in this market research. These are the areas that we really looked at. So the AI usage strengthened our reviews of our statewide contracts and our leveraged procurement agreements. Again, it gave us a structured and uniform process. We had that in strategic sourcing. The manual I saw for strategic sourcing was like, I don't know, 15 pages long. And I was like, no wonder it's taking us this amount of time, right? So we didn't veer from our strategic sourcing, but we used these AI tools to get us a faster way to do this, to give us a consistent way to break down each portfolio on our contract. Again, we were able to reclassify that data in a uniform way. So it's an apples-to-apples comparison because, especially on the state side in government, oftentimes if you have a statewide contract, it's spec'd out.

[16:15](https://youtu.be/2R9_YAW3Cx0?t=975) And when I say spec'd out, I mean it's got specific specifications for what we want to buy and why we want to buy it and how it's used. In California, I'm not buying office supplies. I'm not buying a box of pencils. I am buying paper because I want it to be recycled content in California, but I'm not buying things like notepads and that, right? Because those don't require a specification. And so these are really things that we focus on. We wanted market alignment. We wanted to make sure that we knew that our pricing was good in the state of California, or it's not so good and we need to do something else going forward. Instead of just kind of lagging behind and hoping that in any given day in a competitive bid, you have a minimum of three bidders. You hope that those three competing against each other is the best price in the market. Maybe it isn't. And so this helped us actually see what's happening in the market in a much faster fashion. And again, bringing in that PPI. Again, the prompt development. It got the team practicing. How do you use prompts in an LLM, in that sort of scenario, and working on that skill to be able to keep honing that in.

[17:24](https://youtu.be/2R9_YAW3Cx0?t=1044) Again, pricing insight. So it gave us a clear view of what we're actually doing and where we could have cost reduction. Most states, you're probably looking every year at your budget and saying, I don't know if I can afford to buy these goods or these services again. I'm not sure it's in my budget. This allowed us to be able to say, no, no, we're reducing that pricing or we have saved you this amount. And especially going forward. And then again, I touched on it already, really benchmarking our findings across technology, fleet, food, our printers, multifunction devices, tires, ammunition, EVSEs. Like we did all of this. Our land survey products, I will tell you that was the one area where we fell short. It's a very specific brand for a very specific department. And so but what this did was show us that, wait, OK, we're not showing any cost savings. In fact, we're probably this is probably costing us more. But it was able to tell us we could take a step back and bring in other vendors with other tools that perhaps state departments could use.

[18:29](https://youtu.be/2R9_YAW3Cx0?t=1109) Perhaps not the one specific state department, but other state departments that need land survey products. Perhaps they could use these and get better pricing versus the one vendor and the one brand we had on contract. All right. Next slide. All right. So so these are some of the procurement strategies and the evaluation criteria that we used in the state. And so in California, as I said, we have a very broad negotiation statute that allows us to do negotiation. We don't have to do fully competitive bid. We might post it publicly. We might accept bids competitively, but then we can turn that around and do a negotiation to get a best and final offer at the end of the day. Again, these are for our leverage procurement agreements. And then once we have an overarching agreement in place, because these are zero dollar agreements, my state departments are buying from them. We allow them to do a simplified process where they get to do a request for an offer so they can request it from any number of vendors on the contract.

[19:32](https://youtu.be/2R9_YAW3Cx0?t=1172) Give them their scope. Get a quote, whatever it is. We do require when we have resellers on contract, which is most of our contracts, especially in the I.T. space. We ask our departments to do reseller quotes and they can evaluate it by lowest cost or best value. We ask all of our state buyers to incorporate our state terms and conditions for AI, Gen AI use. So we revamped those back in twenty twenty four and then twenty twenty five on the Gen AI side to make sure that we had very clear guidance on data protection, data security, ownership of data. When using an LLM where you're you know, you're creating prompts, those sorts of things. We had infrastructure standards and guidelines where, you know, again, this goes into our AI, Gen AI terms and conditions where we talk about FedRAMP certification, mod high, those sorts of things.

[20:38](https://youtu.be/2R9_YAW3Cx0?t=1238) And then all of the outputs are owned by the state when we're using especially an LLM, not just an AI tool, but an LLM where you're creating something new. And then we have software as a service and we have enterprise subscription solutions that we also put in place that sort of incorporates all of these things. Also, and this is something I didn't necessarily touch on, but, you know, also something that we were really big on with our state departments and especially in this particular market research is that you have to have a human in the loop. So while everybody's excited, they get to actually officially use chat GPT in state operations, because I'm sure you all know, you know, things like that were banned for quite a while. They still have to have the human in the loop. They had to identify that they had used AI or Gen AI and the human in the loop absolutely has to go back because we're talking about cost savings here.

[21:38](https://youtu.be/2R9_YAW3Cx0?t=1298) They absolutely had to go back and be able to see that this was in fact something that is real. It's not, you know, it's not something and I see the question about bias. Our terms and conditions require our vendors to ensure that they don't have bias, they don't have discriminatory, you know, operations in their LLMs. But again, it's that human in the loop going back and fact checking, you know, is this really true? You know, let's get copies of contracts. Let's say it's something where the AI tool comes back and says, oh, yeah, the state of California and in fact, we had this in a couple of instances, the state of California is getting 110% better pricing. Wait, what? Okay, so we had to go back and actually check that to see were the calculations correct for our benchmarking? You know, was it a similar contract? Could we get a copy of the contract from that particular state or entity that we were looking at? So, you know, we did the best that we could while we were working through that, but keeping that human in the loop. Also, on contract management, you know, ensuring this is an overall item for California, not just what my office is doing around market research.

[22:52](https://youtu.be/2R9_YAW3Cx0?t=1372) So, contract management is a significant element of the use of AI and Gen AI. And they have to, all state departments have to have contract management practices that ensure state staff are properly managing prompt responses. That if somebody begins to use AI or generative AI specifically in a contract that wasn't contemplated when the contract was awarded, that the state is reviewing that tool before they allow it to be fully implemented into whatever it is they've purchased. And also state departments are required now to have a Gen AI liaison in state departments that's aware of the AI, Gen AI tools that are being used by that state department. And they're sort of the subject matter expert, again, on the back end around making sure that these tools are not doing things that they shouldn't be, like introducing bias, making discriminatory decisions. So, the state has put that in as a standard practice and policy for all of our state departments. The other thing that we did, and this was a really great tool that still exists today, but we did some testing of it. And this was our poppy. It was our California Digital Assistance Testing Project, which is now rolled out, in fact, to be a formal program that's offered through our California Department of Technology.

[24:15](https://youtu.be/2R9_YAW3Cx0?t=1455) But what this did is it allowed state departments, I think there was 100 of us at each state department, they got the opportunity to review Google Gemini, Quad, Azure, ChatGPT, AWS, all of these programs that were out back in the 2025 timeframe. I realize they've changed, things have happened, but we got the ability to test it inside of state systems. So, you couldn't go outside of our state websites, but you could test our internal state websites to see how these tools operate for your use case. So, it was a way for all of us to do use cases. And we chose our state contracting manual out of the Department of General Services. And what we found was we've got some improvements to make. So, we were able to use these tools, the different tools to ask our state contracting manual questions. And by and large, they came back wrong. Some of it is because our content is hard to decipher. But it was a really good opportunity for us to test these tools and to be able to see which one fits our best use case.

[25:19](https://youtu.be/2R9_YAW3Cx0?t=1519) And I would encourage everybody, if you can find a way to test these solutions in a secure closed environment, this is one of the best ways to not get yourselves in trouble. So, we compared our copilot to ChatGPT while we were working through this market research and found that ChatGPT really was a better tool for what we were asking it to do. In the Poppy, our resellers agreed to the terms and conditions that we had. Poppy made it sort of a low risk idea for us. And then again, in California, we have a very robust requirement to classify our data before you buy AI or generative AI, especially. And so, we have a form, a 5305F form, where the CIO, sometimes the agency information officers in state departments go in and they have to evaluate what is the use case? What are you trying to buy? What do you want to use it for? What data do you have? Do you have HIPAA data? Do you just have public data? Like in our case, the data that we're using is public data.

[26:27](https://youtu.be/2R9_YAW3Cx0?t=1587) And so, you have to go through that. You have to confer with the Department of Technology. It's low, moderate, or high risk. And you go through all of that before you can buy. All right. Next slide, please. I realize I'm running up a little bit against some time here. So, all right. So, these were some of the lessons learned and our recommendations that we had going through this process and really looking at how do we do this in the future. So, the key takeaway, the first one would really be to test solutions before you buy them. So, again, this goes back to that poppy tool. And as I said, we're still using it in California today. It's become a standard practice where departments can still go in. They can use any one of those tools to test to find out which one works best for them before they go implement a fully baked system. And again, we had some success with this. For takeaway number two for us, so what does success look like for us and what comes next? Next, we are finding that our market research has been very successful for us. Not to say it hasn't been without challenges because anytime you're using AI or Gen AI, you are going to have some level of challenges.

[27:38](https://youtu.be/2R9_YAW3Cx0?t=1658) But we have been actively using this for approximately a year. It's worked great for us. But we're on to version two already. So, my team internal to my central procurement office is using the tool for our leveraged procurement agreements. But what we are working on now is a new tool that will be rolled out to our buyers next month. And what this tool will do is it will actually use AI to compare my contract pricing, my small business disabled veteran business enterprise contract pricing, any non-leveraged procurement, so open competition pricing with that current moment in time market pricing for what you're trying to buy. This will be our first version of it that's coming out. It's based on our historical data. Remember, I said all of my data is tracked in a public database. So, we'll go back one year to be able to compare pricing so that every state buyer can have access to know, are you getting a good pricing for what you're trying to buy? And so, apologies, I had to turn that off.

[28:44](https://youtu.be/2R9_YAW3Cx0?t=1724) And so, that's version two. We're actually using a Google solution this time around. And we're very excited to be able to roll this out this year as well. All right. Key takeaway three. Oops. We lost the slide. So, let me see if I can just touch on it. So, oh, there we go. Thanks, you guys. Okay. So, just quickly for these last two, ensure that your contract terms address data security, the use of AI, Gen AI, and those disclosure requirements. So, while we have had some pushback from our industry partners, especially the big ones where they have their own language around these types of things, by and large, it's been pretty good with our vendor pool accepting the fact that we have data security provisions. They're robust. Our Gen AI, which talks about the use of LLMs and that data ownership, and we classify our data right in our model terms. And so, if you don't have that, I would encourage you to get that because it really is the basis for where you're going. And then, really, the last one is ensuring your buyers know how to buy AI and generative AI.

[29:54](https://youtu.be/2R9_YAW3Cx0?t=1794) We went through, you know, we had an executive order at the end of 2018. And that was one of the key takeaways for us was putting together buyers' guides on how to buy our AI and our generative AI, what are the minimum things you need to do, the disclosure requirements from the vendors, making sure the terms and conditions are in there, you know, again, that 5305-F form, that analysis of your data, all of those things had to be done. And we required every one of our buyers to take training, a very short 15-minute or less training on all of these things that we added in. And we added it into our state contracting manual as well. Because you want people to open up that communication, your buyers especially, with those CIOs in those departments. It should be a team buy. It should never be a buyer just saying, OK, someone's buying AI, let me work through this. It really has to be a good team communication with your CIO so you as an entity know exactly what you're buying.

[30:56](https://youtu.be/2R9_YAW3Cx0?t=1856) All right, I think we have a couple minutes left. So I think that's the end of the presentation. I know I went through that really quickly. And I do talk fast. So I'm happy to share anything, you know, with anybody that's interested. If you want more information, we have a lot of it online. But again, I'm happy to share. All right, Emily, back to you.

**Emily Royall**

[31:16](https://youtu.be/2R9_YAW3Cx0?t=1876) Fantastic. Thank you so much, Angela. This was like really detailed. And I'm going to spend maybe the next few minutes unpacking some of this with you a little bit more for our audience. I'm seeing all the amazing applause and everything on my screen, which is so great. OK, so in short, maybe just to capture like everything that you said. You identified really like three main ways that AI tools can impact the procurement process. You talked about cost savings, primarily through benchmarking, cost analysis, looking at what other states are actually buying that are comparable in terms of your size and purchasing power, marketing, market alignment in general for the use case of the product. The cost savings piece is definitely there. You talked about time savings. You were able to reduce some of your cycle times, for example. You were also able to like reduce market research process from 150 days to seven, which is just like a radical efficiency point.

[32:16](https://youtu.be/2R9_YAW3Cx0?t=1936) That's super awesome. And you talked about process savings. So faster strategic sourcing, being able to reclassify your data in a more uniform way. And all of these things, cost, time, process, it makes you smarter, faster and more efficient as a procurement division within your agency. And at the same time, you also highlighted some of the important guardrails that need to be in place, right? It's not all sunshine and roses. This is built off of a very structured and intentional and thoughtful effort that includes things like incorporating human in the loop at key decision making points in the procurement workflow. Centralized policies, for example, where you are, you know, complying with AI policy that impacts the entire state as an entity. And maybe you have your own sort of internal policies around AI usage that you're dealing with internally as well. Leveraging your contract management as a space to set important requirements for things like ensuring that staff is reviewing the use of AI tools, making sure that your T's and C's are reflecting best practices for Gen AI use.

[33:27](https://youtu.be/2R9_YAW3Cx0?t=2007) And then you also talked about this really cool idea of Gen AI liaisons or AI liaisons who act as subject matter experts and SMEs are kind of being integrated into a more larger public oversight mechanism for the state. And then last, you kind of helped people think about how do we get there from here? You talked about the importance of piloting and testing, evaluating the efficacy of different models against procurement tasks and doing that in a protected sandbox environment. And then you also reflected a little bit on how, you know, when you're testing these tools, it made you reflect quite a bit on the quality of your data, because the quality of their output depends on the quality of your data, how it's structured, how machine readable it is. You know, is it like a billion PDFs with random comments in it or, right? Is it structured data that a machine can actually use and learn from to produce the accurate approach? And sometimes you find conflicting information in your policies, right? And so the AI output is going to reflect that.

[34:34](https://youtu.be/2R9_YAW3Cx0?t=2074) So all of this, you know, taking all of this into account, let's start maybe by talking about what is your advice for how an agency should get started? You know, you've shown us this whole arc, like, how do people start? Where do they begin if they want to integrate AI into procurement processes in their agency?

**Angela Shell**

[34:59](https://youtu.be/2R9_YAW3Cx0?t=2099) Yeah, I think that's a great question, Emily, because, you know, although we're using it in our market research, and it's huge, it's a huge efficiency, it's a core function of our contracting program. We have stopped short of using it to do, you know, RFP summaries and evaluations. You know, we're not using it to tabulate bids, because I think we're looking at it from a risk-based approach, right? We want to use AI. And, you know, we want our staff to use AI as efficiency in part of their process. And so we are trying to be very thoughtful about, okay, well, what are the types of things that could make us more efficient, that we have the capacity to still have the human in the loop, and that right now, as we learn how to use these tools, we're not impacting the procurement process in a way that either makes us have to do double work, because we're not as trained on the AI solution as perhaps we could be, right? You know, maybe we don't know how to spot hallucinations in a standard way, you know, those sorts of things. So I think for us, we approached it in a very sort of risk analysis way initially.

[36:21](https://youtu.be/2R9_YAW3Cx0?t=2181) Not to say that we don't want to use AI going forward. We will be rolling out, coming this next year, a fully electronic bidding solution that will begin to incorporate more AI into that process, but again, using a system that has electronic bidding in it, like full electronic bidding, not just let's do this by email sort of thing. And so that'll be a place for us to start to test this more. But I would also say that for us as just our state in general, when we came out with that very first executive order, it really was around testing AI in a way that made it easier for us to see what was happening. And that was our request for innovative ideas to solution that we've had since 2019. But it allowed us to do a handful of projects that were problem statement oriented, you know, general solutions to that we tested them in a sandbox environment, tested how things would work using our data using our AI solutions that were built.

[37:29](https://youtu.be/2R9_YAW3Cx0?t=2249) They weren't just a Google solution. We're talking five, six, seven, eight vendors that came together to create a solution. We tested it in a closed environment, and then made the decision on a secondary part of the contracting process, whether we wanted to do a fully upscaled solution. Some of them didn't make it. Some of them were, hey, this, this isn't ready for AI. This isn't something that we can do. And we found that in the testing. So I would say that's another way. Look at it from a risk based perspective. Start slow. Don't go gangbusters out in the procurement process. Go slow, but also find a way to test it if you can. We've got Poppy and we also had our RFI squared solution.

**Emily Royall**

[38:08](https://youtu.be/2R9_YAW3Cx0?t=2288) Got it. That's really helpful. And, you know, focusing it in on the use cases, rather than like, we're just going to use AI for everything. Like, what is the specific task or use case? Let's bound the deployment around that task and evaluate maybe the performance of different models accordingly. I'm curious, just to build off of that question. You mentioned that for one of, I forget which one, but for one of the procurement objectives that you had, chat GPT ended up being the best tool. And I'm just curious, how did you determine that? What was your process to evaluate that? How did you objectively decide that its performance was better than other models for that specific task? What's your criteria look like?

**Angela Shell**

[38:49](https://youtu.be/2R9_YAW3Cx0?t=2329) Yeah, so we had staff that were already using chat GPT, probably not in an approved way. But I think most people know that employees are in fact doing that because it was the latest, greatest tool. You have it on your phone, perhaps, but maybe not in the state. And so we had some team members that kind of had been using chat GPT to really test how the solution worked. But we also are a Microsoft organization. And so we also knew that we had access to Copilot. And so we started comparing the two. We started looking at, well, okay, these are the prompts that we're giving chat GPT. This is how we're pulling data. How could we do that in Copilot? And so we tested them both against each other. We did the same scenarios to see, can we go out to the web using Copilot and get the same information back? And what we found is in testing them against each other, some had strengths and some had weaknesses. And we just found that chat GPT was faster. It could find information in a better way than Copilot was able to. But we decided that Copilot was really helpful for us in synthesizing our data because it was existing data we already had in our possession.

[39:58](https://youtu.be/2R9_YAW3Cx0?t=2398) And so Copilot was better at synthesizing existing data and helping us be able to put that data together in a way that was easy for us to use. Whereas chat GPT, we found it was better used in pulling that data, finding it faster, and summarizing it for us. So I would say it just was an easier tool for the team on the research side in the chat GPT space.

**Emily Royall**

[40:29](https://youtu.be/2R9_YAW3Cx0?t=2429) Got it. Maybe you could elaborate a little bit because you've touched a few times on the importance of a good prompt and prompt development and having staff who are capable of building good prompts. Can you give us a feel for in the procurement space, what is the anatomy of a good prompt? And what's the difference if you don't have a structured or good prompt that you get varying outputs in your AI performance?

**Angela Shell**

[40:55](https://youtu.be/2R9_YAW3Cx0?t=2455) Yeah, so I'm going to admit I'm not a prompt expert by any means. My strategic sourcing team really is. But they had to test it first, right? A lot of this just happens by trial and error. And I think what they found is, as they started going through, they just kept revising these prompts. They started with something more broad and realized, wait, that's not giving me the information that I need. And so they had to keep going back and synthesizing that prompt and telling it, this is specifically what I'm looking for. I want it from this perspective. I want data on this. And they had to continue to revise those prompts to get down to what they were specifically looking for. And a lot of it was, again, testing. It was kind of playing with the solutions and seeing, well, if I change it this way, does it give me what I need? If I change it that way, does it give me? And they ultimately got to a standardized prompt that was giving them what they needed virtually every time. And that was what they went through. And they identified all of those prompts because, as hopefully you all know, at least in California, and I think this is pretty universal, those prompts are, in fact, publicly accessible data. So in California, we do get public records requests for prompts.

[42:12](https://youtu.be/2R9_YAW3Cx0?t=2532) And so you have to be careful what you're asking it and how you're asking it and really refining so that it comes back in a uniform way. So it isn't just getting the data to be uniform, but honing in on the specific prompt you need so that it's not a wild response that comes back and you have to decipher it each and every time. And so my team was able to get to a standardized prompt. And they're able to say, these are the prompts that we asked it, and these are the results that we got. We haven't gotten a PRA, a Public Records Act request for California, in this particular space. But if we get one, we're ready to be able to show what we used and how we came about that prompt that got us to. And so, again, they're using that same prompt each time.

**Emily Royall**

[42:56](https://youtu.be/2R9_YAW3Cx0?t=2576) Yeah, that's a great point. Increasingly, we're finding open records requests for prompts related to, you know, various functions of government and certainly transparency around procurement is imminently going to be one of those if it's not already. Another quick question for you, and then I'm going to move into the audience Q&A. So folks, if you're thinking about questions for Angela, now is the time to drop them in the Q&A function. And I will start to search those for some good questions. So Angela, you also shared a little bit about human-in-the-loop. And you talked about how important that is. Do you have any reflections on specific points in the procurement process where human-in-the-loop is just absolutely necessary and should be considered a requirement?

**Angela Shell**

[43:45](https://youtu.be/2R9_YAW3Cx0?t=2625) Well, dare I say, I really think it's every part of the procurement process, honestly, because, you know, if your team is using it to develop scopes of work based on, you know, historical information, and you're compiling a new scope of work, and you want to, you know, you want to give it questions and have it create something, you absolutely have to go back and look at that. And that's really the basis. You have to look at that and see, am I asking for something in this scope of work that, number one, I don't need, number two, wasn't in the last scope of work, and should it be in this scope of work? Like, I just think you can't. It makes it more efficient in starting, right? You don't have to start from scratch. You can ask it to do certain things. But you have to go back and make sure, because the last thing you want to do is put procurement out, that bidders are spending money, because it is very costly to respond to a bid. They're spending money on something that you don't actually need. And so I think it starts in every phase. If you're, you know, collecting information and summarizing that information.

[44:46](https://youtu.be/2R9_YAW3Cx0?t=2686) So, you know, in our competitive bids, we summarize everything into a single sheet, and that's what we use for the procurement file. Well, you might have had to type that out or manually do that. If you're going to use AI, even to summarize that information, you have to go back and look at it. and make sure that it is factual, that it's not, you know, hallucinating, or it's not putting biases in RFP evaluations, any of those, I think that, you know, we've been very sort of adamant in California that, you know, AI and generative AI is not replacing our workforce. It is taking some tasks and making it faster for you to get to a summary or a compilation or, you know, something that you can use, but you absolutely have to go back and look at that work product, because at the end of the day, it's your work product. It's not the AI's work product, right? So I would argue that it is in every point of the procurement process, even if you're advertising something, right? And you're just putting that information out there, you have to make sure it's correct.

[45:49](https://youtu.be/2R9_YAW3Cx0?t=2749) That's the integrity of the procurement process. And by no means does that go away by using an AI tool.

**Emily Royall**

[45:56](https://youtu.be/2R9_YAW3Cx0?t=2756) Absolutely. Great, thank you for that reflection. I am going to head over to the Q&A. There are tons of questions, we may not get to them all, but we will do our best. Zooming out a little bit, let's talk a bit about California guardrails around implementing AI. So a question comes from an audience member, what exact guardrails is California implementing to ensure data safety and procurement AI? Specifically, how are you enforcing data privacy and contractually blocking vendors from using state data to train commercial AI products?

**Angela Shell**

[46:32](https://youtu.be/2R9_YAW3Cx0?t=2792) Yeah, so this goes back to what I talked about before with our state terms and conditions, which were written very, they were crafted very carefully to address these specific points. And so every vendor has to agree to it. But I know it's one thing to have it in a contract and then walk away and never know if that vendor is actually adhering to it. And so again, our IT terms and conditions spell out the definition of data, what is public, what state data, what's not, what's public, how can you use it? What are you barred from using? It goes through all of that in our IT terms and conditions. And then again, it goes back to that contract management function within those state departments to make sure that they are in fact checking what is happening. It's not a one and done. They are checking to make sure that when AI and specifically generative AI is being used, that it's not doing something that it shouldn't be, and that it is aligned with the state's terms and conditions are again, our terms and conditions are fairly lengthy. And they're pretty specific on what we will and won't allow. And we do have requests from vendors at times that want to modify that. We look at it every time we look at it using our CIOs, using our CISOs, our Chief Information Security Officers, to be able to look at what is happening with our data and should we be allowing this? Should we not? What's the risk? So we are doing that every time.

**Emily Royall**

[48:02](https://youtu.be/2R9_YAW3Cx0?t=2882) Got it. Excellent. Follow up question from an audience member. Since AI allows you to understand the market better, do you see the possibility of using AI to help achieve other goals? Either through contracting out or other use cases, including or other than improving efficiency?

**Angela Shell**

[48:29](https://youtu.be/2R9_YAW3Cx0?t=2909) I mean, I would say that sort of the genesis of us using AI really is around, you know, helping our workforce be more efficient at what they're doing, while also, you know, helping the state get the data that we need, you know, the solutions that we need. So it really is, you know, around the idea of government efficiency. And frankly, that's just a term that never goes away, right? Sometimes it comes out as procurement reform, procurement modernization. But at the end of the day, it is all about efficiency. And that's really, and some of it creativity, I will say we are beginning to use AI in our training programs as well, helping us create training videos around procurement that are using AI in a way that speaks to that human centered way. So we also have human centered design initiatives in the state of California. And that's the AI tools are also helping us, especially on the training side, create something that is a way in which our audience needs to receive it rather than us just determining that, well, you know, this is the best way of training from, you know, 15 years ago, that just isn't the case. So I mean, I know, that's sort of a vague answer. But for us, it is about efficiency. But it's also about better customer service to our users. And our users are the thousands of buyers, just in our state of California departments, you know, each and every day.

**Emily Royall**

[50:03](https://youtu.be/2R9_YAW3Cx0?t=3003) Great, um, maybe diving a little bit into some technical questions around contract lifecycle management. So regarding contract lifecycle management, how do you ensure a vendor doesn't offer lower pricing to another entity for identical specs after securing a state contract? And does California include a mandatory price reduction or most favored customer clause to handle this? And can AI potentially help monitor cross agency pricing and compliance? This is a good question. Yes, I can read it again, if you need me. Yeah, I

**Angela Shell**

[50:43](https://youtu.be/2R9_YAW3Cx0?t=3043) may need to come back to you on the back end. But the first part, I'm trying to take notes. So I have a educated response here. So with respect to the pricing, so in fact, we actually allow on all of our leverage procurement agreements, vendors to go lower. Even if they're competitively bid, they can a lot of ours are multiple awards. So this is where we come into that secondary level of competition. Because again, my contracts are $0. So I'm not telling them that you're buying, you know that the state's buying 1000 widgets at this price in this timeframe, I'm saying the state could buy 1000 widgets in this timeframe, and I expect you to offer it at a maximum of this price. So we give them the ability that if they are if they get a department, let's say it's computers, right, and they get a department that's going to order 1000 computers in one shot versus another department that's going to order 100. That's where I want to make sure that my pricing, it may not be the lowest it could possibly be, but my pricing is very, very good. And it is saving the state dollars. So that even if they can't lower the pricing, because they're only buying 100 versus 1000, they're still getting a really good price. So that's part of that importance on market research. But yes, I would say that they do have the ability, my vendor pool does have the ability to reduce their contract pricing, in order to receive part of that secondary competition if they want to get to better pricing, but I am expecting that my initial pricing is very good. And if it's not, and I know because I do have data, again, I touched on all my data as public, and we do use some tools on the back end, that allow us to analyze that data in ways that perhaps the public isn't able to see. So that we can see, well, where is our pricing? Are we consistently seeing that this vendors offering vendors lower pricing, we should look at that. And we should look at whether I go back, because again, based on my negotiation statute, I can go back and say, I want better pricing, renegotiate, or we're going to

[52:46](https://youtu.be/2R9_YAW3Cx0?t=3166) need to move on and and go to another, you know, vendor or do something differently. So we have all of those tools at in our toolbox at any time as the state purchasing office, my state departments cannot renegotiate. Once you're entered into a contract, that's your pricing. But for me, I can

**Emily Royall**

[53:05](https://youtu.be/2R9_YAW3Cx0?t=3185) renegotiate at any time. Okay, and that that kind of falls into maybe the latter half of the question, which was asking about, like, could AI assist you with said analysis? Like, do you feel like AI can support that kind of cross agency

**Angela Shell**

[53:18](https://youtu.be/2R9_YAW3Cx0?t=3198) monitoring? Okay, yes, it could. Because again, and it's also I think there was a question in there about compliance, because again, it is how we are able to see with the market research that we have now, it's how we're able to see if departments are paying more than they should be, right? Because we have over 200,000 plus buys, or more in any given year in the state of California, just from state departments. I have a team that's a data data management team, but they can't possibly see all of those. And so we are using tools that allow us to see, hey, are people paying more than they should be? Are they paying more than that maximum price? And then who's getting consistently better pricing? And what do we do about it?

**Emily Royall**

[53:58](https://youtu.be/2R9_YAW3Cx0?t=3238) Excellent. There's a kind of a group question, like a lot of people are asking about resources and where they can go to find different resources. So one of them is like, are the terms and conditions that you're referencing publicly available anywhere that folks could use or look at prompts and prompt library, perhaps if the state offers or provides a prompt library anywhere for folks to be able to use. And then maybe if you are a municipality in California, where can you go to learn more about the state's approach and how other cities can model the state's approach to procuring using AI?

**Angela Shell**

[54:36](https://youtu.be/2R9_YAW3Cx0?t=3276) Yeah, so our terms and conditions are on the DGS website, dgs.ca.gov. You just look up model terms and conditions. So that's all there. Anybody can use it, all of my various types. So I have non-IT services, we have IT goods, IT services, cloud, non-cloud, so they're all there. And all the modifications over the years, you can see that history's there. We don't have a prompt library, but I will take that back. We don't have something that we provide access to just in general. But certainly, if somebody's interested in a certain type of prompt, perhaps prompt, perhaps around market research, feel free to reach out. My contact information is maybe somewhere, maybe we can get that. But yeah, so we'd be happy to share that. And if you're a municipality, again, also, most all of our information is online. We have a website on AI, GenAI. I can't think of the name of it or the web address right now, but we do have a GenAI website that kind of walks you through a little bit. I will say it's a little outdated, but it does walk you through. Are you a new user to AI and you want to buy it? What do you do? Like it's got a little bit of walkthrough, and that might be helpful for municipalities. But also, we are happy to talk to any of our partners, you know, in California or any state, frankly, about the things that we've done. And we're happy to share our executive orders, you know, any of the other tools that we use. You know, we also use the, the name is escaping me. It's out of the Bay Area. Oh, Emily, you know, this one.

[56:09](https://youtu.be/2R9_YAW3Cx0?t=3369) BAPA? No, it's the, it's the AI, GenAI.

**Emily Royall**

[56:14](https://youtu.be/2R9_YAW3Cx0?t=3374) The GovAI Coalition?

**Angela Shell**

[56:16](https://youtu.be/2R9_YAW3Cx0?t=3376) GovAI Coalition. We also go on there and we look at their templates. If you guys have not looked, they have some great templates online. It is started out of a local entity in California. So it's, it's, it's fairly California based, but take a look at that. That also has some great resources. We went there when we were looking to get our disclosure.

**Emily Royall**

[56:37](https://youtu.be/2R9_YAW3Cx0?t=3397) So, that's awesome.

**Angela Shell**

[56:38](https://youtu.be/2R9_YAW3Cx0?t=3398) Yeah, that's where we started. And obviously we modified, but they really helped us out when we started down our GenAI journey and wanting to know if vendors were in fact using GenAI in their solutions. So, yeah. So, yeah, everybody's asking for the website. I apologize, you guys, I don't have that at my fingertips.

**Emily Royall**

[56:56](https://youtu.be/2R9_YAW3Cx0?t=3416) I can drop it in a moment. And just a plug as well. The GovAI Coalition has a procurement committee. I co-chair that with Aaron Bushman from the state of Colorado. And so if you're interested in continuing conversations about procurement and AI, that's a great place to do it. And I will drop the information in the chat on that as well. It looks like some there you go. Yes. Fantastic. Yes. Fantastic. So Angela, just one last question for you, and then we'll turn it back over to our InnovateUS friends. One key takeaway. If there's one key takeaway that folks should walk away from this conversation remembering and never letting go of, what should it be?

**Angela Shell**

[57:35](https://youtu.be/2R9_YAW3Cx0?t=3455) I would say don't be afraid of AI. And the reason I say that is that it's here. People are using it. My teams were using it before. Maybe they're using it on their private phones or whatever the case may be, but people are using it and it isn't going away. And so we have to embrace it. And that's one of the things I told my leadership team when they said, hey, people are using it to develop scopes. What do we do? I said, you embrace it and you give them the tools to do it properly. Those of us that have been around for a long time, it's a little fearful for us, but at the end of the day, we have to embrace it because if we don't, people are going to use it wrong. It's going to get into the procurement realm in a way that it shouldn't be. And we want to be the best users of it. So that would be my advice is embrace it, learn how to use it properly, take a process, and then use it. And then use it in a prompt class if you have one, do all of those things and don't be afraid to use it, but make sure you're evaluating the

**Emily Royall**

[58:38](https://youtu.be/2R9_YAW3Cx0?t=3518) risks as well. Great. I think that's an awesome place to wrap up. Be the AI procurement change you want to see in this world. Much better said, Emily. Thank you, Angela. Thanks for sharing all of your expertise. I know it's a tremendous support for everybody in the procurement community nationally and beyond. So thank you so much. And I'll turn it back over to Innovate US to close us out. Thank you so much, Emily and

**Bonnie McGilpin**

[59:05](https://youtu.be/2R9_YAW3Cx0?t=3545) Angela again for sharing so much about your work and for that great discussion today. Thanks to our partners at NASPO and the Partners for Public Good. Please take the post-workshop survey when the session closes. Your responses inform our future offerings. All participants of this workshop will receive a follow-up email with additional resources. And then join us for our final workshop in this series, October 19th at 2 p.m., AI Tools Showcase, What's Out There and How to Evaluate It. That's with Santiago Garces, the former Chief Information Officer for the City of Boston. We look forward to seeing you there. Thanks again, everyone. With that, I'll end this session.

**Angela Shell**

[59:42](https://youtu.be/2R9_YAW3Cx0?t=3582) Thank you.

* * *

Original page: https://innovate-us.org/ai-across-the-procurement-lifecycle-where-does-it-actually-fit · Licence: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org