--- id: "workshop:simpler-services-stronger-access-designing-inclusive-systems-with-ai" type: "InnovateUS workshop" title: "Simpler Services, Stronger Access: Designing Better Systems with AI" url: "https://innovate-us.org/simpler-services-stronger-access-designing-inclusive-systems-with-ai" date: "2026-04-27" series: "AI and Human Services: Innovating to Serve People Better" instructors: ["Davyd Smith","Jessica Lax"] moderators: ["Rob Asaro Angelo"] recording: "https://www.youtube.com/watch?v=2bty2lOmjdM" transcript: "automatic, from the recording" source: "InnovateUS (innovate-us.org)" licence: "CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org" --- # Simpler Services, Stronger Access: Designing Better Systems with AI InnovateUS workshop · 27 April 2026 Complex forms, fragmented processes, and language barriers remain some of the biggest obstacles to accessing human services. This session explores how AI, when paired with strong design and behavioral insights, can help agencies simplify services rather than digitize complexity. Participants will examine how AI can support plain-language communication, reduce administrative burden, and improve access for multilingual and limited-literacy populations. The session emphasizes inclusive design choices, safeguards, and testing approaches that ensure AI supports equity rather than introducing new barriers. This workshop is designed for program designers, service delivery teams, digital service staff, and anyone involved in improving client experience ## Learning goals By the end of this workshop, participants will be able to: - Identify opportunities to reduce administrative burden before applying AI or automation - Apply inclusive design principles to AI-supported forms, workflows, and communications - Understand risks associated with AI translation and accessibility tools—and how to mitigate them - Assess whether AI-enabled service changes genuinely improve access for underserved populations ## Speakers - **Davyd Smith**, IT Director, Colorado Governor’s Office of Information Technology (OIT) - **Jessica Lax**, Program Director, State AI Readiness, Center for Civic Futures ## Moderators - **Rob Asaro Angelo**, Senior Fellow, Burnes Center for Social Change & Professor, Northeastern University Part of the series [AI and Human Services: Innovating to Serve People Better](https://innovate-us.org/workshop-series/ai-and-human-services). Recording: https://www.youtube.com/watch?v=2bty2lOmjdM ## Transcript Automatic transcript of the recording: names and terms can be misheard, and the times can be a few seconds off. Each time links to that moment of the video. **Rob Asaro Angelo** [0:00](https://youtu.be/2bty2lOmjdM?t=0) We've come to the penultimate of this series. That's my last one with you all. I'm not going to be here next week, but thank you all for your support and making it very easy to be a moderator because all the amazing questions that you always put in the Q&A. So, thank you for all that. Today, we have two great speakers to help us learn for today's workshop. They're going to help give us the framework to explore how an AI, when paired with strong design and behavioral insights, can help an agency simplify services rather than digitize complexity. Something I always looked out for my time as commissioner was that we're creating a new tool, whether AI or anything else, where we're making a bad process just easier to be a part of. We want to make sure that our processes are being improved along the way when we're improving technology as well. I remember last year at a Center for Civic Futures event, I think it might have been Mikey Dickerson talking about when you're moving from one place to another, you don't take the ratty clothes and old dish towels with you. You make sure you're upgrading when you go there as well. [1:02](https://youtu.be/2bty2lOmjdM?t=62) So, when you're moving from one technology to another, don't take the bad stuff with you. Today, we have two great experts joining us who have amazing experience in the States as well, so that makes them my partners. One with Jessica Lacks, former New Jerseyan in work, but always a Jersey girl in life forever because she was there, but she's now the program director for state AI readiness at the Center for Civic Futures. And my friend, David Smith, who is the IT director in the Colorado Governor's Office of Information Technology. And David, check my Department of Labor friends and CDLE friends in Colorado. They all give you two thumbs up as being a great partner in all the work that you do together. So, thank you for what you do to help workers in Colorado. And as usual, I'm not surprised that Jersey and Colorado are leading the way on so much of this stuff, and we're so happy to be sharing our successes, our trials and tribulations, the challenges we had to overcome to get there as well. So, I want to start with David today. David has a great presentation to talk to us about, and more importantly, just please ask us questions. [2:05](https://youtu.be/2bty2lOmjdM?t=125) David and Jessica come with a wealth of experience and knowledge and know how to help us ask the right questions when we're coming up with our processes also. So, I'll turn it over to you first today, David. **Davyd Smith** [2:15](https://youtu.be/2bty2lOmjdM?t=135) Super, I'm presenting myself. **Rob Asaro Angelo** [2:18](https://youtu.be/2bty2lOmjdM?t=138) I think I'm going to, or Christine are going to do the slides. **Agueda Quiroga** [2:22](https://youtu.be/2bty2lOmjdM?t=142) Okay, David, as you prefer, okay? **Davyd Smith** [2:29](https://youtu.be/2bty2lOmjdM?t=149) All right, can everybody see that? **Rob Asaro Angelo** [2:32](https://youtu.be/2bty2lOmjdM?t=152) Yep. **Audience** [2:32](https://youtu.be/2bty2lOmjdM?t=152) Yes. **Davyd Smith** [2:33](https://youtu.be/2bty2lOmjdM?t=153) All right, super. So, today I thought I'd talk about, these are actually not applications that we built in Colorado. Before I start, actually, I am a Jersey boy too, just to be clear. I think I, you know, I live in Colorado now, but you know, I started in Hoboken, New Jersey. So, I know, a little credit where credit's deserved. But yeah, so what I'm going to talk about today, these are not actually in Colorado. These are things I've been talking about here, but I was thinking of the way that we're approaching AI in a lot of places. We have a lot of sideload things going on. And in government, when people come in, they're often coming into the government, not to like an individual program or service. And so, the first thing I want to talk about is, I think it'd be interesting to see more centralized translation support, language support. [3:33](https://youtu.be/2bty2lOmjdM?t=213) Because what I am seeing today is a lot of one-off like chatbots in the same organization, right? You have a department of public safety, you have health services, you have education. And they're each doing their own thing. And if they want to support multi-language, which is great, they should be doing that. They're all doing it on their own, right? So, you might have three different LLMs to begin with, right? Which may actually handle the language a little differently. You also have a regional localization with whether it's just dialects of existing languages or maybe in the region that you actually are is different than standard translations. So, when a person comes in and they're looking for something from government, they're quite often not like, I'm talking to department of education. They're like, I'm talking to Colorado or I'm talking to Philadelphia. [4:36](https://youtu.be/2bty2lOmjdM?t=276) I'm talking to a state or a city or local government or a province. They expect some kind of standardization on the other side that they're having the same conversation over and over again. And when we have all these separate siloed translation services across our organization, I don't think we're actually creating that connection with our constituency. So, that's one of the important things. And then of course, the cultural nuance that comes with that. And when we have these fragmented systems, we also don't get the scale. If you have five different translation systems and you update them for local nuance and colloquialisms or just vocabulary, sometimes it's different. And if it's actually speech, it may actually be the way words are said from one place to another. [5:41](https://youtu.be/2bty2lOmjdM?t=341) You would be controlling this centrally as opposed to doing it over here and then not doing it over here. So, the same person comes in, they go to this service one day and they're having a conversation that they fully understand. They come in another day and they're having a conversation that they don't fully understand. They'll probably understand that as well as long as they speak the basic language, but it's still a different experience and a different way of dealing with it. If you can centralize that as you find things that you need to find, you can actually add words, add vocabulary to it. This is really just an improvement on current translation services because the AI component gives you some ability to do that. The other thing is I was thinking about acronym definitions. I was actually looking at a couple of different languages for acronyms and when you define it in English and then you define another language, that's something you may want in your system configurations, your system prompts to have your LLM actually looking at, oh, well for us CDHE means, or CDE means Department of Education and for another group, it may mean Department of Environment. [7:11](https://youtu.be/2bty2lOmjdM?t=431) So CDE does not work and when you even are going into a language, it could be interpreted, it could be not be interpreted at all. They wouldn't have any idea what the acronym is if you're using those acronyms, but you can put that in and actually add that. Again though, if you have five systems, you have to do it five times. If you have one, you could actually do it. So I think this is a good example of a lot of times we do want to limit the AI. We do want that retrieval augmented rack so that we maintain the accuracy and the control over the content, but with something like language, I think you want it to represent across the entire enterprise and that's one of the reasons why I brought this example forward. So as you say, from an API cost, one-off implementations, you're going to have multiple, you're not going to get economies of scale and so you may, it may actually cost you more for that. [8:15](https://youtu.be/2bty2lOmjdM?t=495) For development, you're definitely going to have five different groups developing it. So for implementation, it's probably going to cost you more because it's the same thing, right? It's a translation service and then lastly for that maintenance, exactly the same. Instead of having multiple groups supporting multiple translation services, if you have one that everyone can plug and play across your organization, you get the economies of scale, you get the consistency with constituency and you get lower full cost of ownership. And the second one, this is related, but it's a little different, is the search bot for an entire organization. Again, this is just the next generation search, right? We already have search. Many of us may have it all across our enterprise. At Colorado, we have 140 sites, right? [9:17](https://youtu.be/2bty2lOmjdM?t=557) So 140 sites is a pretty lot. Well, that becomes your corpus. This is all public information that you're giving it access to and if somebody comes to any state and says, I'm interested in the services that the state provides for child benefits. Well, that's in CDE, right? That's an education. It's probably in health services, maybe in human services, maybe in department of safety and it could be in others. So I've been in Colorado for six years, right? In the government. I've been here longer than that. Before I came, I didn't really know how to navigate all our websites and find the services that I wanted. Now that I'm there, I could do a much better job, but I still can't get everything where I want immediately. I thought, well, wouldn't it be great if I said, hey, I'm interested in child services and it said, oh, well, these are all the child services and with each one is the link. [10:23](https://youtu.be/2bty2lOmjdM?t=623) This is at CDE. This is at CDEC. This is at CDHS. And now I have a narrative with all the things I could do and I can see the connections between them, which I can't do today very easily on a search. And the idea is that there's no wrong door because we lose people and get a bounce rate because of that, right? People come in, they try to figure it out. I can't figure this out. And quite often, they're actually the people that probably need our help most. And we lose them sometimes by not having an easy way for them to engage with us. And instead of 20 links, they go to Google and ask that question and they will get 20 links and they'll have to go to 20 sites. And maybe they're right because it's Google, right? It's looking at all kinds of things depending on your search term. [11:25](https://youtu.be/2bty2lOmjdM?t=685) If you're actually in ours, which again, it's a large corpus, but it's our corpus that we can control. And we can actually say, no, you're only looking at the public information that we already make available to people. We're just making it easier for you to capture. And then the last thing is if they can't find it, a lot of people don't wanna pick up the phone and they don't wanna talk to a person, right? But if you can't get to where you want to, you are going to, and you're gonna call the first number you have, which may not even be the right department, probably not the right person. And now we're creating all this work for both the person, the constituent, and the people in your organization try to figure out how to help them get to where they want to get to. So for this one, it's pretty similar, but it's a little different. [12:25](https://youtu.be/2bty2lOmjdM?t=745) If you have a siloed corpus, which again, sometimes is good. It's something you do want because you wanna control the information, the accuracy, the veracity of the information. But if I'm in that one from one agency and I'm talking about child benefits, I'm only talking about the child benefits that they offer. I don't know about the five others and other agencies when I'm talking. So the enterprise bot actually gives you that scope and that holistic view of the organization. And then of course, just the time that the constituents spend. We often talk about efficiency inside government as in, oh, how's this gonna reduce the number of hours that our employees have to spend doing it? What about the number of hours that the constituent has to spend to talk to us? Even if they do not get tired and walk away, how much time are we making them spend in order to find out the things that we're supposed to be offering them as easily and as professionally as possible? [13:34](https://youtu.be/2bty2lOmjdM?t=814) So the enterprise bot again, helps the constituent of hopefully saving time and actually getting to the valuable information they need to get to. And then the last one is again, the maintenance and scale. If you have a search bot on every agency, you have to maintain that. You have to implement it, you have to maintain it, you have to keep it updated. If you have a single source truth in one application, you're gonna get the economies of scale on that and actually save money while providing a better service. And then for both of those, both the language services and kind of a universal organization bot, you have security and compliance, which again is economies of scale, but it's a lot easier to take all the money you're gonna spend and all the resources and time you're gonna spend to secure each one of those for just basic cybersecurity, but for PII and everything else, you'll have that in one place. [14:40](https://youtu.be/2bty2lOmjdM?t=880) You can actually get the security that you need and you don't have to worry about, oh, well, this one's okay, but this one we have to actually track down and we have to improve. So these were just two ideas I wanted to put out there because there are probably a hundred more, but I wanted to give you an idea of When you actually look bigger, and because these are simple, these are something we were already kind of doing, right? We quite often have a language tab at the top for a webpage, and we have multiple languages, which is helpful, but this is the next generation. And then for the search bot too, quite often we have searches across the whole enterprise, but not with AI, not that they can actually build it up and actually talk back to the person and help them navigate the differences across many of those agencies. Instead of just getting links, they get a little more information. And with that, I'm going to hand it off. I think we'll do questions at the end when I can do it now. Yeah, real quick. First of all, David, **Rob Asaro Angelo** [15:44](https://youtu.be/2bty2lOmjdM?t=944) thank you for all that. One, I love the idea of like a bounce rate for human beings who can't get the service they need, right? They go to the wrong door, they get bounced. Like an email gets bounced when it goes to the wrong address. And that's something that I'm definitely going to steal, and it's something that's way too common with so many of our government services. Secondly, if you can define for some of our participants today, what do you mean by corpus? **Davyd Smith** [16:09](https://youtu.be/2bty2lOmjdM?t=969) Oh, sorry. Yeah, so the corpus is basically the data that you use in your AI system that is going to help you answer the questions. So to take a really simple one, I created something that just had our security policies in it at Colorado so that I could easily ask questions. It doesn't look at the World Wide Web. It doesn't look at anything else. It looks at the corpus, which just means body of knowledge that I limited it to. **Rob Asaro Angelo** [16:43](https://youtu.be/2bty2lOmjdM?t=1003) Great, thank you. And then also, the things you just talked about, are that's what's aspirational in Colorado, or that's what's actually happening now for your current, or is it varied by- It's aspirational of David. **Davyd Smith** [16:58](https://youtu.be/2bty2lOmjdM?t=1018) I do know that Indiana, just so you know, Indiana has a search bot across though. I did check that one out when I was looking at this, and I do encourage people to go look at it. I thought it was pretty cool. I don't know anything about Indiana, so I'm assuming it's telling me the truth. But I did enjoy seeing that where it covered the state. **Rob Asaro Angelo** [17:19](https://youtu.be/2bty2lOmjdM?t=1039) Great, thank you, David. And now we're going to turn it over to Jessica. Thank you so much for **Jessica Lax** [17:23](https://youtu.be/2bty2lOmjdM?t=1043) being here today, Jessica. Thank you. David, love the no wrong front door, always thinking about that in the government space. Great. Can we pull up my slides, please? Awesome, thank you. Okay. So I'm going to talk about two patterns and one recipe, basically really common set of use cases that we see in government, more so on the reducing the time it takes to do some of these administrative tasks. So the recipe, I'll start with that, and then I'll give some examples, is usually have some kind of messy input, right? There's like some set of data that you have, and it's not really well organized. It might be in different formats, there might be images, there might be scans. So that's a messy input. And then you have a desired output template, right? So for example, you might be drafting a memo with all this messy data, and that's a template. You do the same memo in the same format every time. And then you always need a few good examples, like what are like your creme de la creme of like what you want to come out with. You need a thoughtful prompt, and then just iterating versus a benchmark. And I'll explain all of this, but I like to use this recipe for two different use cases that I'm going to go into a little bit. So we can go to the next slide. So before you start, you need to just check that your LLM is actually reading your data. I've had a few different use cases where, you know, I thought it was reading it. Okay, it was reading this part of it, but it didn't read this part of it. [19:18](https://youtu.be/2bty2lOmjdM?t=1158) So really, you kind of need to investigate for each type of data, like if it's reading it. So I'll just give some examples here. Like, you know, you might want to ask it to reproduce a page or summarize a page for you, and you can tell like that it wasn't able to read that. But maybe there's a chart. Can it read the chart? Or maybe there was a scanned PDF. Can it read the image? Can it tell you what was in the image? Maybe the scanned PDF has a signature. Can it read the signature? And so you really want to start to think about the typology of your data, and like what's in it, and really testing for each one of those. I think if you're starting to see that certain parts of your data aren't, and I know so many people, I think when I talk about data, think about just like a big spreadsheet, but it can actually be a lot of different things, right? It could be like a 500 pages of cases. It could be like thousands of public comments. Like data isn't just numbers in a spreadsheet. It can also be in a lot of different formats. And so I think when you're doing this testing, you know, maybe you are running into, okay, it's reading this type of data, but not this other kind. There's a lot of tools to help LLMs understand that data, whether it be OCR or text extraction. There's also, you know, limits to how much data you can put into a lot of LLMs, and so there's a way to sort of break down that data into what's often called chunks. So it can start to process it, because what happens is if you put really large amounts of data into an LLM, it will start to dilute the quality of responses you get, because the more that goes into an LLM, sort of the less it can process, kind of like a human, right? Like someone's telling you way too many new things, you're just like, well, you took like five of those things away. And so the same can sometimes be true with LLMs. So just, you know, first and foremost, make sure you understand what [21:25](https://youtu.be/2bty2lOmjdM?t=1285) the LLM understands about your data. Let me go to the next slide. So one example I mentioned is you have 100 pages of data. Again, it could be like, it could be case information on a person or a family. It could be, you know, lots of grant, like different data from you need to submit for a federal grant report or other kind of reporting. And so you could feed it 100 page unstructured document, a template of the output you want. So maybe there's a specific spreadsheet you want to use, or there's a specific, you know, doc form formula that you like, you know, there's an overview section, there's a, I don't know, bio section. And then two to three examples of really, really good past outputs, like what your desired output would look like. And then what you'll get back from that is oftentimes you can get pretty close, like you could take that 100 pages of unstructured data and create an example draft of your template, right? It'll take the template and give you the output. I've seen this done a few times. It saves, in other cases I've seen, save 90% staff time for them to do a lot of other important work that they would like to be getting to. [22:55](https://youtu.be/2bty2lOmjdM?t=1375) And then we can actually take the same recipe in another scenario. So if we just go the next slide, their example here is kind of flipping it, you have a lot of raw comments, right? A lot of states and a lot of agencies require, for example, public comment on any new rule proposed, and they'll get hundreds of comments on each of those. I mean, there's so many different things that they get public comments on. And each agency has sort of their own formula, their own way of processing all of that information. And so again, here, you're doing the same thing. You're taking all of those unstructured comments. Sometimes in this example, you might have a set of rules that they're responding to. Those rule pages can be 300 pages long. And sometimes what you want to do is connect the comment to the rule. And then you have an example of a spreadsheet. So you have the example output again, like this could be from a rule or regulation you had like two years ago, three years ago, something that worked really well. Again, you want your best example here. And you can use the same format to get the LLM to basically look at the rules, look at the comments and make a correlation. It can even tell you sentiment, it could tell you keywords, it could tell you themes, really all based off of whatever output you're asking for. [24:20](https://youtu.be/2bty2lOmjdM?t=1460) Based off of your prompt. We can go to the next slide. So I like to talk about some evals beat no evals, because I think people are really afraid of evaluation. And they're afraid of, you know, it's a scary term. And it definitely can be really, really robust. But if you're not going to do any evaluation, it's better to do some evaluation. And I think four things to keep in mind when you're doing evaluation is always like accuracy, completeness, bias, and tone. And so a really simple way, you know, just like a V1, a bit of evaluation, let's say you get an output, you get a spreadsheet that has all the rules, and it's comparing it with has sentiment, and it tells you what rules and regulations it's from, I would always do your first one as a test against work you've already done a human, completely human generated work. And you can check and see, how does this compare to what I created on my own, you can check for accuracy, completeness, bias, tone on a scale of one to five, and see, you know, where's it doing well, where isn't it doing well, you can improve some of your prompts. Based off of that, you might even discover, oh, maybe it actually didn't read certain parts of my data. I mean, this could be a really good indicator for, or maybe it's just not a good use case for AI. Like, I think, you know, there's a degree to which you can tweak things and a degree to which you might realize this isn't the right scenario. I'll say like a lot of the LLMs are constantly changing their quality, their output. And so you know, maybe six months later, it does work. And that's something to pay attention to as well. [26:04](https://youtu.be/2bty2lOmjdM?t=1564) And I think, like I said, in my experience, doing some of this work can really save huge amounts of time. And if you think about that as a desired output for for users for constituents, right, like maybe they submitted a permit, and the permitting process is taking long. If this is improving that timeline, that really helps our users. Thank you. **Rob Asaro Angelo** [26:34](https://youtu.be/2bty2lOmjdM?t=1594) Thanks, Jessica. A couple of questions I have before we get into the Q&A. So I'll go backwards. So I think it's a great point to make, because I think I've seen this myself, even though I'm not a heavy user, that if something doesn't work with your AI product on Monday, try again next Thursday. It might be it might be developed or gotten improved that much by that point. **Jessica Lax** [26:58](https://youtu.be/2bty2lOmjdM?t=1618) Very true. Yeah. I mean, it's changing all the time. I even just noticed on one of the common LLMs I use, they added a new feature over the weekend, which is always fun to see what they're choosing to prioritize. And, you know, I'm sure they're using data to make those decisions about what they're seeing their customers being interested in. **Rob Asaro Angelo** [27:22](https://youtu.be/2bty2lOmjdM?t=1642) And then a quick question. You said more data leads to dilution. When you talk about that, do you mean like a spreadsheet with a thousand lines is going to have more dilution than a spreadsheet with a hundred lines? Are you talking about different types of data that are uploaded at the same time? **Jessica Lax** [27:41](https://youtu.be/2bty2lOmjdM?t=1661) Both and all. I mean, I don't know if you've ever had like a really long conversation with an LLM, but over time it starts to get messier and messier. And I think the more context it gets, including in terms of data, sometimes the more hallucinations it can have, the more diluted its responses can be. And so that's where you want to use something like chunking, which is where you might like take a portion of the data and get it to create a certain output and then give it another portion of the data and create a certain output. So it's, you know, getting less context every time and it's a little bit more focused on each output. **Rob Asaro Angelo** [28:18](https://youtu.be/2bty2lOmjdM?t=1698) Great. Thank you. So just to remind everybody to put questions into the Q&A function. I know there's a tutorial in the chat on there. Christine, if you do me a favor, I see there, I think there are some questions that are going in the chat, not the Q&A. That's fine. We'll figure it out. So the first question here is from Mark about translation support. Shouldn't we be paying translators? And yes, I share Mark's concern as a former commissioner of labor. I care about all workers. On top of my experience with translation in New Jersey, we actually won an award from USDR on this of our translation model with help from innovation, by the way. Thank you. To, you know, what we did was we used, I'm going to back up for a second. In New Jersey, there's, you know, our claimants, our UI claimants speak closer to a hundred different languages. The availability and accessibility for translators for all those language is, is not enough to meet the need of our claimants. And I always felt that, yes, I care deeply about public employees, especially my employees. I always want there to be more of them. I always want them to be better trained. I always want them to be more available. But in the end, we have a duty to serve our customers. And if there's tools that are going to make our lives easier for the folks that we're serving, we're going to use it. We've tried many attempts with translation services and language lines that were not, just were not what our customers needed or our staff needed, quite frankly. Clearly, we have plenty of Spanish speaking staff in New Jersey, but that's, you know, Only the tip of the iceberg for the number of languages spoken. [30:02](https://youtu.be/2bty2lOmjdM?t=1802) So what we did was, you know, we used AI to help us with our translations, but it was always based with, I don't say human, but humans in the lead on that service. And by those humans, I mean our own agents who are the ones taking the calls from UI claimants in all kinds of different Spanish dialects and countries of origin, because those words can mean different things depending on what the country you are from. So we basically had a three-way check on the accuracy between AI, our own agents, and staff and claimants in the field who we check this with as well. So yes, it's not perfect, and in a perfect world, there would be unlimited amounts of human translators to get on every call with every person who needed it, but we're talking about thousands of calls a day from our applicants. That's just not possible. And in the end, the customers have to come first. They're the ones that need the benefits at that point in time, and they're the ones that we have a duty to serve and represent. [31:05](https://youtu.be/2bty2lOmjdM?t=1865) Quite frankly, I completely agree. One of the things I think that AI is super productive for is translation. I'm very clear. I share the concern for the professional translation community, but in my mind, there's enough need out there in our world as well. Would it be my preference to have more human translators for the folks we serve? There weren't enough of them that are available enough on a quick time basis for those that we serve. So that was just what I would say to that question, but I appreciate your concern, Mark. I don't know if David or Jessica have anything to say there or if we should leave it there. **Davyd Smith** [31:42](https://youtu.be/2bty2lOmjdM?t=1902) Yeah, I would ditto a lot of what you were saying, plus, and I would say 24-7. So even with your main language, if Colorado was a very high Spanish-speaking population, we're probably still not going to have 24-7, right? We don't have 24-7 in English. And I absolutely agree with the concern. I'm 100% on that range. I don't think AI needs to replace labor, especially in government, because I don't know anyone who doesn't have things that they don't get to today. And if we can get rid of some of the things that are easy, what I call mundane but important work, right? We have those things that we have to do. But when it comes to language, the 24-7 thing is the first thing that popped in my mind, Mark, when I saw your question, because I do agree with you. [32:46](https://youtu.be/2bty2lOmjdM?t=1966) And at the same time, if someone needs a service, like I said, working many English, but with 100-plus languages that we support at the state of Colorado in our constituency, I'm sure New Jersey has at least that many. It's, you know, we just don't have the labor to address all of them. But it'd be interesting to think about that in when you do do it. And when maybe you don't, like, what if you did say, well, not working hours, you know, they can actually reach a person. But after hours, they can actually still get information if they need it. You know, that might be a great compromise. And it does protect the people that do that hard work. And then you have a whole nother conversation here that we can have probably on another one about copyright, basically, right? [33:48](https://youtu.be/2bty2lOmjdM?t=2028) And how this information is actually gathered in those frontier models. Frontier models are considered like OpenAI, Google, Anthropic. They are looking at the world and gathering all that data to put it together. There are solutions to not using those as well. Not to probably not, maybe not for translation services. I don't know, but for other things. But it's a great question, actually. **Rob Asaro Angelo** [34:18](https://youtu.be/2bty2lOmjdM?t=2058) I just want to clear up also that Mark, or anyway, but we don't, we weren't using AI to have live translations or like as a chat bot or a helper on your benefits. It was really just for printed materials more than anything else. So it wasn't like it was a live interactive thing that might be having mistakes. It was only for printed materials. And quite frankly, it was our experience that far too often the results we got on printed materials from translation services was not adequate, to put it nicely. Anything you want to add here, Jess? **Jessica Lax** [34:54](https://youtu.be/2bty2lOmjdM?t=2094) No, we played a little bit with that too. Again, as you said, just with website translation, using AI as a first pass and website translation just speeds up the time. I didn't do any analysis on the time saved, but I think it's quite robust. It's very, very time consuming to translate all these pages. And some states have large translation policies, right? 12 different languages. And that can be really tricky. So it can be super helpful in that way, as long as you're making sure someone's actually reviewing it, someone in their native language is reviewing it. **Rob Asaro Angelo** [35:32](https://youtu.be/2bty2lOmjdM?t=2132) Great. Then Mark's question is, how can you audit AI's two-page summary of a 100-page document if you don't read that document? I think this is something that is generalizable for a lot of AI issues and the use cases. Quite frankly, I don't think that's a great use case, if you haven't read the document. Jessica gave the example before about regulations. And we had a large, very public regulation in our department last year. We received over 9,000 comments on it. And I use AI personally to help go through these. Not that it was any kind of formal government action of the department, but just for me to want to see what's going on there. We had a bunch that came in in a template, where people fill in what their concerns were about something. So I asked AI to go through these all and to pull out the unique responses, as opposed to the hundreds and hundreds that were exactly identical. I don't think if you're somebody who cares about their work or their work product, is going to trust AI to give you an accurate summary of a 100-page document. [36:36](https://youtu.be/2bty2lOmjdM?t=2196) I use it personally, myself speaking, I use it very often to help me pull out pieces of something I had already read, whether it be a brief in a legal matter, whether it be some kind of long report. Maybe I'm a weird, I love reading all this deep stuff in the weeds, like I'm sure many of us do, but that might be hard for us to turn that around to somebody else who doesn't have the expertise or the time to read something. So I do think that when I use it, I'm always probably using it to manipulate in some way something I've already gone through. But I don't think that all the audit procedure that Jessica was putting up there as an example before, I think it's going to be so unique to each use case that it's hard to have one generalizable way to audit, but I think it's really important stuff. **Jessica Lax** [37:27](https://youtu.be/2bty2lOmjdM?t=2247) Yeah, I'll just add, I love that question. And I think as Rob's saying, for the most part, people are really passionate about their work and really care about what people are saying and also the rules, and they know the rules back to front, they really do. And so what this is doing is just helping them with the administrative task of looking at something and classifying it in one rule or another. And that person is absolutely required to review it and make sure it got classified the right way. And so with everything, and I definitely failed to mention in this presentation, but I think the leading policy has been human in the loop. I think a lot of people are talking about other more robust versions of that now. I think, Rob, what did you say? Human in the lead. I think that's becoming more common phrasing. And 100%, you shouldn't trust AI to do those things for you. [38:29](https://youtu.be/2bty2lOmjdM?t=2309) I think of it as like an intern where they're doing work and you need to double check it, right? They're a beginner level support. **Davyd Smith** [38:46](https://youtu.be/2bty2lOmjdM?t=2326) I use this, I use summary a lot because, especially with like bills or other legal type documents that are really difficult for me to actually get, I do the summary so I understand the concept of the paper before I read it. Because before I could do that, I would read those. And right around the second or third page, my eyes glaze over and I don't know what, I don't understand what's happened. When I do the summary first and it says, oh, this bill is about environmental services. It's talking about a law around coal pollution or something like that. And now I'm like, okay, that's what this is about. Now let me read it. And I save time and I actually grasp more of it when I'm doing that. The other thing I'll point out is, I don't know how many people use notebook LM here, but that creates a very small kind of reg architecture. [39:57](https://youtu.be/2bty2lOmjdM?t=2397) And I mentioned, I think while I'm speaking, I made one for all our security policies in the state of Colorado. Now I'm an IT director, I'm responsible for all those policies, right? And there's several hundred pages and I can't know them verbatim. So I made that so I can just search those, ask a question, I get a summary and I get a link to the actual part of the document that addresses that. And I'll give you an example, our policies change on where we can actually take our equipment, like internationally or ITAR countries, things like that. And the other day, somebody said, what is the policy on that? So I put in, hey, can I take, I simply said, hey, can I take my laptop out of the country? And it came back and it said, well, there are rules and it gave me a nice summary of it, but it gave me a bookmark to go to the actual document and then read that. I would have had to remember where it was inside our Colorado information security policies, which are a list of like 38 documents and then inside of there of where to find it. [41:05](https://youtu.be/2bty2lOmjdM?t=2465) So those kind of summaries actually really helped me actually drill down to where I'm going to get. They are useless if you don't read the document, as you said, right? That summary is to get you there, or if it's something that you're not responsible for and you just want to know about it, because I'm assuming everybody here, we can produce a lot of documentation in government and there's no way you're reading it all. But if you do want to catch up with another agency or another project that you're not really responsible for, like, I'm curious about that, I should at least know a bit about it. That's the only time I think I use a summary without using the whole, reading the whole document. If it's something I'm working on now, I use it to just help me move a little faster on understanding it fully. **Jessica Lax** [41:57](https://youtu.be/2bty2lOmjdM?t=2517) And I'll just add, like, I think a really good point here is, you know, we all digest information differently and in different ways, right? Like, how do you learn? And it can really help meet you where you're at in terms of, like, how you like to learn and how you like to process information, right? David's explaining that he likes a summary before he does a deep dive. Other people might like other approaches for how they want to digest, you know, really robust and complex documents. And again, as David said, there is a plethora of that in government. So it can be really helpful in that way. **Rob Asaro Angelo** [42:35](https://youtu.be/2bty2lOmjdM?t=2555) Yeah, Dave, I think the use case on a piece of legislation is really something I use a lot because at least in New Jersey, you know, there's usually a bill summary that accompanies the bill. That could be as little as one line or two lines. And even if it's extended, it's what the sponsor wants you to believe about the bill. But there might be a bunch of other stuff stuck in there. So I use AI to sort of do that kind of thing as well. While I don't think AI will ever have the knowledge of an expert in that state about how this interacts with other laws, per se, I do think it helps bring to the fore what's really in the bill that might not be mentioned in the summary. And Jessica, I really appreciate you talking about how different people learn different ways. And, you know, I think this panel is a great example of that. You know, a little behind the curtain, Jessica didn't want to do a presentation, so I've more of a conversation. I'm a visual learner, and I think a lot of folks are as well. So I like having the slides. And even before this meeting, I had a meeting with some colleagues at the Burns Center about a project working. I'll say that. And they're going back and forth way over my head about these processes. [43:37](https://youtu.be/2bty2lOmjdM?t=2617) And I'm very excited when this meeting's over that I'm going to go to my Zoom transcript and ask AI to put a flowchart together for me for the processes they were talking about. Then for some reason, I just I can't take auto audio. It doesn't it doesn't translate to me. I'm going to have to see it to sort of understand and have it bake in my head. So I really appreciate you giving a nod to the different learning ways. That's what I think AI is super helpful with, right? And for me, also writer's block, you know, I'm thinking about it. So I need to bear down and get eight pages written on something or a blog post. I very rarely take what it gives me, but I just like put up the prompt in and just give me something I could just I start deleting and cutting and throwing words in. I think it's a real great tool for that as well. You need to put something at the paper and you just to get started on to give you some kind of base in there. Question from Owen. How do you avoid bias when utilizing a chucking process in training? I think it was, Jessica, that was you talking about chucking earlier. Yes. **Jessica Lax** [44:34](https://youtu.be/2bty2lOmjdM?t=2674) Yeah, I mean, chunking as a term has been used in different ways in this context. What I meant is if you have a large amount of data giving an LLM chunks of your data, an amount, a certain amount, and you're like, kind of just cutting off a piece, there's a big picture, cutting off a little piece, and you're saying, just look at this for now. And then you're saying, and now look at this piece. Or you might even do them completely separate prompts. Look at this. Look at this. Look at this. Look at this. And then you pull. I do not think chunking can diminish bias. Bias is built into LLMs. LLMs are trained off of thousands and thousands of millions and bajillions of our own content, especially English content, content from English countries. So the bias that's baked into our culture is in a lot of ways adopted into LLMs. Chunking cannot get rid of that. So that is, again, where we need humans to look at that, what it's producing, and really think through if there is bias in there. [45:40](https://youtu.be/2bty2lOmjdM?t=2740) But yeah, that is not something that's improved by chunking. **Davyd Smith** [45:47](https://youtu.be/2bty2lOmjdM?t=2747) Yeah, there is a question in the chat there. **Rob Asaro Angelo** [45:50](https://youtu.be/2bty2lOmjdM?t=2750) Yeah, I saw that from Sergey. When we submit documents on Google tool or LLM Studio or any other GPT, those are used down the line for training. Do we compromise our privacy data like SSN address and other data? I mean, in New Jersey, that was a very clear, bright line that we were never entering anything person identifiable into any of our AI tools. So I think it has to do with the governance and the governing body you're working for. And that might be separate from an AI tool that's specific to an agency that we're using in-house that already has all the proper protocols and security limitations on it. But that was, again, it was a clear, bright line for us. I would never put anything that wasn't even a nonpublic document, even if it didn't have PII, even into our own state internal AI. I didn't think it was appropriate or safe. I just feel that's the protection that we as government leaders, we do all these trainings. [46:53](https://youtu.be/2bty2lOmjdM?t=2813) We have to go online and do our learning links, whatever it's called in your state, that you know about data privacy and data security. And AI is no different than any other technology in that regard. How do you guys handle that in Colorado, David? **Davyd Smith** [47:07](https://youtu.be/2bty2lOmjdM?t=2827) Well, the first thing we did was when we, you know, when this hit us three years ago now, right? Ironically, I'm going back to the Colorado information security policies, right? We looked at those and said, where does this fit? We already had rules on sensitive information and how you can actually use it. We were already on Google Workplace, right? Which Gemini is just part of Google Workplace. So if you are allowed to use it and save it in your drive, a document or an Excel sheet, you're actually allowed to use it in Gemini because it has the same kind of walled garden, not training the public LLM. Well, we did have to drive home, which was already a rule in Colorado. But no one is actually allowed to get a state account or Gemini, for instance, for free, right? You can't do that with that and connect to your state information. [48:11](https://youtu.be/2bty2lOmjdM?t=2891) You shouldn't get any software, AI or other, for free and use it at work, right? Because that's where the danger comes in. You have absolutely no protection. Even if it's zero dollar, we have to go through procurement because they have to look at it and look at the rules. Are they actually training a public LLM with us, right? And basically, if it's zero dollars, they're probably doing something, right? There is no free. So, you know, I don't know what everyone's organization does, but from our standpoint, it's very simple. If you're going to use software on a state device inside a state system, you have to go through procurement and contracts, even if it's zero dollars. And then we look at that and decide, oh, yeah, no, you can't use that because we're actually looking at it. And for, you know, CJIS data and HIPAA, if there's not a BAA that specifically shows that this is a protected environment, you're not allowed to use it. [49:21](https://youtu.be/2bty2lOmjdM?t=2961) And that's, as I said, those are rules we already had in place, which is nice. A lot of the things that people worry about in AI, and this is one of them, is more document management than it is AI, right? It's how do you protect that document today? And is your, whatever AI application you're using, does it follow the same rules that you do for how you actually protect data and information? **Rob Asaro Angelo** [49:56](https://youtu.be/2bty2lOmjdM?t=2996) Anything to add there, Jessica? Same experience? **Jessica Lax** [49:59](https://youtu.be/2bty2lOmjdM?t=2999) Yeah, I mean, I think David's point is a good one. I mean, it's private data, regardless if you're giving it to an LLM or elsewhere. But 100%, I think, in general, you're not supposed to be using LLMs that you were not given permission to use by your, you know, IT department, government in general. So, if you find a tool that's AI powered that you really like, and you're paying for it on the side on your personal credit card, and now you're giving it even, you know, any data that's private, even if it's not like, you know, social security numbers, right? Like, that's, you're not supposed to be doing that. And then I would say, your state probably has more specific rules on what you can give to LLMs that they do give you. So, right, you might have copilot that you've been provided access to, or maybe some other kinds of tools. A lot of your existing software might now have AI powered tools in it. You know, every state is probably handling that differently and has probably different contracts with those providers around how their data is being used. [51:05](https://youtu.be/2bty2lOmjdM?t=3065) But generally speaking, I think, you know, making sure any private information, social security numbers, names, addresses are not included. And it is a lot of work to redact those and be super careful about that. But it's pretty critical. So. **Rob Asaro Angelo** [51:24](https://youtu.be/2bty2lOmjdM?t=3084) Well, thank you to both of you for your presentation today and for being part of the team here. I know everyone learned a lot today, as we always do in these sessions. I hope you have a great week. I'm going to throw it back to Agata for the closing housekeeping. **Jessica Lax** [51:37](https://youtu.be/2bty2lOmjdM?t=3097) Oh, just could I quickly make a quick plug in the chat. We do have an award program open right now for folks who are, you know, exploring how AI can support the civic sector. And we're celebrating everyone from, like, people who made really big wins, but also people who maybe tried something, it didn't work out. And they learned something really valuable for their, you know, community of practice on it. And they're due at the end of the week to a short 15 minute submission. So check it out. **Rob Asaro Angelo** [52:08](https://youtu.be/2bty2lOmjdM?t=3128) Thanks, Jess, and everyone can see the link there in the chat. And please don't all nominate me all at once, please. Great. Thanks, everybody. Over to Agata. **Agueda Quiroga** [52:17](https://youtu.be/2bty2lOmjdM?t=3137) Thank you so much, everybody. My final remark, sorry, is that Thursday is the final session in this series. So please do not miss. All information is in the chat. And thank you so much for your participation. And I am going to close the session. Thank you, everybody. * * * Original page: https://innovate-us.org/simpler-services-stronger-access-designing-inclusive-systems-with-ai · Licence: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org