---
id: "workshop:from-policy-to-practice-pennsylvania-s-pilot-with-chat-gpt-in-government"
type: "InnovateUS workshop"
title: "From Policy to Practice: Pennsylvania’s Pilot with ChatGPT in Government"
url: "https://innovate-us.org/from-policy-to-practice-pennsylvania-s-pilot-with-chat-gpt-in-government"
date: "2025-04-16"
duration: "60 minutes"
language: "English"
instructors: ["Harrison McRae"]
recording: "https://www.youtube.com/watch?v=mCGuWjzwfn8"
transcript: "automatic, from the recording"
source: "InnovateUS (innovate-us.org)"
licence: "CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org"
---

# From Policy to Practice: Pennsylvania’s Pilot with ChatGPT in Government

InnovateUS workshop · 16 April 2025 · 60 minutes

What happens when over 175 public employees—from different roles, backgrounds, and levels of experience—get the chance to test out ChatGPT at work? Pennsylvania’s statewide pilot program offers a rare, behind-the-scenes look at how generative AI can support real public sector challenges.

In this workshop, we’ll share key lessons from the Commonwealth’s ChatGPT Enterprise Pilot.

Whether you’re just curious or already experimenting with AI, this session offers practical insights on how to responsibly introduce generative AI in government — and what it takes to make it work for everyone.

## Learning goals

- Learn how employees across departments used ChatGPT to save time and improve their work
- Learn the biggest benefits (and barriers) to using AI in government — from trust and accuracy to privacy and adoption
- Learn real examples of how AI helped staff become innovation engines, bureaucracy hackers, and stronger communicators

## Speakers

- **Harrison McRae**, Director of Emerging Technologies, Commonwealth of Pennsylvania

Recording: https://www.youtube.com/watch?v=mCGuWjzwfn8

## Transcript

Automatic transcript of the recording: names and terms can be misheard, and the times can be a few seconds off. Each time links to that moment of the video.

**Harrison MacRae**

[0:00](https://youtu.be/mCGuWjzwfn8?t=0) In this pilot program, my name is Harrison McRae. I'm the Director of Emerging Technologies with the Commonwealth, and a large part of my role is helping the Commonwealth figure out how we're approaching generative AI and when it can be potentially the right tool to help solve some of the different challenges or problems that are in front of us at the Commonwealth. I'm going to spend most of today walking through some of the public report that we put out on this pilot program, a little bit of the context that came with it, and where that leaves us in next steps. And I believe you all can see my screen. So as I said, we'll start with just some of our general findings, some of the key use cases we saw, experiences, user types, and really importantly, what were some of the barriers that some of our folks faced while participating in this pilot program, and then how that all informs some of our next steps as we think about where, when, and how generative AI tools can be a useful tool for folks moving forward. Now, to ground this work for us, we want to go all the way back to September 2023, where the Commonwealth signed an executive order on generative AI that really tried to frame this work as how can we take a proactive as opposed to reactive approach to exploring these new technologies. For us, proactive doesn't mean trying to use AI everywhere all the time, but really being thoughtful and intentional on where there are opportunities where we can pilot, explore, take those learnings, and share them with the broader community as well as inform our teams on what are potentially the best practices and how we can best set ourselves up for success when leveraging AI or other new tools or practices.

[1:45](https://youtu.be/mCGuWjzwfn8?t=105) This executive order that we signed really had a couple key components to it that have been foundational to our work so far. The first is that it established a generative AI governing board, which has had additional oversight over all of our gen AI work here at the Commonwealth, but it's also outlined key values that we wanted to embed in all of this work. Of particular note, one of them being employee empowerment, and that's been a key aspect of this pilot where we felt that there's been a lot of questions around when, where, and how people can use generative AI tools, what are those use cases everyone are looking for, and we really wanted to start this work by asking our folks, do you find this helpful? Where do you use it? That's informed a lot of our approach and helped enable our strategy to understand where these tools can be useful, enabling, and things that our employees appreciate using in their day-to-day. In trying to answer this question, all of this led to doing a limited pilot with ChatGPT Enterprise and partnering with OpenAI. Again, this work really got started near the fall of 2023, and for several reasons, using ChatGPT Enterprise gave us the best tool to test with a collection of our folks over the following year. In total, we had about 175 people participate in some form, and again, while this was a fairly small footprint, it was really a great group of participants who came from a range of experiences and familiarity with AI to work with us over an extended period of time and help us understand, again, what is it like for folks to use GenAI tools in their day-to-day work?

[3:24](https://youtu.be/mCGuWjzwfn8?t=204) Now, I'm going to circle back to all of these as we move a little bit forward, or as we move through the presentation, but our key big kind of highlight findings that we saw throughout all of last year running this pilot with ChatGPT and our employees was first that the vast majority of our folks had a very positive experience using these tools. I love working with AI tools. I've gotten to test a number of them over the years, but didn't want to take it for granted that people would actually enjoy or find valuable some of these tools in their day-to-day, and on the whole, the vast majority of our folks did find using these tools fairly, they had a very positive experience with it. Second, our employees self-reported saving about 95 minutes per day on average on the days that they did use ChatGPT in their work. We'll note that this number is a self-reported benchmark from folks, and while that is not prescriptive or predictive across our workforce at large, it was a really helpful directional piece of information in terms of, did people perceive that they were getting some of that time efficiency value from using these tools?

[4:28](https://youtu.be/mCGuWjzwfn8?t=268) Third, ChatGPT is not a substitute for the nuance and experience employees bring to their work. We'll talk a little bit more about this as we get through the presentation, but something that was key was that when our employees were talking about what they had to do with those outputs that they received from ChatGPT, the most common types of edits they needed to make were ones where they were using their own nuance and experience to enhance that work product and make it their own or what they needed for that situation, and really underscores the need for and really underscores the need for keeping people in the process and having them hold ownership over any instances where these AI tools are contributing to work product that they're working on. Fourth, there was no singular profile successful ChatGPT user of our pilot. We've done a range of challenges and successes amongst a range of employees. We did really try to have a varying amount of different employees from different job types of different experience levels, but we definitely were not representative of the full 80,000 person workforce at the Commonwealth. That being said, it was really interesting to see that we had a range of different roles could find some of those great opportunities or setbacks, and that's something we'll discuss a little bit more as well. And last, when we looked at the types of use cases that our employees were using ChatGPT for, we were really excited to see that we felt it was commonly types of uses that really enabled people to be successful amid a lot of the kind of common nuances and challenges of public sector work in particular. In government, right, there's a lot of processes, procedures, documentation, really tech-based processes, and we found our folks were continually using ChatGPT to drive value against many of these things that are so common within government.

[6:26](https://youtu.be/mCGuWjzwfn8?t=386) So those were our big five findings, but to give you a little bit more texture of kind of the who, when, where, what of these results, so ultimately we were working with our employees in the pilot for about 12 months. We had five different cohorts of folks who were onboarded on more or less a rolling basis. These cohorts were a mix of different teams, sometimes they were aligned by agency or types of work domain areas that they were focused on. About half our group had never used an AI tool before, which to us was a really important part of this pilot. We, you know, to the disappointment of some of our AI enthusiasts across our workforce, you know, this was not a pilot focused on all the people who were the AI power users off the bat. We had a range of folks who loved AI and folks who were extremely skeptical or had never engaged with these types of tools before, and we feel like this has been really helpful for getting a more comprehensive picture of, again, some of those opportunities and challenges across our different folks who participated.

[7:29](https://youtu.be/mCGuWjzwfn8?t=449) Something that's been key throughout this pilot and, you know, what's really added value to that limited footprint we have is we did that limited footprint of, you know, just barely a couple hundred folks so that we can work hand-in-hand with them throughout the year. In total, we had just shy of 600 different feedback surveys that came in from our employees that asked a mixture of quantitative and qualitative questions, having people describe their uses of ChatGPT, what worked well, what didn't work, and this all complemented a mix of focus groups, user interviews, and other feedback pathways that we provided throughout the entire year. And lastly, while we did have a couple agencies that were representing the majority of our employees, we still had about 14 different agencies participate in this pilot, again, really adding a range of experiences and employees throughout our process. To give a sense of timeline, so this all kicked off around March of 2024, excuse me, where we onboarded that initial cohort of 56 folks who were in our Office of Administration here in the Commonwealth that is primarily our centralized IT and HR professionals, and they made up the majority of this initial cohort, and OA was the largest representative agency throughout the pilot. After we worked with these initial folks, we continued to onboard additional cohorts of agencies, employees throughout most of the summer and the fall, and continued working them throughout the first quarter of this year. We will note that while there was about 175 people who participated in total, a smaller subset of that, about 135, were our collaboration group, and these were the folks who really provided that feedback. There were a range of other folks who had either a short-term project or someone who we really wanted the perspective on engaging with AI tools, but they were not part of our more formalized feedback structured process throughout the year. This difference gave us a lot of versatility in terms of getting different people

[9:35](https://youtu.be/mCGuWjzwfn8?t=575) onboarding, testing these tools, while also being able to make sure that any of our employees who are participating in that feedback process were aware of that and had the bandwidth and ability to do so. Once we started onboarding our different pilot participants, we took a number of different practices that we experimented throughout the year to test different ways to both drive adoption, but also really set people up for success. This started with everything as simple as just live onboarding sessions. When folks joined our pilot, we would virtually convene them, make sure that we could answer any initial questions, make sure they got situated, and just understanding also what is ChatGPT. Again, some people really familiar, other folks not really familiar with different AI tools that were out there, and try to use this as a great foundational place for our different folks. OpenAI was able to help provide a range of additional trainings, and we allowed our folks to engage in a little bit of a choose-their-own adventure in terms of what skills they needed. These were a mixture of different general prompt 101 type trainings, trainings to help understand ChatGPT as a platform and some of the different features. Again, we were using their enterprise tool, which had some additional opportunities for folks to collaborate, upload context, build versions of the tools that really helped enable them specifically, and so these trainings were really beneficial for our folks. Throughout the year, we tried to also build a bit of a community of our pilot users. We had almost every Friday for the entire year, we had office hours where folks could drop in, ask questions, and just hear general ideas or updates from people in the pilot or our pilot team as well. These office hours has also continued to progress both the pilot and is definitely a model that we're hoping to continue to use to build community of people who are trying to learn and explore what these different types of AI tools might mean for them.

[11:22](https://youtu.be/mCGuWjzwfn8?t=682) We also, in addition to office hours, had more targeted sessions we do with employees where some of these were very AI-specific, but some were also much more about general just problem scoping and something that we saw commonly was people were given these really powerful, versatile generative AI tools, and it would kind of invite questions of where, when, and how can I use this? And we found that doing problem scoping sessions were a great way for folks to kind of better understand the when, where, how, and give them some brainstorming ideas for what they could start to apply these types of tools to, which we always like to call it just because, again, while it was related to AI, it was actually more of kind of a different type of enablement session than a standard prompt training or AI targeted training. And then lastly, throughout the year, we had a range of different kind of general AI learning opportunities. This is something we found was both requested from our pilot participants, but really helped build confidence for users throughout the year. You know, it's really hard right now to, I would say, kind of go through your day-to-day and not hear AI discussed positively or negatively in kind of different spheres.

[12:32](https://youtu.be/mCGuWjzwfn8?t=752) You might be watching TV and see tons of software AI commercials. You know, maybe you have children at school and they come home talking about all these things they started doing with AI tools and how it may or may not be showing up in their classroom. You know, there's a lot of news articles talking about, you know, some of the amazing things that I can do with, say, medicine and drug discovery, or also talking about some of the challenges these organizations have faced in terms of how they collect their content and the energy impact of AI. All of this provided a lot of external information for our pilot participants, and we felt it was helpful to offer different training opportunities and resources for folks just to kind of understand what was happening with AI broadly, grounds themselves in some of those conversations. And throughout most of the back half of the pilot, I had monthly seminar series where we would crowdsource different questions on AI topics, folks in the pilot had, and would provide 30 to 45 minute overviews of those topics just to, you know, again, ground people and provide them with some of the informational context to help drive confidence when they're using these tools and understanding a little bit more about the AI movement we're in right now. In addition to all of these different training and onboarding pieces that we used, a key part of this work for us was, again, not just providing a handful of folks AI tools and kind of letting them run free, it was about working hand with hand with them throughout the year and providing a lot of opportunities for them to provide their feedback. You know, we definitely acknowledge that the footprint of the pilot was fairly limited in scope, but we did this so that we could really have that more direct dialogue with our folks. I mentioned earlier that we had just shy of 600 feedback surveys submitted by our pilot participants throughout the year, but we also did a lot of targeted focus groups, user interviews to get at some of the texture of what people were discussing when they were using these AI tools.

[14:30](https://youtu.be/mCGuWjzwfn8?t=870) This has been also facilitated by the Commonwealth Office of Digital Experience, Code PA. The UX team has been a huge value add as we've collaborated with them to really lead this research and help build reports and insights that we're now using to take a more employee, user-informed approach as we consider when, where, and how we might adopt additional GNI tools for the Commonwealth. So what were some of the findings that we saw? I spoke at a very high level of kind of our top five, but we'll touch on a few others and go in a bit more detail for those five I called out. And most importantly, you know, something that I think everyone always asks about when it comes to AI and AI tools is what are the use cases? What were people using these types of tools for? And this is something where we really saw our folks using it for a range of different ideas and topics. And ultimately that's something that I really have enjoyed in this work is, you know, how can we help create opportunities to allow our different teams, employees, folks who might be interacting with the systems, be the ones who can surface ideas, opportunities, and call out some of the things that we should maybe have on our radar as potential problem areas or challenges in the future.

[15:51](https://youtu.be/mCGuWjzwfn8?t=951) And so we did a number of different, we included in our research a number of different pathways for folks to talk about their use cases and what worked well and didn't. And when we looked through all of that, there were three kind of primary personas that we saw our public sector employees embodying when they use Chat2BT in their work. And those public personas, which I'll speak to in just a second, were made up of a couple of key different topic areas. Topic areas and task areas. This included, you know, research, idea, exploration, different types of just general text work, writing, text generation, drafting emails, outline project descriptions, a lot of summarization of policy, large documents. We have coding, Excel, other types of kind of tech enabling work. in terms of not being able to do that. I will say that this coding one is probably a little bit of an underestimate It's probably a little bit of an underestimate in terms of the value that folks saw from it.

[16:53](https://youtu.be/mCGuWjzwfn8?t=1013) While pretty much everyone in our pilot used Chat2BT for some sort of text work, not everyone in our pilot was doing work streams that engage with coding or some of those more technical practices. And so our folks who that was relevant to, almost all of them would use Chat2BT for this work, but it just wasn't as relevant for all of our pilot participants. So that is the one that we do like to call out as a little bit probably of an under kind of estimate in terms of its importance. And again, across all of these different types of tasks we saw these three personas emerge that people reported, of our public sector folks, which was people really being enabled as innovation engines, bureaucracy hackers, and strategic communicators. I think in government in particular, there's a lot of routine processes, structures, fairly or unfairly, people are given very narrow lanes in which they have to operate. And we saw people continually using Chat2BT to come up with new ideas on how to solve problems, drive value, and be creative in their work. This could be something as simple as someone just getting a jumpstart on a large memo that they had to work on where it was really helping them kind of move through some of that writer's block to injecting kind of little bursts of creativity in terms of how we would name programs or how we might approach certain different types of problems in a new way given some of the constraints or guardrails that are already there.

[18:10](https://youtu.be/mCGuWjzwfn8?t=1090) For our bureaucracy hackers, this is something where we saw a lot of value being gained for our folks. There's so many different opportunities within government work where people do a really great job to move through procedures and steps, take information from one place and make sure it then satisfies requirements for different types of documents that need to be submitted in other places. Really, to enable to drive value in their work, have to follow a lot of text-based processes that have a lot of usual requirements. We saw people continually using ChatGPT to both improve the quality as well as the efficiency moving through these processes, whether it was in one hop capacities or really building some of their workflows around how they could move through the bureaucracy of their work and continue to better drive value and results for our end users and constituents. And lastly, strategic communicators. This was one where I think no matter what work anyone does, there's always opportunities to be better communicators.

[19:12](https://youtu.be/mCGuWjzwfn8?t=1152) And in government, there's definitely an additional nuance of, we work with a wide range of internal and external stakeholders. And ChatGPT was excellent at allowing our folks to get that extra set of editing eyes, help turn language that was meant for one audience to be much more applicable to another audience, really building just communications information that is more suited for the different stakeholders that we have to work with in a time-effective way. These slides are the abridged version of some of our public report. We have a couple of different use cases that we called out in that version. But to give you one example of this, we always like to highlight some of the great work done by one of our HR teams that were part of our pilot. And we had a couple of different folks who were on this team, who ranged kind of from the AI excited to very AI skeptical. And they would tell me about how AI skeptical they were on the onset. And for this group, one of the key functions that they helped work on is we have a lot of very clear job classifications within the Commonwealth of what types of positions can exist at different types of classifications.

[20:21](https://youtu.be/mCGuWjzwfn8?t=1221) When any team is going through the hiring process, there's needs to be able to make sure that any of the positions descriptions we're working on align and are within kind of the correct form and function for that different position description and classification as well. And there's a lot of manual work and matching that happens here. A lot of reviewing of what people have submitted and aligning it to this database of classifications. And our team that works on this work was able to re-engineer some of their workflow where they're able to much more quickly analyze address and align different requests for openings and positions that people had with the correct classifications that needed to match. This is really great work that's enabled this team to spend a lot more of their time focusing on some of the strategic work, the nuance area, and continue to serve their different agencies that they work with much better. It's been really exciting to talk to them and hear how they've done a lot of A-B testing in terms of how some of these AI enabled workflows would either get things right or wrong compared to what their team would do in a vacuum.

[21:31](https://youtu.be/mCGuWjzwfn8?t=1291) Some of the things that they need to be on the lookout in terms of how they could have this be a tool that helped enable them as opposed to providing inaccuracies or other outputs from ChatGPT that might cause some issues. And ultimately, we really liked this quote from one of these participants who I'll just read here, which was, for me, using GenAI is about getting closer to a solution faster. I can use it to sift through lots of information, to make suggestions, and ask it why if I disagree. From there, I can use my own expertise and consult with others to validate and make a decision. Embodying this quote is really a lot of what we tried to kind of preach when we were enabling folks and getting them on board with ChatGPT, which is ultimately they own their own work product. This is a tool that can help enable them, but they're responsible for the ones who are making sure that that work product aligns with the work they're doing, the objective they're trying to accomplish. And in some cases, it can be really helpful, But in this case, in work like this, in others, not so much.

[22:32](https://youtu.be/mCGuWjzwfn8?t=1352) it was really helpful in them getting through the initial part of that work more quickly, and then they were able to drive that kind of last mile value and use their expertise in that capacity. Again, we have a couple more of these different specific use cases in the full report, which is available on our AI Pennsylvania website, and I'll share the link at the end. So in addition to use cases, we had a couple of different types of user profiles and user experiences that were represented throughout our pilot. I'll say to start, and this was one of our kind of key findings we shared in the beginning, but people on the whole felt really positive about using these tools. When I would talk to people about some of our pilot ending, a common note I'd hear from folks was that people were worried these tools would get taken away, they really enjoyed some of the value they'd driven. And it's helpful for us to know that not only did they find some efficiency value, maybe they felt it helped their work product, but they also felt good, they felt the experience of using these tools was positive as well.

[23:39](https://youtu.be/mCGuWjzwfn8?t=1419) An additional set of questions that we asked our folks throughout the pilot was to discuss a little bit more about what it was like when they were using ChatGPT, and there was a use case that was kind of the most helpful to them in those past couple of weeks, not just the generic or average, but when they were using ChatGPT, really honing in, what was it at its best for them? And this was interesting for us because when folks talked about ChatGPT in those most helpful instances, compared to the rest of the results, on average, they felt that it did more to improve their final work product, reduce some of the complexity of their task, required review, but less review than some other types of uses that they've tried, on average cut their final, that task they were working on, or not, excuse me, on average, but on the majority of the time, the task that they were using where it was the most helpful task, it would cut completing it down to under 15 minutes. And all of this is, I think, relevant to note that they saw these extra benefits because it speaks to some of the problem scoping, adoption type work that's important to compliment some of the AI tools to really help folks understand kind of what are some of those anchor use cases I can find in my day to day to help drive value in my work.

[24:53](https://youtu.be/mCGuWjzwfn8?t=1493) And this is something we definitely saw for the folks who found those anchor use cases, there was extra value to be had. To narrow in on one of these results in particular, I'm gonna talk about the task complexity piece. So this was an interesting result we saw where, again, when people talked about this most helpful use case, we asked them to say, what was the complexity, perceived complexity of this task for them before using chat GPT, and then after using chat GPT. And you'll see here, not only did they find some of the efficiency and final deliverable value, but they also felt that the actual complexity of this work moved to be on average, moderately complex to a plurality of responses saying that it was simple. And to this us, this was a key call out because it spoke a little bit more to, again, there was value to be had, but also the work itself became more manageable for these folks and they were able to move through that task in a less complex way.

[25:54](https://youtu.be/mCGuWjzwfn8?t=1554) And that was part of what was the value add for them. Another one of our top five takeaways. So we talked about how people self-reported saving about 95 minutes per day on the days they most recently used chat GPT. I will say, this is not a predictive, again, benchmark. We wouldn't expect every employee to save this amount of time. It definitely is gonna vary domain function, the type of person, the type of tool that they're using. And also not everyone was using chat GPT every day. For some folks, maybe they got into a routine where they were using it for a specific task or ad hoc, but for other folks, they were using it all the time. So some of those results varied as well. Ultimately, what we found really valuable as this number, again, it being a self-reported time saving and time investment was on the whole, did our folks feel like they were getting some time efficiency, felt like they were saving time in their day. And there was a fairly consensus, yes. Again, varied for different folks, but we found this as a really helpful affirmation as we started to explore these tools that there was this type of value that people felt.

[27:00](https://youtu.be/mCGuWjzwfn8?t=1620) It has been a great launching point as we continue to explore more on what some of these efficiency gains might look like in specific spaces. Another one of our top five takeaways was this idea that human nuance is essential. So whenever people were working with Chattopadhyay, again, we really tried to reinforce for them that this is a tool that they can use, that they have to own their work product. An example we'd like to give folks is just as it's not, it's a socially or professionally acceptable to copy and paste something from Wikipedia, put it in a memo, and then if there's an issue with what you put in that memo, kind of say, well, it was on Wikipedia, it's fine. There's a similar kind of muscle memory there of Chattopadhyay can be helpful in getting to your answers, your resources, your workflow, but you need to own that work product and kind of be able to build out whatever that review process is that allows you to move with Chattopadhyay confidently. And when people were doing that process, their own expertise and nuance to the work was essential in them being able to provide output that allowed them to add value.

[28:09](https://youtu.be/mCGuWjzwfn8?t=1689) I spoke earlier about how we talked about asking people about their most helpful use cases. We also asked people to flag when they were using Chattopadhyay and it was a lot less helpful for them in their work. And what was interesting was across both those times when people kind of saw Chattopadhyay at its best for them and at its worst for them, the most common type of correction in both those instances, they needed to make to outputs was to include additional nuance from their perspective. Yes, there were inaccuracies. Yes, there were hallucinations. Yes, there were biases, but ultimately people would self-report by far the most common types of corrections they were making again were these ones where they were adding their own expertise nuance to that output and reinforcing this idea that we feel like these can be enabling tools for folks, but ultimately that public sector, Pennsylvania perspective is really essential in this work.

[29:10](https://youtu.be/mCGuWjzwfn8?t=1750) So to move a little bit more into some of our user types in addition to user experiences, we talked about those personas earlier, but then how did this translate into more of what we saw in terms of the types of uses of our folks? Broadly, we would say that there's about, there were four kind of key user groups. We had our sporadic general users, our specific consistent users, our super users, and then our non-users. Our sporadic general users, again, really kind of the predominant group of this bunch. And they would use ChatTBT a little bit more ad hoc as needed when an idea came to them for how they could apply it. And I think there's definitely more opportunity with some of this group in terms of how can we spark ideas, give them examples, and allow them to continue to use it more and more regularly to get kind of the most out of it. We also had a lot of folks who were these specific consistent users where pretty quickly or over time as they got ramped with ChatGPT, they identified a handful of different tasks that really added value. They'd work ChatGPT into that workflow, but then wouldn't really go exploring for more use cases beyond that.

[30:17](https://youtu.be/mCGuWjzwfn8?t=1817) Our super users, people who really were excited about applying it to kind of anything they could, sometimes it wouldn't work out and they wouldn't use it again, but oftentimes they would discover new use cases and continue to integrate it into their work. And then for non-users, we will call out, this is probably a little bit of an undercount. We did have different degrees of survey drop off by different participants throughout the pilot. But even so, when we look at our total numbers, still a vast minority of the folks were these non-users who for one reason or another, using ChatGPT just really didn't stick in their day to day in the pilot. And we have some additional information for these folks specifically, but broadly, what were some of the adoption barriers we saw throughout the pilot. Just to share a couple of quotes from some of our folks throughout the pilot. And I will add here one thing about our full report, which again is available online. We do sprinkle a few more of those quotes and testimonies from our folks throughout. That to me is one of my favorite parts of this report where you may or may not find it fun to hear from me about the pilot, but getting to hear directly from some of our participants through their words is a highlight.

[31:45](https://youtu.be/mCGuWjzwfn8?t=1905) While this identifies, I think some of the sentiments from those users specifically, definitely check out our full report to see more of these types of conveyed experiences. So as we're moving forward with ChatGPT, again, we saw people of a range of different levels of success. And what were some of the key barriers we saw that held people back in different ways? So first I would say it was inaccuracies. And while I showed that slide earlier when people talked about some of the corrections to prompts they made, inaccuracy was definitely noticeable as a type of correction, not the most predominant one. So why was it a key barrier adoption? And this really comes down to, there is a range of tasks where either because of the skills of the ChatGPT user, how ChatGPT was being used itself, limitations of the model and their platform, or kind of some combination of all of the above, a user would use one of the tools and an error that would come up or be identified when they were doing the review at the end, that kind of spoiled the whole work product for them. An example I'll give is say that you were asking ChatGPT to help analyze an Excel document, and maybe the Excel that you've provided is messy, the data is not that clean, maybe the columns aren't, for some reason, it's not the most kind of analysis-ready Excel document you could provide. You also maybe don't use the best prompt when asking for it to be analyzed, and maybe this was one of the versions of ChatGPT earlier on, where they haven't continued to enhance some of their data analysis functionality. You put that in, you ask for some key takeaways, you're reviewing the outputs, and you notice that one of the calculations or part of what some of the information represented in one of the cells is inaccurate. I think someone might need to mute, but you might notice that one of the cells there

[33:46](https://youtu.be/mCGuWjzwfn8?t=2026) has an inaccuracy in it. You know, if you were using ChatGPT to help analyze the Excel document, you might have done that because it is hundreds or thousands of rows, and now that you've found one error, it isn't like an error in an email draft where it's pretty easy for you to comprehensively review or not, or correct or not. You now have to go back from scratch and redo a lot of this work, and for some of these types of work streams, it was that type of inaccuracy that really kind of spoiled that type of use case, and then sometimes require a duplication of work, or if someone didn't have some of that capacity to really experiment, was a little bit of a non-starter for them. Second was we saw a lot of different ways in which habit forming was key for this. You know, going to a new site, a new platform like ChatGPT, using it for different types of work streams, and building up some of that muscle was definitely something that some folks kind of took to right away, and other folks, it was definitely something that they needed to remember to do. I know for myself, as I've used these tools, kind of that is a muscle memory piece I've struggled with, where getting the habit for certain types of domains or tasks, remembering to use them when I've already identified it to be useful or helpful in that work stream.

[34:59](https://youtu.be/mCGuWjzwfn8?t=2099) A third, a lack of learning time. You know, we have a lot of folks in the commonwealth who are extremely busy. They already have a lot on their plate, and because of that, these tools can offer a lot of value, but it can often cause issues in terms of having some of that upfront time to be able to invest into learning how to prompt or get oriented with a new tool, and some of that initial learning upfront time was definitely a barrier for some folks. There also was just this steep learning curve of the idea of prompting and engaging with these tools with a new experience for our folks, and as with any type of new learning content or experience, some folks took to it fairly quickly. Some folks still have struggled with, you know, what it means to prompt, how do I apply these tools to a certain set of problems, and lastly, I think we had a really healthy and skeptical kind of first reaction from our folks in terms of their data privacy, security, how they could use these tools responsibly, and while we felt comfortable putting these tools in front of our folks and setting them up for success as they use Chat2BT in this pilot, it was actually really helpful to see some of those concerns and worries from our folks, and in some cases, those concerns also helped folks back in terms of kind of not really diving into Chat2BT to its fullest. We'd outline some specific examples of these barriers in our final report, but all of them have helped inform some of how we're approaching what it could look like to continue to use Chat2BT or any kind of Gen-AI tools in the future, and for us, there's a couple key steps in this process that we think are some of the structure that we would approach for anyone considered in scaling generative AI use, and the first is, you know, creating those clear pathways for usage, making, excuse me, making sure people feel confident with the type of tool that they're using, and they have whatever those trainings or communities are to set them up for success, whether it's, you know, fairly common generative AI tools, something that your organization might be promoting internally, or is, you know, just an

[37:03](https://youtu.be/mCGuWjzwfn8?t=2223) AI feature that exists in an existing software suite you might use. Once people are set up for success and understand what that usage pathway might be, being able to drive value and kind of identify those key personas, the key areas where you think that there are some initial value folks, and that kind of really healthy kind of starting ground for people in terms of where, when, and how they can use these tools. From there, being able to set our folks up for success is essential, and we talked about some of the different training and onboarding processes we experimented with, and we think really building up some of those communities of practice, having a range of different trainings, very structured ones, but also spaces for people to kind of choose their own adventure, and really go down the rabbit hole if they are looking to use these tools more effectively is helpful. And lastly, investigating impact, you know, both from trying to reinforce that muscle for our Gen AI users themselves to review outputs, validate results, but also really be able to understand the when, where, how people have been able to find value in these tools, make those playbooks, and then be able to scale them to other teams that might have common uses is essential to being able to understanding if we're going to use generative AI tools, are they actually helping us accomplish the goals that we're looking for? You know, ultimately, there is a range of things that can be done with AI tools. There's no kind of shortage of webinars and discussions about the top, you know, five or ten AI use cases. I think our approach is really about what are the problems we're trying to solve, and when, where, and how can AI potentially be a tool that's helpful?

[38:39](https://youtu.be/mCGuWjzwfn8?t=2319) And in guiding that conversation, we want to put these tools in the hands of our folks, give them the tools to be successful, investigate that impact to understand it's, you know, real and actually driving the value they want to see, and then packaging that in ways that can help drive value for a range of our folks in the future. For all of this and more, again, I'm a broken record, but would encourage you all to check out our public report, which has, it's a little bit more detail on a few of these different pieces, but with that, really appreciate you all listening in. Happy to answer questions as time allows, and you can go to oa.pa.gov.ai to access our Commonwealth AI website, and for any, you know, specific questions on this report or scaling some of this, feel free to reach out to artificialintelligencefda.gov. If you're reaching out with a vendor solicitation, please do not do so. We would love to share our resources and trainings, but that is not the best way from a procurement standpoint to reach out to the Commonwealth. And with that, I will turn it over to you, Jess. Awesome. Well, I'm gonna stop the recording.

* * *

Original page: https://innovate-us.org/from-policy-to-practice-pennsylvania-s-pilot-with-chat-gpt-in-government · Licence: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org