---
id: "workshop:a-state-based-approach-generative-artificial-intelligence-learning-from-new-jersey"
type: "InnovateUS workshop"
title: "A State-Based Approach Generative Artificial Intelligence: Learning from New Jersey"
url: "https://innovate-us.org/a-state-based-approach-generative-artificial-intelligence-learning-from-new-jersey"
date: "2024-10-01"
duration: "60 minutes"
language: "English"
instructors: ["Chris Rein"]
recording: "https://www.youtube.com/watch?v=LhfXfjUh3D8"
transcript: "automatic, from the recording"
source: "InnovateUS (innovate-us.org)"
licence: "CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org"
---

# A State-Based Approach Generative Artificial Intelligence: Learning from New Jersey

InnovateUS workshop · 1 October 2024 · 60 minutes

- Hear an overview of what the State of New Jersey is doing with respect to generative artificial intelligence (AI)
- Learn from case studies of three projects that are currently underway
- Discuss generative AI and return on investment for states, now that genAI has permeated many products and services

## Learning goals

- Hear an overview of what the State of New Jersey is doing with respect to generative artificial intelligence (AI)
- Learn from case studies of three projects that are currently underway
- Discuss generative AI and return on investment for states, now that genAI has permeated many products and services

## Speakers

- **Chris Rein**, Chief Technology Officer for the State of New Jersey

Recording: https://www.youtube.com/watch?v=LhfXfjUh3D8

## Transcript

Automatic transcript of the recording: names and terms can be misheard, and the times can be a few seconds off. Each time links to that moment of the video. The transcript alone: https://innovate-us.org/a-state-based-approach-generative-artificial-intelligence-learning-from-new-jersey.txt (plain text) and https://innovate-us.org/a-state-based-approach-generative-artificial-intelligence-learning-from-new-jersey.json (every segment with its start and end second).

**Chris Rein**

[0:00](https://youtu.be/LhfXfjUh3D8?t=0) All right, I will go to presentation mode. And we're all set. Good afternoon, everyone. Well, good afternoon, if it's afternoon. I know we have folks in quite a number of time zones, so it's good to be with you. A couple months ago, I believe, I joined a similar workshop with Hannah, who is Agia's predecessor, and it's nice to be back with Innovate US. They've done some terrific work with the state of New Jersey, and, you know, if I can just give back a little bit, it's my honor. Today we're going to talk about a couple of kind of different perspectives involving artificial intelligence. So the last time that I spoke with this team, we talked a little bit about the formation of Governor Murphy's Artificial Intelligence Task Force.

[1:06](https://youtu.be/LhfXfjUh3D8?t=66) So we kind of covered that. It's underway right now. And we also worked with the Innovate US team and put together with our chief innovation officer and our AI strategist a really robust training course that we made available for all state, some 65,000 employees. So those have been covered, and I'm going to jump into a few use cases that we've been working on within our various state agencies. And then I'd like to take a different, a slightly different slant on things and have an interactive discussion on what the team feels like it's realizing or seeing within the area of return on investment for Gen AI. So without further ado, you know, when I speak of these kind of four high-level items, right, in the midst of all that, we've been doing a lot as well, right? We've been setting up our AI policies, looking at different platforms, technologies.

[2:10](https://youtu.be/LhfXfjUh3D8?t=130) We're looking at our data. That'll be a theme that you're going to see coming up in a little bit. And then just understanding what kind of an AI inventory should we be taking? How much information is it prudent to collect about these projects? We don't want to slow projects down, but we want to make sure that we have a good accounting of what's going on. So a lot of activities have been going on between then and now. Coming up soon, we're rounding the bend in the fourth corner, so to speak, and we're nearing the time that we'll be delivering our artificial intelligence report to Governor Murphy. We've got different working groups. We've focused on, as you can see here, several different aspects of artificial intelligence. And that'll be after we do the final reviews with governor's office, with council, and with our policy team, we'll be sharing that with Governor Murphy. We've worked quite a bit on that, and we're proud to deliver a robust report to the governor.

[3:17](https://youtu.be/LhfXfjUh3D8?t=197) So I'm going to jump in now. I'm going to talk about four little case studies where we obviously don't have the time to dive in very, very deeply with any of them, but I think it'll give you a sense of what's going on. Our New Jersey Motor Vehicle Commission, they are doing some really good work with using AI to augment their contact center. They want to improve the customer's experience with questions, with problems, with guidance. So they had recently moved to another cloud-based call center support system, and they're expanding that investment. They're learning right now, as you'll see several times, that really good data, they can work really hard on the algorithms, on the AI platforms and the APIs that they have, but without really good data, it's really going to be a limiting factor.

[4:19](https://youtu.be/LhfXfjUh3D8?t=259) It's really that they took this on themselves. They had no additional staff, so they really worked hard and they tried to make some of their key, very innovative, very experienced staff available, and it's really nice to see what Motor Vehicle have done. My team have helped them out with some infrastructure that they needed, and that collaboration has really been exciting to see. So that's what Motor Vehicle is doing. Initially, it's going to be internally used by Motor Vehicle staff, contact center staff, but they do have an eye toward next phases when they plan on making some of this AI-infused technology available directly to the motoring public. Next, we're going to jump into another artificial intelligence use case. Our Division of Taxation is working collaboratively with our Office of Innovation, with my office, the Office of Information Technology.

[5:23](https://youtu.be/LhfXfjUh3D8?t=323) We have a vendor that worked with us, Boston Consulting Group, as well as their own call center staff. It has a lot of similarities in terms of its use case is intended to be used by a subset of call center staff. What I think was interesting about this choice was they took a subset that were using this AI tool and a subset that were not, and they're trying to kind of develop some metrics and trying to see what kind of trends those that use the AI and those that are not using it. So that's going to be exciting to see how that pans out. As you can see down the bottom of the screen there, they've made good use of this AI over the past several weeks, and they've gotten some really good focused feedback. As we learn with a new technology like artificial intelligence, the feedback that we get from the customers, from the users of this technology is so important because we all know we're at a rather early stage of it.

[6:32](https://youtu.be/LhfXfjUh3D8?t=392) In this particular case, this proof of concept was not built on a large-scale, robust platform. It was specifically designed for a small user group. We're learning that some of the responses, especially when they're a little bit more complex with more tokens and a more complex response, the response time will need to be made faster. And when we move such an operation like this into our production cloud environment, we'll have all the computing resources that we need. So this has been really exciting. A lot of collaboration there, a lot of joint learning, and that's been exciting. One of the things that we also did, jumping to a third use case, was we wanted to provide in the training that we work with Innovate US on, we wanted to provide like a pointer to how can you now use this? And we didn't want to point all of our many, many thousands of state workers to just one of the open, available, free services.

[7:43](https://youtu.be/LhfXfjUh3D8?t=463) Even though they're there, we wanted to give them an environment that they could work more securely in. So we built and stood up in our environment. In fact, we actually stood up two of them. One's in production now for several months, and the other one will be coming online either the end of this week or the beginning of next week. And we chose different cloud environments. We chose different language models. In fact, one of our tools, we're actually going to give the users of this a choice so they can see how that same prompt, how the same question yielded different responses in this language model versus that language model. So they may use an anthropic model. They may use one of the many, many choices that we're making available to them. Not only are they going to be able to see the difference in the responses, but they're also going to be able to see a cost difference. From these cloud service providers, they charge different rates for the use of the language models.

[8:49](https://youtu.be/LhfXfjUh3D8?t=529) So we're actually going to be able to show if this query were run in language model A, it may cost one-tenth of a cent. And if it were run, the same query were run in this model, maybe it costs two cents because it's a more expensive model with more rich features. So we're making that assistant available later on this week or next week, and we're excited to see. Our Office of Innovation, they have really some tremendous forward thinking and fast learning folks that work with the state of New Jersey and work within the state. And collaborating with those folks have been tremendous. We've learned much. We've been able to provide them infrastructure, and it's really been an exciting experience from my perspective. And they came up with like that initial one, they spun it up really fast and they did a great job. So we were able to put that out there.

[9:49](https://youtu.be/LhfXfjUh3D8?t=589) At the same time, we put together all of our training courses. So the many people in the state of New Jersey, by the thousands that have taken this training now have a secure environment that they can work in thanks to our innovation team and Office of Information Technology. Okay, last but not least, and this is important because I think it showed another concept. There's an entity, it's a commission within the state of New Jersey, it's called the Public Employment Relations Commission. And they deal with matters where maybe an employee has a matter that they want to settle or they feel like there's an interpretation of rules and regulations. So that's the function they perform. And they're not a large entity, right? The PERC commission is small and they have a small, but high energy, very focused set of IT professionals there.

[10:51](https://youtu.be/LhfXfjUh3D8?t=651) And they dove in and they sought to set up a tool and they wanted to see how helpful this tool was. So they wanted to be able to make it easier for their caseworkers, for their attorneys to do research on precedents on prior cases. So sure enough, they went in and they trained a model and they pointed it at their data. I'll point out that they recognized that they had some really older legacy data. It was in an older case management and case repository, a database. It didn't lend itself well to enriching or tagging a lot of it or adding a lot of metadata. And as they brought this online internally for a small subset of staff, they actually after using it for a while, they actually saw that the amount of, and they also looked at the costs to operate this. And they saw that when they look at the return on investment, that they could use it, but it wasn't really yielding them an order of magnitude increase in results.

[12:00](https://youtu.be/LhfXfjUh3D8?t=720) So whereas they parked all of that, they saved all of their work, all of their code, all of their training materials, and they have it there, they're going to reevaluate. But for right now, they actually made a learning decision and they say, look, the ROI for this specific use case, given the constraints on the data that they had, just wasn't realizing it and wasn't fitting into their budget. So I think this was an excellent learning opportunity. And I know the CIO of that agency really did a great job and he worked very, very closely with his constituents on the business side. So there's a lot to learn about that because, you know, as we deploy AI infused solutions, it's a different type of a culture. We kind of have to be accepting of an agile mindset and learning and improving. The AI is not going to get every answer correct, it's just not.

[13:01](https://youtu.be/LhfXfjUh3D8?t=781) So we have to adopt that culture that allows us to improve in that regard. So I was really fortunate to have spent time with their CIO on this. All right, so a couple of things. You've heard me make references to the data. And yes, I wanted to depict a learning organization. So sure enough, I asked our AI that we have to draw me a picture that depicts and connotates a learning environment. So this photo here, this little icon is brought to you by our artificial intelligence. And it is all about the data. Your AI solutions, I think that the industry on the private sector and the public sector is learning that without really good data and preparing that, it's just you're going to be limited in the effectiveness of your results.

[14:01](https://youtu.be/LhfXfjUh3D8?t=841) So all about the data. I was thinking as I was putting this presentation together, do I want to say that our three learnings are it's all about the data, it's all about the data, it's all about the data. But there are some other things. Sometimes, and I think that public employee commission, their learnings speak to us and say, AI is really fascinating. As a technology focused individual, my whole career, fascinating to me and many others. But those of us that are in a government setting, our obligation is providing good government services, not just to provide cool technology necessarily. So let's make sure that we find a business problem that we can solve with a solution It does involve AI.

[15:01](https://youtu.be/LhfXfjUh3D8?t=901) You know, that's kind of a cliche, but I think it's been borne out with our findings. And that culture of learning, that's going to be hard because it's going to take time. In state government or in all government, right, we see it as our obligation to be correct all the time, and we want to. We develop our systems, we perform quality assurance for our systems, and we want to make all the answers as correct 100% of the time as we can. We know that in many cases with artificial intelligence, even with good effective data curation, there are going to be times when we're going to have to take back a learning and correct either the model or the algorithm. And so we're going to have to encourage that culture of learning with our technology folks and our business folks on both sides so that we can, you know, really provide a good, rich, you know, set of solutions that do make government better, that improve the resident experience of government.

[16:08](https://youtu.be/LhfXfjUh3D8?t=968) Okay, so we talked about the second half of this, and in doing some research, I found some interesting, I think, items that I'd like to share. You know, we made use of some of the research resources that we have within the state of New Jersey. Some of the research we have is provided with our contract with Gartner. They have done some really good research, and I pulled some of the research, and I got their permission to share on this, so, you know, thanks to some of our colleagues. So we look at these three statements that I just pulled out of some key findings, and each one of these very clearly mirrors some of the trends that we see in the industry, right, with our vendor partners, with the vendor community. Really a lot of enthusiasm, a lot of hype on the use of Gen AI, right? Top priority, right?

[17:10](https://youtu.be/LhfXfjUh3D8?t=1030) More than two-thirds of the executives clearly see it out, the benefits outweighing the risks. So it's interesting to see another whole set of perspectives, and this I literally found by just doing some simple, simple searches across the journals, across the internet, that the ROI has not necessarily always proven to be quick, to be returning the ROI that either private sector investments or even some governments are seeing. So I wanted to take some, you know, and you can see some of these different, just look at the, just look at the journals here, pretty credible journals, and pretty, you know, well-respected industry professionals that are doing this research, right? Goldman Sachs, right? They're all about return on investment and investments in this market.

[18:11](https://youtu.be/LhfXfjUh3D8?t=1091) They've done a lot of research in this regard, and not all of it's positive, right? They've come up with some question marks. So in some cases, they're tempering their, some of their enthusiasm for investments. So what discussion on any type of a new trend or technology would be complete without, you know, some depiction of, you know, we've all recognized this as, you know, the Gartner Hype Cycle, where, you know, it depicts almost any technology, almost any new capability, right? Whereas, you know, it starts with the innovation, it jumps up, and enthusiasm is high, and right, there's all the buzz on this new technology. And then, as you see later on down the cycle, it falls into what, you know, again, this is the Gartner term of a trough of disillusionment when, you know, the promises may not be realized, and it's not yet a material, or it's not yet become a habit by employees, by workers, and so forth, right?

[19:18](https://youtu.be/LhfXfjUh3D8?t=1158) Sometimes when the ROI is not there, right, the shareholders or investors, right, can really start to put pressure on it. So and in some cases, spent a year or more than a year on pilots, but there may not be a real, real corporate or government ambition or strategy. So you know, I ask the question, is it, you know, are we still here with all of the excitement and the hype? Or have we gone down into the trough? Or probably more likely, we're somewhere in the middle, right, where some projects are realizing, you know, some investment for sure. You know, we know that vendor community has many, many products that are already showing demonstrated use and benefits of artificial intelligence. So you know, I just pose that for, again, for thought. And I want to gain your thoughts on this. And so, as we talked about at the beginning with Aguiar, we're going to jump into that.

[20:20](https://youtu.be/LhfXfjUh3D8?t=1220) But I think there's enough time that before we turn off the recording, I have just a couple additional perspectives that I found really interesting in doing some of the research. So one of the areas of research that Gartner did was to try to depict on the, in this case, on the x-axis, on the horizontal axis, they took a look at the experience of the worker that was going to yield the benefits of this AI investment, right? On the y-axis, on the vertical, right? How complex is the task that the artificial intelligence itself? So in other words, how complex is the algorithm? How rich are the data sources? How complex of a task is it going to complete? So they depict somewhere, and this is notional, right? It's indicative. You'll see no specific scales on either of these scales.

[21:22](https://youtu.be/LhfXfjUh3D8?t=1282) But I found it interesting, they call this zone of really effective or deep productivity. And then when they depict that against this, I found this pretty fascinating and a good depiction. It's certainly logical when you think about it. They say, look, when you have low, I'm going to go to the lower left side. When you have, right, low experienced workers, meaning the tasks that they are performing or the time on the job and the time they have been able to realize rich, rich experience in that job, right? And those are the cases where you see it depicted in orange there, that they should be receiving whatever that corporation, or it could be a state government, or it could be other entity, whatever they're investing in this generative AI tool or tool sets, they make very good candidates.

[22:27](https://youtu.be/LhfXfjUh3D8?t=1347) And then we'll contrast that to the upper end where, you know, if these are highly experienced workers, right, and only the very, very complex gen AI tools and technologies are really going to yield them any benefit. And that stands to reason. We've already seen that, as I explained earlier, we've already seen that some of the call center agents that have been there for quite a long time, they can answer almost every question that a caller or that a, you know, a motoring public or a taxpayer would have. They can answer them because they're so experienced and they can even go quickly right to the source. But yet those that have been on, in one case, more than 80% of that agency, right, of that agency's call center staff had less than a year, in many cases, less than six months of experience. So you're not going to build up right away, the ability to go to the sources real quickly or answer the questions quickly.

[23:34](https://youtu.be/LhfXfjUh3D8?t=1414) So I thought this slide was a pretty, pretty good depiction that can be shown to investors or CFOs or business leaders as to where we want to, what kind of an AI technology project you may want to build. So without further, I'll go right here. Aguila, if you want to now stop the recording and we can jump right into the interactive dialogue.

**Unknown speaker**

[24:01](https://youtu.be/LhfXfjUh3D8?t=1441) Absolutely. I'm excited to hear that. Thank you.

**Chris Rein**

[24:05](https://youtu.be/LhfXfjUh3D8?t=1445) Okay.

**Unknown speaker**

[24:06](https://youtu.be/LhfXfjUh3D8?t=1446) Just for your reference, this is exactly 30 minutes. Okay.

**Chris Rein**

[24:09](https://youtu.be/LhfXfjUh3D8?t=1449) Oh, great. Okay. Perfect.

**Unknown speaker**

[24:12](https://youtu.be/LhfXfjUh3D8?t=1452) I just stopped.

* * *

Original page: https://innovate-us.org/a-state-based-approach-generative-artificial-intelligence-learning-from-new-jersey · Licence: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org
