---
id: "workshop:accessing-ai-safely-setting-up-an-ai-sandbox"
type: "InnovateUS workshop"
title: "Accessing AI Safely: Setting up an AI “Sandbox”"
url: "https://innovate-us.org/accessing-ai-safely-setting-up-an-ai-sandbox"
date: "2024-12-03"
duration: "90 minutes"
language: "English"
instructors: ["Dave Cole","Naman Agrawal","Amani Farooque","Ruthie Nachmany","Santiago Garces"]
recording: "https://www.youtube.com/watch?v=M9gbSLUnWe0"
transcript: "automatic, from the recording"
source: "InnovateUS (innovate-us.org)"
licence: "CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org"
---

# Accessing AI Safely: Setting up an AI “Sandbox”

InnovateUS workshop · 3 December 2024 · 90 minutes

Ever wondered how states are safely experimenting with AI to drive innovation while keeping ethical and governance concerns front and center? This is where AI “sandboxes” come in. They offer a controlled environment for testing AI technologies before scaling them across government services.
In this workshop, we’ll showcase the AI sandboxing initiatives in the State of New Jersey and the City of Boston highlighting how these governments are giving their public professionals access to the latest generative AI models. 
You’ll learn how AI sandboxes can help governments experiment safely, accelerate innovation, and ultimately improve service delivery.
Understanding AI sandboxing is crucial for anyone working on the future of public sector services. Whether you’re managing digital transformation or developing AI policy, this session will equip you with a clear understanding of how an AI sandbox can make your team more adaptive and efficient, without compromising on ethics.
In this 90-minute session, you’ll explore real-world case studies from governments actively using AI, learn best practices for policy development and deployment, and understand the critical role sandboxes play in fostering responsible innovation.

## Learning goals

- Understand the concept of AI sandboxes and why they matter for public sector innovation

- Explore real-world examples of AI sandbox initiatives in New Jersey and Boston

- Learn best practices for balancing innovation with ethical and governance concerns

- Discover how sandboxing enables public sector employees to shape AI policy

- **[Register here](https://airtable.com/appdAyP4l2W9gxQdc/pagx9sXF2F53dI7kn/form)**

## Speakers

- **Dave Cole**, Chief Innovation Officer, The State of New Jersey
- **Naman Agrawal**, Director of Engineering, New Jersey Office of Innovation.
- **Amani Farooque**, Director of Product, New Jersey Office of Innovation
- **Ruthie Nachmany**, Product Manager, Office of Innovation for the State of New Jersey
- **Santiago Garces**, Former Chief Information Officer, City of Boston

Recording: https://www.youtube.com/watch?v=M9gbSLUnWe0

## Transcript

Automatic transcript of the recording: names and terms can be misheard, and the times can be a few seconds off. Each time links to that moment of the video. The transcript alone: https://innovate-us.org/accessing-ai-safely-setting-up-an-ai-sandbox.txt (plain text) and https://innovate-us.org/accessing-ai-safely-setting-up-an-ai-sandbox.json (every segment with its start and end second).

**Santiago Garces**

[0:00](https://youtu.be/M9gbSLUnWe0?t=0) Sure. Hi, everyone. I'm Santi Garces. I am the Chief Information Officer for the City of Boston. As Agueda mentioned, we're very excited to share with you about our journey at the City.

**Dave Cole**

[0:16](https://youtu.be/M9gbSLUnWe0?t=16) Pass it over to Dave. Very good. Agueda and Santi, thank you both for your efforts in pulling this together. Our work in trying to put together an AI sandbox in New Jersey started well over a year ago when the governor announced through an executive order a focus for New Jersey to leverage artificial intelligence to responsibly improve the service delivery in the state, as well as use artificial intelligence as a way to grow and continue to enhance the state's economy. So that put us on a path of working with Dr. Beth Novak, who is New Jersey's first Chief Innovation Officer, my predecessor and now our Chief AI Strategist, on a strategy that included providing training for the full state workforce on use cases, things to think about when using AI tools, potential guardrails and guidelines, as well as opportunities to think about how AI can improve the day-to-day work of staff across the board. So that training program was put out by Innovate US. New Jersey was very happy to participate and develop and make that available as strongly encouraged by our governor for our full state workforce. To pair with that, we put together a sandbox tool, which we're very excited to share with you now. The team's going to present on how it works, how we've thought through some of those questions around what the right guardrails should be and what the right constraints should be and how we can enable responsible use of AI. As well, we want to make sure that where we've learned something, we can share it with the broader community here. So I'm super excited. So many folks have joined. Thank you all for your interest in this topic and I'm excited to share our work and hear your questions.

[1:58](https://youtu.be/M9gbSLUnWe0?t=118) So I think with that, I'll hand it off to Imani, who's our Director of Products at the New Jersey

**Amani Farooque**

[2:02](https://youtu.be/M9gbSLUnWe0?t=122) Office of Innovation to take it from here. Awesome. Hello, everyone. And Dave, thank you so much for that introduction and tee up. We're very excited to share with you today a little bit more context about how we built the New Jersey AI Assistant. And we'll also talk through some of the considerations that we were thinking through that you might likely think through in bringing this to your own state or government, as well as what we've learned and where we want to go next in our product roadmap. So to briefly overview what we built and what the New Jersey AI Assistant is, we've created this generative AI powered platform, meant and built for our 64,000 plus New Jersey state employees to learn how to use experiment with generative AI in the context of their government work. So here we'll show a little bit of a brief demo. The interface intentionally is very familiar, that kind of chat interface that folks are used to using. People can enter prompts, interact, send follow ups. And then also, you can see that we have the ability to upload files. So adding more context in the prompts themselves to refine and get better answers. You'll also see at the bottom of this page, a feedback widget that allows people to give a quick kind of diagnostic. Did they find what they were looking for and give a little bit more feedback on their experience using this tool. So next to talk a little bit about the goals that we had when setting out creating this tool. Our first kind of short term or immediate goal was to make sure that there was a safe sandbox type environment for those state employees who are going through the responsible AI training created by the Innovate US team to actually put those examples and learnings into practice and really solidify what they're learning as far as guidelines on responsible

[4:02](https://youtu.be/M9gbSLUnWe0?t=242) use of AI with something more concrete. The second and more kind of broader goal is to more generally lower the barrier for public professionals in New Jersey to actually understand how they can use this technology as a tool to improve their government processes or solve their agency problems. And then lastly, kind of the undercurrent to all of this is making sure that we really offer a good viable alternative to some of the open source or quote unquote free generative AI platforms that might be more at risk of sensitive government data being used for things like model training or other unintended purposes. So creating that kind of safe, isolated environment for our employees. So to talk specifically about the features that we really wanted to include at this initial launch and then we thought were critical. First of course is just ensuring a secure platform, making sure that we're able to protect and authenticate users, make sure we know that these are New Jersey government employees accessing the system, as well as making sure we're hosting this on our own instance so our data is protected. The second consideration was making sure that we can handle large amounts of input content. So this kind of drove our decision for which models to use and knowing that government data can often be lengthy and the real value of generative AI is to be able to kind of process this large amount of data. Having kind of a high token input count or a high amount of data that we can process in those prompts was really important.

[5:40](https://youtu.be/M9gbSLUnWe0?t=340) And similarly, the form factor in which this information presents is also important. So being able to support PDFs and images initially at launch was really critical in making sure that those pieces of content could be used by government staff. We also thought about how we needed to create an infrastructure that could support the demand and load on the system. So of course this was a new product launched alongside the training. We wanted to make sure as adoption increases we have the ability to scale up our infrastructure, you know, increase instances or increase the size of those to make sure that whenever anyone wants to use the AI assistant it is available and accessible and reliable. We also wanted a way to capture asynchronous feedback. So you saw the feedback widget in the short demo that we presented. That allows, again, that pulse check, that quick diagnostic tool to see are users interacting with this tool in the way that we expect? Are they finding the information they need? And then capturing some really helpful initial troubleshooting issues or things that needed to be fixed after launch as well as kind of longer-term feature ideas or other requests that could make government workers' lives easier. And then last but certainly not least, we wanted to make sure we included robust analytics and monitoring. So not only web analytics to understand, again, how people are navigating through the assistant itself, but also API and server analytics that gives us the ability to see the types of prompts and questions that people are inputting, the types of responses they're getting back, as well as being able to monitor the performance and cost more easily. And the next thing I'll note, this was only possible through the great partnerships we had throughout this work. Of course, as the Office of Innovation New Jersey, we kind of took lead in developing the tool itself, but it took a lot of coordination and

[7:43](https://youtu.be/M9gbSLUnWe0?t=463) collaboration with our Office of Information Technology to set up the infrastructure to help manage the vendor relationships that powered this, as well as do some robust testing ahead of launch. The New Jersey Civil Service Commission was taking lead on making sure the learning content was available and accessible to all New Jersey government employees, and it was really helpful in figuring out how to position this AI assistant in context of where people might be interacting with it in the training itself. And then, of course, our partners at Innovate US created really engaging and interactive content, which really helped solidify the lessons learned about how to generate effective prompts and how to use AI in a responsible way. All right. Next, I'll pass it to Naman to talk about some of the key questions and considerations

**Naman Agrawal**

[8:37](https://youtu.be/M9gbSLUnWe0?t=517) we had when building this tool. Thanks, Amani. Hi, everyone. My name is Naman. I'm a Director of Engineering at the New Jersey Office of Innovation, and I'm going to walk you through some of the questions we were thinking about as we built. Hopefully, they could be helpful for you if you are going down a similar path. So, the first thing when we were releasing this tool, it's a novel technology, and it is a very powerful technology. And we wanted to make sure that we're setting up our users to have education and the guardrails in place to use it responsibly and effectively as well. So, as Amani mentioned, we launched this tool alongside the Responsible AI Training course, and we strongly recommended that users go through the full training before they use the tool any further. And the training actually walks them through in a hands-on fashion how to use the tool and what are the tradeoffs to be thinking about as they're asking different questions to the chatbot. We also launched this in the context of an AI Acceptable Use Policy by the AI Task Force in New Jersey, which gives some concrete guidelines on things to think about as you're using the tool. So, a few to highlight here. One is to be fact-checking the output, because the output from the AI model is not going to be verified for accuracy. Not to be using sensitive or confidential information, and then to be disclosing the use of AI when that's relevant.

[10:04](https://youtu.be/M9gbSLUnWe0?t=604) Another area that we were thinking through is privacy and safety. So, specifically for data privacy, there were a few questions that we have been thinking through. One is around chat history, whether that should be saved. We found that that could be potentially a good user need or like a feature for users to be able to see their past conversations, but we also want to weigh that against how we store this data and what are the guidelines generally in the state on how to store the data properly. Similarly, we think about user inputs, whether that should be logged for analytics, and then generally how we should handle sensitive information as it's passed through the system. So, all of these questions are things that your team should be thinking about based on your state guidelines, what are your best practices that you want to be following, and your user needs. There's also an area around safety. When you're working with such an open-ended tool that can give you all sorts of answers and take in all sorts of inputs, we want to make sure that we have proper safety in place. So, when we were using Azure in this case, they had a feature where you can set or configure different safety filters to say we want to first automatically identify and block any harmful inputs or outputs. So, anything that it deems could be related to hate speech or something like that, we would want to block those requests from even going through the system.

[11:29](https://youtu.be/M9gbSLUnWe0?t=689) And we have regular monitoring in place so that we can track if there is unsafe usage and adjust our policies if needed. And then there's an area around the technology itself. So, how should we build this? Our approach was we wanted to launch something quickly alongside the training course while still adhering to our office principles on design and how this can be a usable product. So, first thing regarding the AI model, we worked within our existing platform so we could move quickly. In this case, we were using an Azure service which leverages the open AI's GPT model, but just calling out that there are counterparts and other cloud providers. And for the chatbot itself, we built this off of Microsoft's open source chatbot application. Our source code is also open source. In case other folks would like to see that, we can link that later. But we chose to do this because this allowed us to deploy something pretty quickly. It's low engineering effort because we didn't have to build something from scratch. Yet, we also had some room for customization. So, we were able to make accessibility improvements to make sure that we can reach the full set of our users. We also wanted to add in analytics, as Amani talked about, and certain specific features like PDF uploads. I just want to call out, though, again, that that's the way we went, but there's a large spectrum in terms of whether you want to fully buy enterprise solutions or create something from scratch by yourself or some pieces in the middle. And then, one final thing to talk about is the financial considerations. So, we were looking at two different pricing models for us. One was pay as you go and the other is provisioned. Pay as you go basically means that for every new question that somebody asks to the model, for every new prompt, you would be charged some fixed amount. So, as usage increases,

[13:30](https://youtu.be/M9gbSLUnWe0?t=810) your costs would increase. The other option is provisioned, which is basically a dedicated model resource to answer to handle the prompts. And this is going to be based on hourly pricing, so time-based, not usage-based. So, for our approach, at least the way we started, we wanted to go with the pay as you go so that we can learn about usage and see how that changes over time, since this was a new tool that we were experimenting with. So, we set a budget, made that clear with us and our partners, and then we've been monitoring both the costs as well as the application performance to see whether it still makes sense to keep this model or it may make sense to go to a dedicated infrastructure in the provisioned case. And I will pass it to

**Ruthie Nachmany**

[14:18](https://youtu.be/M9gbSLUnWe0?t=858) Ruthie now. Thanks, Naaman. I'm Ruthie, Product Lead at the New Jersey Office of Innovation. I'll talk about how we built it and what we are doing next. So, I'll first talk about how we got out the first version of the New Jersey AI Assistant. So, as Amani and Naaman mentioned, we developed trainings for New Jersey state employees in collaboration with Innovate US, the Civil Service Commission, and OIT. We wanted to get those into our state employees' hands as soon as possible so they could learn about how to use generative AI safely, responsibly, and effectively for their work. So, we identified July 3rd as a date that we could get that into our state employees' hands, and then we worked backwards from there to figure out what we could develop alongside the trainings. Again, the training has people who are going through the trainings using an AI assistant to test out different prompts and generate responses and learn about how to use the tool in a hands-on way. So, we wanted to give them the New Jersey AI assistant as a tool alongside the trainings. So, we started to work on this on June 11th.

[15:20](https://youtu.be/M9gbSLUnWe0?t=920) We evaluated different solutions, as Naman discussed. We settled on this Azure solution for version 1. We began identifying which are the most important pieces of functionality to include in that first version. We started developing that. And on June 20th, we started testing that out internally, identifying what infrastructure needs we had, validating those, getting feedback from users, and starting to understand if there were any improvements we needed to make before launch. To prepare for launch, we worked with the governor's office and agency heads to develop communications and with our partners as well to ensure that the rollout would go smoothly. And on July 3rd, we released the training and also the New Jersey AI assistant. And since then, we've been monitoring usage, evaluating if there are any bugs that users are hitting, making fixes to those, making incremental improvements such as improving the user experience, things like text box width, bumping up models as new versions are released.

[16:22](https://youtu.be/M9gbSLUnWe0?t=982) And we've also been doing user research through sessions with users by analyzing the feedback that we get in that footer of the AI assistant that Amani showed and also doing surveys. So, trying to gather a sense of how people are using the AI assistant, what they'd like to be able to do that they aren't able to do today, and then using that to prioritize and develop improvements. I'll talk about some of the common use cases that we're seeing in the state. We're seeing state employees summarize PDFs, notes, and transcripts. A lot of state employees analyze documents that are hundreds of pages long, and so sometimes they just need to do a quick evaluation to figure out kind of how to triage their workflows. Sometimes they need to assess if documents include certain information. So, those are some of the use cases we're hearing about. We're seeing employees extract text from image files, draft emails and letters for internal use, and also as a first draft for external use. We're seeing them analyze resident feedback at scale, and we're also seeing it used for coding support. Everything from analyzing mainframe COBOL code to generating templates, helpless debugging, so kind of as a supportive tool there.

[17:33](https://youtu.be/M9gbSLUnWe0?t=1053) This is some of the feedback that we're seeing. We're hearing from people who are using the AI assistant every day. We're hearing the ways that they're using it and how it's supporting them with their workflows, and we're also hearing about ways where it's falling short, things they want to be able to do that they aren't able to do today, and then we use that again to guide our roadmap improvements. I'll talk a little bit about the metrics that we're seeing so far. So, since we released in July, which I think we released on July 3rd, it's December 3rd, so exactly five months ago, we've seen over 10,000 users, so that's about 16 percent of our state employee workforce. We see a number of repeat sessions, an average of seven sessions per user, and about three prompts per session per user. We've seen 79 percent positive rating on the feedback widget at the bottom of the page. We evaluate if the model is successful in answering or if there's error, so that's that 97 percent rate. Then Numm and Talk to Load at the budgeting, so this is an average of our spending per month, so this is well below the budget that we allocated per month for the AI assistant. And then, yeah, if you go to the next slide, we're continuing to identify and make improvements to help the public sector in New Jersey use generative AI safely and effectively.

[18:49](https://youtu.be/M9gbSLUnWe0?t=1129) These are some of the things in our roadmap now, bumping up file sizes, expanding the file types that we accept, allowing users to upload multiple files, making it easier to copy and share the output, continuing to make user experience improvements, and providing more guidelines, templates, and instructions for our workforce. And so I'll now pass it over to Aguera. Thank you.

**Agueda Quiroga**

[19:12](https://youtu.be/M9gbSLUnWe0?t=1152) Thank you very much for this. So now we are moving to Santiago Garcia's presentation, and then we will open the Q&A session. Santiago, when you're ready, the stage is yours.

**Santiago Garces**

[19:25](https://youtu.be/M9gbSLUnWe0?t=1165) Perfect. Well, thank you again. It's great to go after New Jersey because we're, I think, maybe a step or two behind, though. But I think that what I would like to share first is a little bit of the motivation of why. So foremost, we haven't deployed a sandbox, but we're in the process of planning the deployment of one and kind of thinking about the rollout of the next phase of our generative AI work. But I think that I would start maybe by asking, why are we trying to deploy a sandbox? And I think that when we think about the current state, given the guidelines that we released over a year ago, almost two years ago, the way that most employees are able to access generative AI tools is through the freely available models or through subscriptions that the departments or people are able to get through individually or through a department account. And that means that we have a really limited footprint for people to be able to, and especially in government, there's concerns from people to thinking about when is it fair game to do a particular task using some of these tools. So I think in terms of motivation, I think that it aligns closely with what New Jersey described around kind of creating a more seamless access experience for our employees. And I think that the other piece that we start thinking about is as we're getting to a point where there's a number of different ways in which employees can access generative AI tools through the applications that they already use. So like

[21:26](https://youtu.be/M9gbSLUnWe0?t=1286) if a person's using Photoshop, there's features within Photoshop that leverage generative AI. We were able to acquire licenses for Gemini, which is the kind of equivalent of Copilot in Microsoft, but for Google Workspace. And ultimately, what we're trying to do as an IT department is find the balance of how is it that we enable access to a broad group of users to these tools, while also remaining within the bounds of our fiscal constraints. So we're on the one hand seeing that, I mean, we hypothesize it's exciting to see the numbers that New Jersey has now through experience, but we hypothesize that the patterns of usage of generative AI for our workforce are going to be such that a few people are going to use these tools regularly, but the majority of the users are going to be more infrequent. So when we think about the counterfactual of a sandbox, we think that the licensing model for a lot of the commercially available tools is per user. And we're seeing numbers around $20 per month per user, $30 per month per user. So in some cases, you're seeing almost the doubling of the cost of productivity software for an employee. And when you think about it at a state level or at a city level, that can be quite daunting. And we don't necessarily have the resources to double the IT budget. So we are forced to think about other ways in which we can provide access to these resources. Knowing that making the access to these tools, going through a different

[23:32](https://youtu.be/M9gbSLUnWe0?t=1412) application reduces some of the value that you have of it being embedded within the application itself. Like if you have to go and summarize an email by copying the email from your email account, putting it in a sandbox, and then doing something with that that's much less convenient than just clicking a button that says summarize this email. But as validated a little bit by what the numbers at New Jersey are sharing, if it costs $1,000 per month to support 10,000 users, that's much more reasonable than going in and saying that you're going to increase your IT budget by half a million dollars. So anyways, generally, again, some of the same motivations around security and privacy with the notion that you have access, that we're able to provision a generative AI access within a cloud provider that is completely within the control of the city is very appealing. Because all of the record around the session, so what users used it when, what did they prompt, what was the response, we're also incorporating user experience feedback. So all of that information stays within the city. It's not shared with a provider. And we know that the actual inference, so the magical calculation that the large language models are doing behind the scenes, all of that is also happening within the session, within our cloud envelope. So as long as we still believe that public clouds are sufficient in terms of security and privacy, which I think that we still all feel that way, and we can go and get the suck to audits and whatnot from these providers. And increasingly, some of the services that underpin these sandboxes are also

[25:41](https://youtu.be/M9gbSLUnWe0?t=1541) compliant with other. So the decision in Boston is using an AWS-powered sandbox. And Bedrock, which is the AI kind of service, the LLM service for Amazon Web Services, is undergoing, so it's FedRAMP certified, and it's undergoing a further review for a higher level of FedRAMP certification. So again, just sharing, generally same approach, but it's amazing what the work that New Jersey is doing. I think for us, getting the resources to commit a team of three people to work on it and be able to do it with the level of rigor that they're doing is difficult. So I'll share with you what we're doing instead.

[26:42](https://youtu.be/M9gbSLUnWe0?t=1602) So the Bern Center at Northeastern University has been partnering with the state of Massachusetts to build a sandbox for the state of Massachusetts. And thanks to the leadership of Secretary Jason Snyder, who had the vision of thinking, yeah, like it'd be really great, similar motivation, how is it that we enable access to a broad use, a broad set of employees to these tools, they started developing a sandbox. And in full disclosure, I also work as a faculty member at the Bern Center, so I'm the technical advisor for the students. We've been building an open source tool for the Commonwealth of Massachusetts, and we got inspiration from Harvard University that Erica Bradshaw, who I think that is now moving to the state of Massachusetts, she had built, her team had built the first sandbox for Harvard. And again, with the same motivation, how is it that you enable access to something that looks like ChatGPT, but that is private, that is a little bit more cost efficient, and where the IT department has a little bit more control. And with the added value that one interesting feature that they had that seemed very appealing is the ability of testing different models within the same interface, so it's not being constrained by just having one model, but being able to test different models.

[28:07](https://youtu.be/M9gbSLUnWe0?t=1687) So I'll show you briefly the project. The sandbox is open source. That's why we're planning on using it within the city of Boston. We're now trying to figure out how is it that we do the rollout, partially because we also are interested in seeing what are the outcomes that are enabled by getting access to these tools, and we're also thinking about it in the context of giving access to these co-pilot experiences and some other tools. We're thinking about it almost like an experiment with the hope that we learn what is the overall outcome as we start thinking about ultimately we're going to have to justify our spending in these kinds of technologies and right now we're stuck in between people that are excited and think this is incredibly useful and we should have infinite amounts of usage of these tools, and people that are like this is terrible and it will never work or it's threatening, but we don't really know what is the overall outcome for the public sector of getting access to these tools, so we're thinking about as we roll out these tools how is it that we can maximize our learning, but let me share briefly the page for one second with the GitHub repository. Again, you can go and deploy this.

[29:30](https://youtu.be/M9gbSLUnWe0?t=1770) This is built on Amazon Web Services and the code itself has the cloud development kit that enables you to deploy within your AWS instance and provisions the different components. All right. So, again, if you go to the burn center GitHub account, this burns Gen AI sandbox is the project. So this, again, all the credit goes to the Commonwealth of Massachusetts and the EOTS team that has been a great partner to us and now the city is going to be able to use this. So one key question that we keep asking ourselves is why use a sandbox?

[30:35](https://youtu.be/M9gbSLUnWe0?t=1835) You can get access to this stuff, right? So besides cost, why use a custom built interface for generative AI, knowing that open AI has almost infinite amounts of resources that they're putting towards the development of their products. Like all these companies, if Google with Gemini has access to all these things, we're always going to be behind in terms of features. And the main thinking here is actually for public employees, it might be better to have less features and have more features that are more relevant to government. And that might be worth being a little bit behind in terms of features. So let me share with you a little bit of what that means. So obviously, you have like the typical chat interface to interact with the models. But one thing that the students, Drew and Elena, who developed this tool have done is enable a couple of features that are really relevant for us.

[31:38](https://youtu.be/M9gbSLUnWe0?t=1898) So the first piece is getting access to people to be able to compare different models for the same prompt is useful because it helps, well, one, develop the awareness that different models work slightly differently and they are better at different tasks. And then kind of the side-by-side comparison is really helpful. One feature that the students have developed that is also really helpful is particularly for prompt engineering with the same model using different prompts and being able to compare side-by-side. So again, most of the development like prompt engineering is kind of like a heuristic exercise. You try to different approaches and different things to see how good of a result. And this gives people the ability of doing prompt engineering to being able to compare the results. The smart model, which was an innovation from Drew, basically puts the prompt through an LLM that then tries to select a large language model that is best suited for that task.

[32:47](https://youtu.be/M9gbSLUnWe0?t=1967) So the thought here is different models have different costs. The bigger models are best performing for everything, but they tend to be, they're more expensive. So here, like you're basically, the smart model is just like parsing the prompt through another model that just makes a selection based on an assessment of the complexity of the prompt. It's kind of like a cute little feature. But again, we've built tasks for document summarization, for translation, for writing a memo, writing an email, like tasks that are very, that are the meat and potatoes of what we do in government, where we spend 90 percent of our time in a bureaucracy. And again, that's where we think like, even though we're sacrificing a lot of features and functionalities in some sense, like it's been working with the Commonwealth of Massachusetts to try to figure out, like, what are the things that are the most relevant that that are helpful?

[33:48](https://youtu.be/M9gbSLUnWe0?t=2028) Again, for if you're interested, again, like this is the architecture, the sandbox is built on the Amazon Web Service architecture reference. There's also an open source project that Amazon put out and is being tweaked by the students in partnership with the Commonwealth. And you pay per use, and we're excited to compare our numbers with the with the New Jersey numbers, but again, like we're hoping that it is very sensible in terms of cost to be able to provision to to make it accessible for a large group of users. And as we start thinking about the training materials and other parameters that might impact usage, we can have a little bit more flexibility before making a commitment around like what types of tools we make available with some of the same kind of considerations that New Jersey had around integrating with our identity provider with and the city is going to be paying ID, which is our single sign on.

[34:50](https://youtu.be/M9gbSLUnWe0?t=2090) So again, we. And the code for the sandbox that we're going to use is here, it's on GitHub, I know New Jersey has theirs out there because I go and check it out every once in a while as well to see how we can learn from each other. And and again, it's just trying to give us an opportunity to get to the next level, even if we think that the AI is helpful. How is it that we really provide access to a large group of users and do it in a way that is relatively economical and then understanding a little bit of the tradeoffs and thinking about like how does that fit in like a broader kind of IT department strategy provisioning a new technology to a large group of users, knowing that there's still a fair amount of uncertainty around what what features the integration between the features and the applications that people use and doing it in a way that's very collaborative and we're grateful for the for the Commonwealth of Massachusetts and to the Berne Center and the students that have built this.

[35:59](https://youtu.be/M9gbSLUnWe0?t=2159) So so I'll stay there.

* * *

Original page: https://innovate-us.org/accessing-ai-safely-setting-up-an-ai-sandbox · Licence: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org