---
id: "workshop:the-ai-solution-smarter-tools-for-public-call-centers"
type: "InnovateUS workshop"
title: "Helping the Helpers: AI Tools for Frontline Resident Service"
url: "https://innovate-us.org/the-ai-solution-smarter-tools-for-public-call-centers"
date: "2026-07-15"
duration: "60 minutes"
series: "Reimagining the Public Contact Center: Modern Strategies for Resident Service"
instructors: ["Beji Joseph","Tosin Adebiyi"]
moderators: ["David Fields"]
recording: "https://www.youtube.com/watch?v=AAxtoXbmAn0"
transcript: "automatic, from the recording"
source: "InnovateUS (innovate-us.org)"
licence: "CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org"
---

# Helping the Helpers: AI Tools for Frontline Resident Service

InnovateUS workshop · 15 July 2026 · 60 minutes

This workshop explores how governments are using AI to help frontline staff answer resident questions more accurately, navigate complex policies more efficiently, and provide better service across public contact centers. Rather than replacing workers, these tools are designed to help staff find information faster, reduce time spent searching across systems, and handle a wider range of resident inquiries with confidence. Drawing on examples, including the MassHealth Helper initiative in Massachusetts, participants will examine emerging approaches to AI-assisted resident service, lessons from implementation, and practical considerations for deploying these tools responsibly.

## Learning goals

By the end of this workshop, participants will be able to:

- Understand how tools like MassHealth Helper can support contact center staff in delivering faster, more accurate service to residents. 

- Evaluate the opportunities and limitations of retrieval-augmented generation and similar tools in high-stakes public service contexts.

- Identify key considerations for implementing new tools in contact centers responsibly, including trust, equity, and maintaining human support.

## Speakers

- **Beji Joseph**, Deputy Director, MassHealth Medicaid Systems Development
- **Tosin Adebiyi**, Director - Special Eligibility Programs and Audits (SEPA) MassHealth

## Moderators

- **David Fields**, Senior Fellow, Burnes Center for Social Change

Part of the series [Reimagining the Public Contact Center: Modern Strategies for Resident Service](https://innovate-us.org/workshop-series/strategies-to-modernize-public-call-centers-for-better-resident-service).

Recording: https://www.youtube.com/watch?v=AAxtoXbmAn0

## Transcript

Automatic transcript of the recording: names and terms can be misheard, and the times can be a few seconds off. Each time links to that moment of the video. The transcript alone: https://innovate-us.org/the-ai-solution-smarter-tools-for-public-call-centers.txt (plain text) and https://innovate-us.org/the-ai-solution-smarter-tools-for-public-call-centers.json (every segment with its start and end second).

**David Fields**

[0:00](https://youtu.be/AAxtoXbmAn0?t=0) We've looked at what well-designed contact centers look like and what it actually takes to modernize them from the inside. Today we're turning to one of the most consequential questions in the space right now. How do you bring AI into a public service environment in a way that actually helps the people doing the work? The tool we're discussing today was built not by a large vendor but with the help of student fellows in the AI for Impact program, a partner project of InnovateUS where student teams partner with government to build tools that solve some of the most challenging problems facing governments. It's a program that's product-based and it's not a traditional classroom course. So the students are working full-time on real AI projects for state and local governments and non-profits and every project is selected because it solves an important public problem, has committed leadership, and we know that it can be implemented, not just researched. So the program really has two goals. It is to deliver impact by improving government services for residents and it's to develop talent by preparing students for careers building responsible AI in the public space. So in the spring of 2024, two of the AI for Impact students worked full-time alongside the MassHealth team to build an AI tool called the MassHealth Helper. I'm really delighted to welcome Beji Joseph and Tosin Adedbi from MassHealth in Massachusetts and now we'll turn it over to them to present. So thank you.

**Tosin Adebiyi**

[1:45](https://youtu.be/AAxtoXbmAn0?t=105) Thank you so much, David. We're very happy to be here today. So today Beji and myself are going to be presented on the MAG-AI from ideation to adoption, how the MassHealth Enrollment Center AI knowledge tool was conceived, built, governed. We also talk about the rollout, the adoption, and how it was positioned for scale. So today's story is not just about launching an AI tool, it's about how an operational business need became a government production-ready solution. So our MAG-AI was created to help our enrollment staff find policy and procedural answers quickly. And the reason for this is just to have a better consistency. So the presentation will walk through the full journey. We'll also talk about the problem, the discovery, the development, rollout, the adoption, and then lessons learned. We'll also talk about the next steps as well. So the key message is about responsible AI work here because business, technology, privacy, security, and operations all work together to deliver the finished product. So this slide talks about our AI journey at a glance from the operational need to government production rollout and the 2026 scale path. So as you can see on the chart, the journey started with a clear operational need. Our staff was spending a lot of time across multiple sources to get answers, you know, from policy questions to process answers.

[3:27](https://youtu.be/AAxtoXbmAn0?t=207) And the discovery phase helped us understand where our staff were losing time and confidence so we were able to identify what their pain points were. And then the build phase translated that need into a secure RAC-based AI assistance. And then the governance shaped the solution from the beginning, including privacy, security, accessibility, compliance, and responsible AI review. And for the rollout phase, we went with a phased rollout across our seven enrollment centers and adoption data now informs us about the next roadmap. So the main takeaway here is this was not just a technology first project. This was a business problem that was solved through responsible technology delivery. So what's our need? Why this all started? So our policy and procedural knowledge was fragmented. Our enrollment center staff support complex member data every day, but the information they rely on was spread across multiple sources where they had to search policy manuals, go on our state website, the mass.gov website, go on SharePoint, you know, to be able to find information on MEC weekly updates. And some of them even created their own personal notes for reference. So that fragmentation created operational friction. Staff had to spend multiple times searching instead of fully focusing on member interaction. So you created knowledge disparity, you know, different staff have different levels of confidence depending on what sources they found and maintained.

[5:05](https://youtu.be/AAxtoXbmAn0?t=305) So when reliable answers were had to locate, common questions could easily be escalated to a supervisor or sometimes they work within themselves to find the answers. And for a program like ours, where we serve around 2.1 million members, improving staff access to trusted knowledge has direct operational and member experience value. So the internal problem, the mass enrollment center needs a more centralized accessible source for our policy and procedural information. So the discovery, what our staff research reviewed. So we didn't just want to assume anything upfront. We validated the problem through on-site research by having conversations with our staff, conducting interviews. We also did reviews to our SMEs. So the research showed four key consistent patterns. So staff used multiple sources to obtain information. And in the eligibility world, and even in the present world, information is constantly changing. Our staff maintained reference guide documents.

[6:13](https://youtu.be/AAxtoXbmAn0?t=373) They relied on each other and supervisors for confirmation. So this told us one thing. We needed a tool that was fast, something that is plain English and something that is source packed and safe. So the business requirement became very clear to us. We need to make sure we had centralized access to approved knowledge in a way that supports real workflow. So we didn't just want to change their workflow. We want to find something that supports what they're actually doing, especially during member interaction. So this discovery work is made to a solution that is practical rather than theoretical. So what success had to look like? So I'll turn it over to Beji. Beji will talk more about the technological aspect of it.

**Beji Joseph**

[7:02](https://youtu.be/AAxtoXbmAn0?t=422) Thanks, Dawson. From the technology side, the success was defined by four release standards, functional, intuitive, accurate, and secure. Functional meant the tool had to answer real staff questions in plain English using approved or internally known and approved content. Intuitive meant it had to work naturally in the staff workflow without adding additional steps or friction. Accurate meant answers needed to source grounding, citations, and guardrails to reduce unsupported responses. Secure meant protecting PAI and PHI, enforcing access controls, and aligning with government standards. The important point is that the production readiness was not based on one factor, but it required multiple factors together.

[8:08](https://youtu.be/AAxtoXbmAn0?t=488) How the application came about? The project moved from statewide AI readiness into a concrete mass health use case during 2024. And the early work started through Executive Order 629, the AI Task Force, Innovate MA, and the Northeastern Cohort Partnership. The first prototype used a RAG system to test whether the approved knowledge could be retrieved and summarized in a useful way. So we started really small to show if this can work. In August 2024, the second cohort helped to move the work into the EHS AWS environment. January 2025 is when we released our first production base, followed by releases focused on accuracy, stability, compliance hardening, UX, and operational maturity. The April 2026 release, the release 1.4, that added MEC Weekly Update Search, which is important because the staff needed a reliable access to the recently changing guidance. The journey shows that the product matured through multiple releases or iterative releases, not through just a one-time launch and forget it. How do we develop it? The multidisciplinary delivery path.

[9:45](https://youtu.be/AAxtoXbmAn0?t=585) The delivery model was intentionally multidisciplinary. The operations, IT, EODS, privacy, general counsel, Northeastern co-op students, and special eligibility SMEs all had a role. This was a business-led and technology-enabled project. Business SMEs validated whether the tool was answering the questions in the correct way. Development followed agile delivery method with continuous business feedback loops. CICD guardrails, automated testing, and accuracy relevance monitoring were all part of the operating discipline. The team treated feedback as a product input, not as noise. The result was a product with both business ownership and technical ownership. Now let's look at the technical architecture.

[10:46](https://youtu.be/AAxtoXbmAn0?t=646) The architecture uses a retrieval augmented generation pattern. That means the model does not simply generate answers from general knowledge it has. It first retrieves approved content source and then processes it. The staff access the tool through secure user access layers using single sign-on and Cognito. And Comprehend Medical supports the privacy layer by redacting PAI, PHI before processing any of the input. API Gateway and Lambda provided the serverless orchestration layer, that is the standard AWS structure, and the Kendra indexes approved SharePoint and mastered GovContent so that the system can retrieve relevant policy and procedural information. The Bedrock Models summarize retrieved results into a readable human-readable response. DynamoDB was used for the logging and then the CICD infrastructure management and controlled deployment. Two key controls or requirements for us from a technical side. No user data is used to train models and sensitive data is not stored in any of the saved logs. Safety by design, privacy, security, and compliance controls. The safety model was built into the lifecycle, not left to the user behavior alone. Access is controlled through single sign-on, the user leased access privileges, and perimeter protection such as a cloud front and web access firewall. Data protection included, as we said in the previous

[12:50](https://youtu.be/AAxtoXbmAn0?t=770) slide, the PAI, PHI detection and redaction with no sensitive data stored in the saved logs. Model controls include managed Bedrock Models with no fine-tuning with customer data and prompt boundaries. The compliance path included security review, accessibility, WCAG, JAWS, FedRAMP, and HIPAA considerations. The human oversight remains the heart and center of this. Staff review source links and business IT team monitor feedback, accuracy, and content quality over time. The governance journey from pilot guardrails to enterprise practice. Governance also evolved alongside the product. At the Commonwealth level, EODS provided statewide AI policy and processes. At the EOHHS level, the steering committee reviews use cases, set expectations, and support responsible adoption. At the project level, governance included privacy, security, labor, policy, technology, and operations input. The community of practice helped share reusable patterns, lessons, and responsible AI practices across agencies. The goal is trustworthy AI, valid, reliable, safe, and secure. And key aspect was transparency. This project became a practical example of moving from pilot guardrails to repeatable enterprise approach.

[14:42](https://youtu.be/AAxtoXbmAn0?t=882) With that, I will hand it back to Tosin. Thank you so much, Bhaji. So for the rollout

**Tosin Adebiyi**

[14:48](https://youtu.be/AAxtoXbmAn0?t=888) from the pilot users to all enrollment centers. So the rollout was phased intentionally. We did not want to move from a prototype to full deployment overnight. So we started small from Our smallest enrollment centers, we reviewed feedback, stabilized the product, then we expanded to our other enrollment centers, which are much larger. So the tool was rolled out across all seven Maslow enrollment centers, Springfield, Quincy, Taunton, Chelsea, Tewksbury, Worcester, and Charlestown in Boston. And more than 400 of our staff members were onboarded. We had four releases in 2025, and we have two planned releases in 2026. So the business lesson is that trust has to be built through controlled rollout, visibility support, and continuous improvement. So this data shows the tool moved beyond a pilot curiosity into real operational use. So in 2025, as you can see, we had 48,842 user sessions. The average monthly usage rate was 59%. We had usage picked in July of 2025. As you can see on the chart, this is when we completed the onboarding of our teams. And all sites had come online at that point, and then 6,392 sessions and 253 active users. So the average was around 31 sessions per active user per month, roughly one question per active user per working day. So this tells us that our staff are actually using the tool as part of their daily work, especially when they need support to answer any complex eligibility or changing policy questions.

[16:48](https://youtu.be/AAxtoXbmAn0?t=1008) So the next opportunity is to make it even more routine and embedded in their daily workflow. So this slide talks about the feedback. The quality improved through monitoring and iteration. So user feedback became one of the main mechanisms for improving the products. So the thumbs down signals, question logs, the SME review, and release fixes gave the team a practical feedback loop. More than 99% of the sessions did not receive a thumbs down feedback. We had 813 positive feedback signals. And for the automated regression testing, it showed 70% to 80% accuracy range, which helps the team identify if we need to do further tuning. So the most common question areas also showed the type of questions that our staff are asking, from the asset and income questions, to life insurance, to retroactive backdated coverage. So the key takeaway here is that our MECH AI is most valuable for complex eligibility research and recently changing rules.

[18:01](https://youtu.be/AAxtoXbmAn0?t=1081) And for the feedback, the feedback also helped us identify release priorities, which includes hallucination prevention, context indexing, and data quality improvements. So lessons learned from year one. So as we can see, what worked and what we need to improve on. So several things worked very well, from the simple UX, to the early privacy and security controls, to the accessibility guidance and SME review. So the biggest lesson is that AI success depends on operating model. As a technology, areas for improvement are clear. Content quality indexing requires continuous monitoring and tuning. And then we need to also make sure that adoption needs supervisor reinforcement and outreach regular reminders. Users need to be educated on what the capabilities that they can request through the AI tool. And then personalization is another area that we need to improve on as well, which includes pain conversations, reasonable questions, guided help, workflows for specific support. So the reasonable playbook is a stat to a clear business pain point. To validate with users, be able to source back AI, govern early, and also measure adoption and improve true release cycles.

[19:31](https://youtu.be/AAxtoXbmAn0?t=1171) So here's our roadmap. So the next phase is about operationalizing Mac AI as a sustainable product for us. So first thing we want to do is to increase adoption to a supervisor reinforcement, to our Mac weekly bulletin that is issued to our staff, to user surveys, and chat first habits prior to escalation. Secondly, we want to improve the content intelligence, TrueSmarter, mass.gov, which is our state website indexing. Also our Mac weekly update that is issued to staff as well. Summarization of data sensitive retrieval and data quality reviews. And then thirdly, we want to strengthen reporting and also scale a true KPI dashboards, productization for broader mass health and executive office of health and human resources. And then finally, we want to improve the quality of our data. So we want to enhance the user experience with bookmarks, hands, a guide on how to use the tool, FAQs, questions, and personalized labels as well. So the roadmap is both a product roadmap and an operating model roadmap as well. So the goal is not just to improve the Mac AI, but to create a usable patent for our Mac users. So we want to do that.

[20:48](https://youtu.be/AAxtoXbmAn0?t=1248) But to create a usable patent for responsible AI adoption. So this is just a demo of what our Mac AI looks like. But before we actually play the demo, this is a product for our staff where a plain English question is asked and an answer assistance. So the goal is to make the tool simple enough for our staff so they can understand. We don't want to create something that was more complex and hard for our staff to use. So staff can ask operational and eligibility related questions and then receive a response that is grounded in source content. So the product experience matters because adoption depends on usability, trust, and speed. So we're just going to play the demo right now and then I'll just talk to it and you can see it. So a question is being asked in the tool and it's going to provide a response.

[21:54](https://youtu.be/AAxtoXbmAn0?t=1314) And right underneath we have the thumbs up and thumbs down functionality. This is where our staff can let us know if the tool provided the right response or an incorrect response. And right underneath, if you don't mind just pausing for a minute, right underneath is the sources. That's where we have access where our staff can see where that information is coming from so they can view it. And all those documents are connected to our SharePoint where the tool is pulling information from. Please go ahead. Okay, so the tool is providing a response. Did we show the redacted information yet? If not, if we can just take it back so we can show how it was redacted.

[23:02](https://youtu.be/AAxtoXbmAn0?t=1382) Okay, there we go. So this is just showing how the tool redacts information. So a name was entered and the tool provided a response saying please do not attempt to share sensitive member information. So it redacts that information. As you can see, it's very fast in terms of the responses it's providing. And again, just the sources where that information is coming from and our staff have access to be able to view those documents as well.

[24:02](https://youtu.be/AAxtoXbmAn0?t=1442) And that's just our document lookup functionality. This is an internal feature where our staff can view any eligibility updates that has been issued to staff on a weekly basis. And if that information is not present at that specific date, our staff will be provided with the closest date and they can see what information was released. And this is just an admin feature that is not available to all staff. Just admins like myself and Beji can see how well the tool is performing. Where we can see the accuracy, relevancy, and similarity. Questions that have been asked, the amount of unique users. So it's very customizable in terms of a report that we're looking to pull.

[25:21](https://youtu.be/AAxtoXbmAn0?t=1521) That is correct, Patrick. Yes. So we work with our team L&D to make sure that information is updated in SharePoint because the tool is only as good as the information that is on SharePoint. So yes, we do work with our training unit to make sure that information is updated in SharePoint. And that leads us to our final slide. So this completes the story from ideation to adoption. So the MEC-AI started with a business pain point that was shaped by our staff discovery, which was mutual multidisciplinary delivery, governed through privacy, security, compliance, review, and adopted through a phased rollout. So we welcome any questions on the business problem, the rollout, the adoption, data and technology and governance, or the roadmap. Thank you so much for your time.

**David Fields**

[26:15](https://youtu.be/AAxtoXbmAn0?t=1575) Thank you, Tosin and Beji. This was great. I've got a couple of questions for you and that'll give our moderators a chance to kind of capture what's in the Q&A. But out of the gate, you know, the benefits here are real, but seemingly hard to quantify in sort of a traditional sense. And so I'm I'm thinking, you know, maybe reduce stress or more confident workers or faster resolution. How are you capturing that? How do you make the case for a tool like this when the impact shows up in worker experience rather than a clean metric for residents?

**Tosin Adebiyi**

[26:53](https://youtu.be/AAxtoXbmAn0?t=1613) Well, I can take that, Beji. So we rely on our staff a lot in terms of knowing how well the tool is performing. Our staff are not the ones to hold back. They're very vocal. And if they don't like something or if something is not functioning, they're not going to use it and they'll let us know. So based on the feedback we receive from our staff, we take that feedback and work on improving the tool. So we rely on them, you know, whether it's through us attending their weekly meetings or them providing the thumbs up, thumbs down functionality, giving us a review in there. You know, so we rely on them by providing us information and we take that information and

**David Fields**

[27:34](https://youtu.be/AAxtoXbmAn0?t=1654) work on improving the tool. That's great. And I, as you were going along, was sort of thinking as other states might be contemplating this, sort of what was harder than you expected when building and deploying this either technically or organizationally or with the staff who were going to use it? You know, what did you find that sort of was a bit harder than you expected

**Beji Joseph**

[28:03](https://youtu.be/AAxtoXbmAn0?t=1683) when you came at it? I'll probably take that question. So a lot of challenges in all areas you highlighted. Let's start with technology. AI technology is changing like by minute. A lot of new things are available. What you are familiar with today may not be the thing that's most popular tomorrow and capability is changing. So first challenge was to identify the right technology to use given our environment, what is available to us and our data and policies, what fits within that environment. So in our case, we had to choose AWS because of all of those criteria and filter that we applied. Now start building it organizationally. It's a government system. So we need to make sure we are adopting responsible AI and whatever we do, it has to be transparent. People should be able to see what we are doing and we have to make sure that we are not causing stress for folks. So because first thing people ask is, okay, what's the intention behind building this tool? Is it for replacing human beings or replacing workers or is it as an assistant? So we have to build trust, make sure everyone is on board in building this because we don't want stress from people who are going to use this thinking, hey, this is going to replace. No, this is built as an assistant. And technology, once we started implementing it, we had a lot of challenges, right? Learning this new, this was new for us. Learning this was a challenge. We had a lot of help from the Northeastern cohort, very smart students. So they were really helping us.

[30:00](https://youtu.be/AAxtoXbmAn0?t=1800) We ran through a lot of technical challenges. There were hallucination and related challenges that we had to address. We continuously tested and built a lot of guardrails around what it can search, what it can summarize, how it can respond. So a lot of help from Northeastern students in building it the way we have it today. And then rolling out was the other challenge. With any AI tool, people are skeptical, especially at work. When you bring an AI tool, people are very skeptical. The usefulness of it, how good is it, how bad is it? So we had to be very careful. We didn't want to just roll out this tool to everybody and then if it doesn't respond the way we expect, people will not go back to it. If you ask a couple of questions, if you get wrong response, you will never go back. And you will tell everybody, oh that AI tool, that doesn't work. So we did not want that. So we partnered with the business team.

[31:03](https://youtu.be/AAxtoXbmAn0?t=1863) We formed a pilot team who would help us, who would test this for us. Very controlled UAT, small group of folks who knows this is not really a perfect tool, who were willing to partner with us and help us improve, identify problem areas. So a lot of people from Toswin's team helped us test this thoroughly, ask all sort of questions that people will ask on the field. And then we improved from there. So we fine-tuned based on all the feedback we collected and that really helped. And then we did a phased rollout. So we added one location at a time and rolling it out. So that really helped. So challenging all the way, but great problem to have

**Tosin Adebiyi**

[31:52](https://youtu.be/AAxtoXbmAn0?t=1912) as well. And I'm just going to add to what you said. We had over 200 test cases and every time we had a release, we created more test cases just to make sure that the system was functioning

**David Fields**

[32:04](https://youtu.be/AAxtoXbmAn0?t=1924) correctly. That's great. I mean, I think it's impressive, you know, one in four residents in Massachusetts is using MassHealth for their primary health insurance. So it's the volume that your team is working to support. I mean, the need for tools like this to be able to surface information and to know it's accurate. There's a lot there. So kudos to you all. I have one last one and then I'm going to turn to our questions from our audience here. And that's really for participants that are listening today that are intrigued by what you've done. They're thinking about bringing something like this to their agency. What would you suggest are the most important things to get right before you launch? And what might be signs that you're not ready?

**Beji Joseph**

[32:57](https://youtu.be/AAxtoXbmAn0?t=1977) People who want to roll this out, I would say putting the governance in place up front makes a huge difference because you need guidance from your privacy team, your digital council, your security team, compliance team, or your user base. So think of this more as a business solution with AI or technology assistance. Don't build this out like your traditional software and then figure things out as you go. But if you have the support structure in place, the governance in place up front, that makes rolling it out easier. And even when you think you have it ready, use a small group of users who would really test it, push it to the boundaries, and roll it out slowly. Do not roll out to everyone at once. If you want more adoption, definitely take one step at a time. That's the lesson that we learned.

**David Fields**

[34:09](https://youtu.be/AAxtoXbmAn0?t=2049) Great. So we have a question here from Misty Newell. What were the biggest challenges experienced with the implementation? And what were the lessons learned? Sort of what would you have done differently or changed a second go around?

**Beji Joseph**

[34:30](https://youtu.be/AAxtoXbmAn0?t=2070) I can take from the technology side of it. Tosin, if you want to add anything, please go ahead as well. Technology-wise, two lessons for us when we rolled out. One, we did add guardrails, but we realized as the real users started with the pilot and the UAT program, we found it does hallucinate more than we expected. So we had to add more guardrails, make sure that it only reads from the content that we approved. And we forced it to give us links to any source it is using. So that way, the worker, when they get the response, they're not just reading off of the summary that we give. They are responsible for looking at the links and make sure the AI, the response that they are giving to the constituent is something they would sign off, right? So that the actual response to the member is the worker's responsibility.

[35:36](https://youtu.be/AAxtoXbmAn0?t=2136) So we made sure that the chatbot is citing all the sources it is taking the data from, and it will not leave that source. It will not go search on the internet, for example, and come back with the general information. So that was one part. And then, yeah, so the people who use it, so the workers, when we look for enhancements, we take feedback from the workers or the users to build on what we want to build next. Because this is new, they did not know what kind of features to ask. So they didn't know what it is capable of, or they didn't know what can they ask for. So we had to sort of train them to look at different ways, and then encourage them to request features. So there was a little bit more collaboration required than what we typically anticipate from regular software releases.

**Tosin Adebiyi**

[36:39](https://youtu.be/AAxtoXbmAn0?t=2199) I completely agree, Bhaji. And we used to, you know, working with our staff also to so they have a better understanding of how the tool works, you know, the type of questions that you can ask in the tool. I had one of our teams reach out the other day and tell me that they asked the same question, and the tool gave a different response. So I did ask them, well, can you provide me with the question that you asked? And lo and behold, it was two different questions that they asked, even though the question still had one or two different answers. So I think it's just a matter of just training our staff to understand how the tool works. And the tool is only as good as, you know, you have to also make sure that the information it's providing, you're validating that to make sure it's correct. You know, and we rely on all of our staff to make sure that the information is correct. So I think it's just a matter of training our staff to understand how the tool works, and the tool is only as good as, you know, and we rely on other teams as well to make sure that where our tool is pulling information from, those documents are updated. So it's a constant reminder to all teams involved.

[37:51](https://youtu.be/AAxtoXbmAn0?t=2271) Beji and I meet regularly to go over the tool to see how well the tool is functioning. We take reviews from the team based on their feedback, and then we'll look at, okay, what's actually going on with the tool? Why is it taking X amount of time to provide a response? Is it that those questions are not in SharePoint? You know, so we look at a lot of things, but again, it's all coming from our staff. They tell us how well the tool, but we need to do a better job in terms of educating them so they understand how well the tool works. That actually aligns well with the

**David Fields**

[38:26](https://youtu.be/AAxtoXbmAn0?t=2306) question that we got around kind of how frontline staff have responded to the new tools being introduced and, you know, if there were concerns about trust or, you know, job security or workflow changes. And so it sounds like that has evolved, but curious if there's other elements there, especially around sort of you hit upon this a little bit, Beji, in terms of like not even knowing what it's capable of, right? So not quite knowing what to ask for. But how about the other pieces around the trust or feeling like maybe this is going to replace me or job security?

**Tosin Adebiyi**

[39:06](https://youtu.be/AAxtoXbmAn0?t=2346) Yeah, I can talk about that from a business perspective. So, yes, we hear that all the time and we just have those conversations with them, like, okay, something that is taking you 10 minutes prior, now it's taking you less than five minutes and you still have your job and we're still hiring more people. So AI is not here to take your job, it's just to enhance what you're doing. It's to make it better. And it's not just about talking about it, it's actually showing them what it's capable of doing. And it can be done in a safe way, you know, as long as we put guardrails in place where we're protecting PII data. So it's a matter of education and having those conversations with staff and letting them know what AI is and how we can use it to better enhance our operational activities with our members in a safe way, obviously. Right. And actually a question came up

**David Fields**

[40:05](https://youtu.be/AAxtoXbmAn0?t=2405) around that piece of engagement. So any advice you might have on how to promote actionable feedback, like to support continuous learning? One of the questions that kind of came along with it is if staff are asking questions that they don't know the answer to, how can the staff, you know, verify whether the answer is correct or not? And so it seems like there's kind of two pieces to that, and I'm curious how you've handled those. Absolutely. So, oh, sorry, Basia. Go ahead.

**Tosin Adebiyi**

[40:44](https://youtu.be/AAxtoXbmAn0?t=2444) So we do have our EOMs, which is Eligibility Operational Memos and other updates that goes out to staff on a weekly basis, you know, telling them if we have any eligibility updates or any systems updates. So those goes out to staff regularly. That's aside from that information being in our SharePoint, where they also have access to, and the tool is pulling from SharePoint. So if they go into the tool, ask a question, the tool provides a response. What we always tell them is yes. Now, can the tool maybe give an incorrect response? Yes, there's a chance that it may do that. So we always tell them, you know, you need to look at the sources. Where is the tool pulling that information from? Validate that information and see to make sure it's correct, because sometimes it may be like maybe we haven't had the chance to update the guidance in SharePoint because the tool is only going to pull old information if a new current information is not in there or something that trumps the formal directive that was put in there. So we always encourage our staff. We have weekly meetings. Teams have weekly meetings at the enrollment center every week to go over any changes from operations, privacy, policy for that week. So they're aware of changes, but the only problem is that there's a lot of information. That's where the tool comes in place, where the tool can just summarize it in plain English and they can get that information versus having to search multiple sources and getting that information. So it can be a little challenging, yes, in terms of validating, but on a weekly basis they have meetings, you know, to talk about recent updates that are coming out. But we always tell them, do not rely 100% on what you need to also do your due diligence by checking the sources, make sure it's correct. That's where that human in the

**Beji Joseph**

[42:43](https://youtu.be/AAxtoXbmAn0?t=2563) loop becomes crucial, right? Because it's not, AI is not answering directly to the members. The workers are really center of this whole solution. They are responsible for looking at the sources in the dropdown, read the responses, and also they have their, all their existing sources are still there. Their reference materials are still there. Their supervisors and peer support is still there. And they are still getting trained on these policies and procedures like they were doing before. So all of those still exist. So that's where the human in the loop become crucial in that aspect. We also tell the workers if something is wrong, do that thumbs down and give a comment on what went wrong. So we then capture what question was asked, what response was given, and what is the response from the Mac workers. And Tosin and I meet every week or every month or more based on the frequency that we need. We meet very frequently and then we review all of these and we identify, is it a missing document or is it the way they are asking the questions? For example, the chatbot works better if you ask it as a human readable question, like you would ask your peer, how would you phrase it? Ask it the same way.

[44:06](https://youtu.be/AAxtoXbmAn0?t=2646) Don't do the Google type of search where you do a keyword search, just type one word and you get 500 documents and then you scan through all of it to find the answer, right? This works better if you ask it as, phrase it as an English question, then the chatbot responds in a much better way.

**David Fields**

[44:24](https://youtu.be/AAxtoXbmAn0?t=2664) So we train them on that too. There was a question, you mentioned sort of the references and I'm going to kind of put two questions together into one. So there was one question about hallucinations. And so I know you mentioned it's kind of using the source of truth through SharePoint, but there was also another question about how customizable the output code is. And you know, for instance, could you have the sources or the references indexed right into the output? As opposed to down at the bottom. Do you have flexibility in terms of some of those pieces?

**Beji Joseph**

[45:07](https://youtu.be/AAxtoXbmAn0?t=2707) That depends on the implementation. What we chose to implement is give a summary. So you do this question, it does the search, it may come back with 10 documents. In the past, if you were searching, you would come back with 500 documents on the SharePoint every day, right? So this now comes back with five documents, six documents, a smaller set. LLM is summarizing based on all the content in that. It's understanding what's the question, what's the most relevant response. So we tried to make it simple for the worker by providing that summary there. And then all the source material is given in the dropdown because some of the documents could be like 30 pages long or 40 pages long. And then you have four or five of such documents, it becomes huge. And then it becomes harder for the workers to process it. Because if you see like 70 page response, you'd get tired. You don't want to use it. You want to give them a couple of paragraphs. So they understand the gist of what they want to respond. And then they have the sources to go back and

**David Fields**

[46:14](https://youtu.be/AAxtoXbmAn0?t=2774) reference. That's great. All right. Last question, because I know you gentlemen have to run. Just kind of why, how might you weigh building a tool like this internally versus buying from a vendor externally? Pros, cons versus building it yourself or off the shelf?

**Beji Joseph**

[46:39](https://youtu.be/AAxtoXbmAn0?t=2799) I will tell you my selfish reason. It was very cool. I wanted to jump right in and learn. We had support from Northeastern students. That was a great program. It benefits both. And we have a lot of our own requirements. So it worked best if we build something with this partnership. After we built this and when we went to some of the conferences, a lot of the leading vendors were still figuring this out at that time. They were taking notes from this implementation. We found that very interesting. We now see a lot of vendors providing very similar tools or even with additional capabilities that they're building into it. It's a very repeatable product type. Anybody can implement this. And it's very usable because all you have to do is point this to a SharePoint or your internal intranet. So it has known sources. And then it can search and respond in English. So that's a common use case, not just a product support perspective, even for internal development teams, product teams, your sales and support teams. They all have knowledge bases. Or say in the development organization, if a new person comes in, they want to learn about the product team that they are joining. If we have a tool like this internally, that can look at all the product documentation, give them guidance. And this is what the product does. This is how it works. This is what you want to do if you are developing. And all of those can be in that knowledge base. Very, very handy tool.

[48:23](https://youtu.be/AAxtoXbmAn0?t=2903) So I would recommend everybody to build something like this. And a lot of these AI tools make it very easy to implement these days. So definitely, you should look at that as an option. And the other part of it is pre-built solutions. If you buy that, support cost and the upfront cost is often much larger. But if it is part of a bundled solution, sometimes it works cheaper as well. But at the time we implemented it, this was our best option.

**David Fields**

[48:59](https://youtu.be/AAxtoXbmAn0?t=2939) Well, thank you, gentlemen. This was really, really exciting to see and appreciate you sharing all the great work you've done. I really appreciate you joining. For all of you with us today, over the past three weeks, we've covered a lot of ground from a well-designed resident-centered contact center to what it takes to modernize one from the inside to what's possible when you put the right AI tools in the hands of the people doing the work. What Beijing and Tocin shared today is a powerful example of really what's possible when the government is willing to try something new and when students who care about public problems get the chance to build real solutions. We'll be sharing more information about the AI for Impact program. Please don't hesitate to reach out if you're interested in learning more. Thank you to our speakers today, to everyone who joined us across all three sessions, and to the work you're doing in your own agencies. We hope you leave with at least one idea you can act on. I'll now turn it over to Bonnie to close out the workshop.

**Bonnie McGilpin**

[50:08](https://youtu.be/AAxtoXbmAn0?t=3008) Thank you, David and Beijing and Tocin and all of our participants who joined us today for this great conversation. Please fill out the post-workshop survey. It'll pop up on your screen when we close the workshop and suggest topics you'd like us to cover in the future. We use your feedback to plan future workshop series. This is the final session in the series. Thank you for those who joined us across all workshops. Please join us in the fall for a new lineup of workshops. Christine is adding the info in the chat. With that, we're ending this workshop. Thanks, everyone. Bye-bye. Thank you so much.

* * *

Original page: https://innovate-us.org/the-ai-solution-smarter-tools-for-public-call-centers · Licence: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org
