---
id: "workshop:modernizing-unemployment-benefits-a-hands-on-workshop-for-states"
type: "InnovateUS workshop"
title: "Modernizing Unemployment Benefits: A Hands-On Workshop for States"
url: "https://innovate-us.org/modernizing-unemployment-benefits-a-hands-on-workshop-for-states"
date: "2023-09-20"
duration: "90 minutes"
language: "English"
instructors: ["Jill Gutierrez","Dave Cole","Giuseppe Morgana","Laurie Harrington","Noah Marcus","Janice Cho","Naman Agrawal"]
recording: "https://www.youtube.com/watch?v=S7ECHtu0smM"
transcript: "automatic, from the recording"
source: "InnovateUS (innovate-us.org)"
licence: "CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org"
---

# Modernizing Unemployment Benefits: A Hands-On Workshop for States

InnovateUS workshop · 20 September 2023 · 90 minutes

With the reduction in federal funding for unemployment modernization, it is more important than ever for states to adopt more cost-effective and agile ways of providing unemployment benefits. In this 90 minute virtual training with leaders from the New Jersey Office of Unemployment Modernization and the New Jersey Office of Innovation, participants learn how New Jersey moved to an agile, human-centered approach to improving benefits, resulting in 48 minutes saved per application and an increased percentage of unemployed workers who successfully receive benefits. Participants will learn about the state's approach to making short and long term modifications that enhanced performance and helped the state continue to deliver improvement despite exploding numbers of unemployment insurance applications during COVID-19. Join us for this interactive how-to design session designed for public officials looking to improve their unemployment or other benefit systems.

## Learning goals

- Learn how New Jersey moved to an agile, human-centered approach to improving benefits resulting in 48 minutes saved per application.

-  Learn about the state's approach to making short and long term modifications that enhanced performance.

- Hear about the approach which helped the state continue to deliver improvement despite exploding numbers of unemployment insurance applications during COVID-19.

## Speakers

- **Jill Gutierrez**, Former Senior Advisor/Director, New Jersey Office of Unemployment Insurance Modernization.
- **Dave Cole**, Chief Innovation Officer, State of New Jersey.
- **Giuseppe Morgana**, Senior Advisor, New Jersey Office of Innovation
- **Laurie Harrington**, Assistant Director of Evaluation, Heldrich Center for Workforce Development, Rutgers University
- **Noah Marcus**, Technology Operations Engineer, New Jersey Office of Innovation
- **Janice Cho**, Design and Product Expert and Consultant, New Jersey Office of Innovation
- **Naman Agrawal**, Director of Engineering, New Jersey Office of Innovation.

Recording: https://www.youtube.com/watch?v=S7ECHtu0smM

## Transcript

Automatic transcript of the recording: names and terms can be misheard, and the times can be a few seconds off. Each time links to that moment of the video. The transcript alone: https://innovate-us.org/modernizing-unemployment-benefits-a-hands-on-workshop-for-states.txt (plain text) and https://innovate-us.org/modernizing-unemployment-benefits-a-hands-on-workshop-for-states.json (every segment with its start and end second).

**Dave Cole**

[0:00](https://youtu.be/S7ECHtu0smM?t=0) Thank you, Hannah. Thank you, Hannah, and to the team at InnovateUS for pulling us all together. It's extraordinary to see the turnout here. Folks from New Jersey, across the country, and I also noticed some of our friends internationally as well, so thank you all for joining. What I just wanted to say before we get started here is just on the side of how we'd like to run the sessions here. This was really inspired in coming out of a presentation at the Heldrich Center a couple months ago, which focused on some of the high-level themes and strategies of modernization that we've been using here in the state of New Jersey, and what we hope to do in this session is extend that into some practical hands-on examples of what some of those changes look like and talk about how we were able to get to those, so we've got a panel of folks here to do that. I want to also just acknowledge a couple folks I saw who joined from New Jersey here who have been instrumental in this work but are not speaking today. Chris Ryan, our state CTO, welcome, Chris. Always good to see you, and thank you for you and your team's support that has enabled this work. Julie Diaz, Deputy Commissioner of Labor at the Department of Labor, obviously one of the key partners in allowing us to do things a little bit differently and try to make some changes here for what we've been able to do, so as we're going through this, we have a couple sessions that we are parts of the session that we'd like to present to you some of the work and then have a discussion alongside, so if you have questions, please feel free to drop them in chat. I'll try to facilitate, ask the questions of each of the speakers so that we keep those in context for each section. We can also, if you have something that you want to say verbally at the end of each section, feel free to raise your hand or otherwise we can facilitate conversation that way as well, and we'll do introductions along the way too as we go so you get to know some of the folks who are here and their backgrounds. I think without further ado though, I will hand things over to Jill Gutierrez, who is the Senior Advisor at the Department of Labor in New Jersey for Unemployment Insurance Modernization. Jill has experience both at the federal and state level now and has been for several years leading the effort at the State

[2:01](https://youtu.be/S7ECHtu0smM?t=121) of New Jersey from within the Department of Labor to be able to change some of the existing systems with a particular focus on claimants' needs and the responsiveness to the feedback that we get through the conversations we have with users. So Jill, I will pull up slides here for you and hand

**Jill Gutierrez**

[2:18](https://youtu.be/S7ECHtu0smM?t=138) it over to you if you'd like to help us frame the conversation. Thanks, Dave. Thanks, everybody, for being here. It's really an honor to speak with you and to have so many friends and colleagues here. I'm just gonna set the tone for sort of how we got to the place that we are in. For those of you who are from New Jersey or those of you who work on UI, I'm not going to say anything groundbreaking, but for those of you who come from a different context, perhaps it is of interest. So the pandemic in New Jersey really showed us two things. It showed us, well a few I will say, it showed us how inflexible our technology that served UI benefits was, right? So how immobile it was. It also showed us that we had a long way to go in terms of thinking about government technology as a way to serve human needs. And I think those two lessons are important when you're thinking about why did New Jersey, why did states have a lot of challenges in trying to meet the legal requirements that the pandemic created for us? Dave, you can go to the next slide. Thank you. So the thought of, right, we had all of these laws, all of these regulations put on each of the states trying to implement them on technology that wasn't very flexible and in systems that perhaps already weren't built around what a human actually needed at a time of crisis, personal crisis or national crisis. And I would say the third thing that it really showed us in New Jersey was how dedicated, how hardworking, how creative our own staff were.

[4:02](https://youtu.be/S7ECHtu0smM?t=242) So the staff working in New Jersey, either on the UI program area side, the ones like understanding the policy, understanding the process that had been followed, as well as on the IT side, imagine, you know, keeping the system running, one, but also thinking about what were the ways we could sort of hijack the system as it exists in order to make these new policies, procedures, these new requirements from the federal government actually work in our older system. So all of this put together and the sort of creativity that the staff showed and leadership showed are the reason why New Jersey went forward on trying to imagine what would it take to build a claimant experience, a new intake, a way to receive applications that reflected a modern technology, a modern base to build on, as well as the human needs at the moment that people were in this crisis. So these are the lessons that we hope are reflected in the strategy that we're working on today. Next slide, please, Dave. So what is New Jersey doing?

[5:09](https://youtu.be/S7ECHtu0smM?t=309) New Jersey is taking an agile approach of thinking about our unemployment system, reimagining it in a different way. And you can read some of the things here that we're working on. Nam and Janice and Giuseppe will talk a little bit about some of each of these examples. But we're really looking about how do we make constant improvements for the claimant? How do we listen to what the claimant, the worker needs and make those improvements, giving them new value every step of the way? Like maybe those new values aren't explosive big changes, but we're working on delivering things constantly. And at the same time, understanding we do live in a mainframe system, we do live in an inflexible place and working on what is the strategy and how do we start picking apart that mainframe to create that more flexible base on which the system can be built and the system can be constantly reimagined as both worker expectations change, employer expectations change and just society's expectations of what technology is supposed to do to deliver benefits change. And the U.S. Digital Service Playbook is part of how we make those decisions about what are the strategies that inform us and how do we really break apart the different changes that need to be made in order to deliver value quickly. And with that, I will pass it over to Lori.

**Dave Cole**

[6:34](https://youtu.be/S7ECHtu0smM?t=394) All right. Thank you, Jill. I just wanted to say at the outset here that Lori Harrington is the Assistant Director for Evaluation at the Heldrich Center of Workforce Development at Rutgers University. I have to imagine the reason many of you are here is because of all of the effort that they have put into studying and helping tell the story of what's going on in New Jersey. So thank

**Laurie Harrington**

[6:52](https://youtu.be/S7ECHtu0smM?t=412) you so much, Lori, to you and your team for that. The floor is yours. Thank you. Thank you, everybody, for the opportunity to be here today. Like Dave said, my name is Lori Harrington. I'm the Assistant Director of Evaluation at the Heldrich Center for Workforce Development at Rutgers University. And I'm here today on behalf of the Center's Director and the Distinguished Professor, Dr. Carl Van Horn. At the Heldrich Center, our mission is really threefold. It's to assess and identify workforce best practices. It's to empower job seekers through technology. And it's to transform the workforce through research. And today's workshop really speaks to all three of our core missions and the reason why we focused our research efforts on the UI modernization process in New Jersey. For more than 25 years, the Heldrich Center has aimed to understand, create, evaluate, and implement policies and programs that help people look for work and prepare for jobs. In our work, we focus on career and technical education, data analysis, disability employment, older workers, unemployment and reemployment, and workforce policy, among other research areas.

[8:05](https://youtu.be/S7ECHtu0smM?t=485) Naturally, unemployment insurance is an integral part of our research focus because it's a critical element of the American social policy network. It's the emergency life vest for workers, their families, and their communities when job loss strikes. The critical research question for all of us today is, how do states deliver UI benefits to workers who are entitled to receive them in a timely way? As we all know, during economic recession, this has always been challenging as unemployment swells and employers lay off workers. This became exponentially more difficult during the pandemic and the resulting economic crisis. And although it was welcome policy, it was even more complicated by the addition of the supplemental insurance authorized by Congress. As states like New Jersey confronted this crisis, they were faced with legacy computer systems that hadn't been updated for years or even decades. So with that understanding, it was truly exciting for us to partner with Michelle Evermore, a leading national expert in UI policy in the United States, with support from the Families and Workers Fund, to document and showcase New Jersey's experience at modernizing its approach to administering UI benefits, an approach which aimed to create a more agile, customer-focused process and one that had excellent results. So we're grateful to the Families and Workers Funds for supporting this partnership and our research and for the tremendous work of the New Jersey Office of Innovation and the New Jersey Department of Labor. We're happy to be here today to continue our participation in this conversation and to observe this hands-on workshop to get a better understanding of how New Jersey tackled this UI modernization process and to get a window into the challenges and opportunities for improvement and advancement where possible in other states.

[10:00](https://youtu.be/S7ECHtu0smM?t=600) When we presented our research back in June, state representatives from all over expressed a need for more ongoing peer learning in this space, so this session couldn't be more timely. So thank you for inviting the Heldrich Center to this session and I'm really looking forward to it. And if it's all right, I can drop the report in the chat for everyone in case they haven't seen it. Okay, I'll do that.

**Dave Cole**

[10:25](https://youtu.be/S7ECHtu0smM?t=625) Thank you. Sounds great. Yes, please do check that out. Thank you, Lori, again for all of your support and for being here today to help us kick things off. What we wanted to do in the next few sessions here is take you through a couple stages of problems that we, in approaching modernizing unemployment insurance, face and at least our perspective on how we work through those. The first, I think, probably that most of us are thinking about is how do you get a team that has the experience working in human-centered design, in agile software development, but also the expertise to know how to work with a complicated system like UI. So Giuseppe Morgana is the founding digital director for the state of New Jersey, helped put together the part of the innovation team that I and the other presenters all have the privilege of being a part of and working closely with the Department of Labor here. He has since moved to the private sector but has joined us again to help share some of the strategies that he helped pioneer here at the state and put in place that enabled us to build the team and find the talent to be able to do the work that you're about to

**Giuseppe Morgana**

[11:29](https://youtu.be/S7ECHtu0smM?t=689) see here. So Giuseppe, thanks. Perfect. Thanks, Dave. And excited to be here and it's awesome to see all of the faces here. You'll see some of them mentioned in a couple of the slides that I'm going to share as I talk through here. My big message before I even jump in here is there's way too much to cover in just a few minutes, but if there's something that is exciting here or you see an opportunity to take hold of the strategies that we used here in New Jersey, please reach out to us and we're more than happy to set up time and kind of dive further into the details. So the big message here that we often hear and we've been on many calls on this is we just can't get the talent or resources needed to pull this off. And that often is kind of a starting point of how do you actually pull together the right team, the right expertise, and set them up for success. And so what I want to talk to you today about is how we went about working with folks in the state to actually use some innovative methods to actually procure agile development talent that was then deployed not only to the unemployment insurance project, but we've also used these strategies for other projects in the state.

[12:42](https://youtu.be/S7ECHtu0smM?t=762) And there's sort of three kind of like key takeaways that I'll start here that I want to share, and I'll dive into each of those. The first is partnering with experts in your government, especially the first time. Second is don't reinvent the wheel, accept where you need to. And the third is project manage the process, being flexible along the way. So let's just dive into the first one. So the first time that we did a procurement to say, hey, we need talent in the state. We can't just do it with the folks that we have. We need to scale how many people are working on this project. There was a lot of trying to figure out how we're going to even start. But the key to that was actually partnering with the experts. You heard that the CTO for the state is on this call. He was a critical person in actually working with us to understand the mechanisms that were available and making sure that we applied them to this project. Same with the deputy director of procurement agency leadership and then folks from the innovation team, both in terms of those that were dedicated to the project and those that came with expertise in engineering, design and product management. So the key here for us was there is no way we could have predicted everything that we would have had to do. But having a group of folks who work together, understand the goals and like trusted each other to be able to work through this process allowed us to overcome all of the uncertainty and get to the point where we actually were able to have a team on the ground. And so that's kind of like the first big message and an obvious one, but one that was critically important to this process. The second was don't reinvent the wheel. And this one is a big, every time I say this, I say thank you to a ton of people who we turn to for help in making our process work. So we looked at federal documents, we looked at documents from other countries, we looked at other documents that had been done in the state, but the key for us was don't reinvent the wheel. And the thing I will give to you as part of this is if Dave can post a blog post that we had written, it will include links to a bunch

[14:47](https://youtu.be/S7ECHtu0smM?t=887) of those materials that we used and then the actual procurements that we did in the state that you can actually use for your own procurements if you'd like. And we'll talk a little bit about that in a few seconds here. But the key things here... For us, in terms of not reinventing the wheel, we're understanding what contract mechanisms were available to us. For us, it wound up turning out that we can buy off the GSA schedule. It did involve a process. We did a procurement, we solicited bids, but it provided a way for us to have a streamlined access to a pool of talent that had already been curated at the federal level. And so being able to use that as leverage to be able to find the right talent for us wound up providing a ton of benefits and allowed us to move a lot quicker than we would have normally. So the first is looking for those types of mechanisms like the GSA schedule. The second, as I mentioned, was finding templates and sample materials. The actual procurement itself was close to 100 pages, if I remember correctly, but a lot of that was just understanding what do we reuse? What do we reuse from the state? What do we reuse from other entities? And then what is unique to us? Again, I'll talk about that in a second. The third is reaching out to us, right? Like, please, please, please do send a note to the folks on this call, and we're happy to spend time kind of diving into it. And the fourth was make sure that you start with your standard government contracts and then layer in the domain-specific materials. What this allowed us to do was to really focus on the areas that were different as opposed to kind of like toss everything out. And so that was part of the reason we were able to move quickly. So to kind of summarize that last point here, when we actually put the procurement together from the government, we had boilerplate terms, we had standard terms and conditions, and then we had some sections that were mostly adapted for our project. So the background information, the technical environment, a sample set of user stories that explain the type of work that we would expect that talent to come on board and do, and then mostly copied from other

[16:52](https://youtu.be/S7ECHtu0smM?t=1012) examples like the federal level and other states were things like scope of work, performance standards, technical submission requirements, and the price submission template. Again, like, we will give you all of that as part of what we did, and hopefully you're able to reuse some of that. When we actually, I want to double click now on some of the more specifics that I think beyond just the process that was really important for us, is that when we were actually evaluating candidates and folks who were submitting, we use these four components, right? It was technical approach, staffing approach, similar experience, and price. And the first three came first, and then price is layered on at the end. Now, the first three are really important because we wanted to hear how are you going to actually approach these projects? Are you going to follow the methodology, the agile methodology that we had set forth? And how are you going to specifically apply it to the user stories and examples we have given you? The second was actually learning about, will they actually adapt to our needs? What are actual resumes from the key personnel? So not just a narrative here, but like, what are the skills and experience of those key folks that are going to come on the project? And then lastly, and I think this is kind of most critical, but also most time consuming, is we want to see the similar experience that these companies have had in the past, and particularly ones that might be in the same sort of method, agile method. And we want to actually receive the coding, design, and product samples that they used on those projects. So in our case, we would receive actual code samples that our engineers would look at and apply very standard rubric against to be able to make sure that we were not only just hearing the right words, but also seeing how they completed that in practice. And then finally, and I know I'm flying through this here, but we'll take questions in a second if there are any, is project managing the process was absolutely critical. So the first time we did this, we

[18:53](https://youtu.be/S7ECHtu0smM?t=1133) wanted to make sure it wasn't just, hey, yes, it's the first time we're doing something new. We are figuring things out. Yeah, there are things that we are going to figure out along the way. But what we did was we documented everything we knew. And then as things change, we updated those. We worked with the folks on the panel to make those changes. But for the most part, everything had a flow, and everyone had a set of expectations regarding what should be consistent here. The scorecard, for example, was done up front, factoring in that information I just shared you. The calendar, and this is one that was highly important, was we set out a very transparent calendar for folks to be able to understand when does the bid submission close, when are folks going to be actually expected to review those documents, when are we going to do group reviews, when are we going to actually open up the Q&A, when are we going to respond to the actual applicants with our decision-making. And by doing that, we were able to consolidate the calendar. And honestly, the initial response sometimes is, how are we going to do all of this in a dedicated time? But by planning up front, we were able to do what we normally stretch out over months in a couple of weeks and have a very high-quality process because everyone kept the context. We didn't lose context over the months. Everyone remembered the context of the discussions and were able to move very quickly. And so I would highly encourage being able to follow that process, regardless of the type of procurement that you're doing.

[20:26](https://youtu.be/S7ECHtu0smM?t=1226) Finally, and this kind of comes back to the original point of experts, is don't just set up individual time for evaluations, but we set up actual group time so that we could come together and evaluate the actual submissions together. And it allowed people to have dialogue that really pushed our understanding of the submissions. And then you had experts in the room who can kind of chime in on their area of expertise that helped inform the overall procurement. So there's a lot there. I just wanted to touch upon some of the key things that we kind of followed, but I'll pause there if there's any questions.

**Dave Cole**

[21:05](https://youtu.be/S7ECHtu0smM?t=1265) Thank you, Giuseppe. For folks who have questions, feel free to raise your hand or unmute or drop them in chat if you'd like us to read them out. And I'll just reiterate what Hannah has put in chat if you hadn't seen it. We will send the slides out afterwards if folks have that reference available, as well as some supporting links, some of the blog posts and other resources that folks are mentioning in the presentations will be all part of an afternoon here. Great. Thank you. I'll keep us moving here. And then again, if you have questions, drop them in chat. We can always come back as we come to the next section. So thank you, Giuseppe, for that section here. Moving into our next area, we want to now have two examples of real concrete things that were designed, built, tested with real users and implemented in the state of New Jersey. And the first will focus on the communications and the notices related to the Unemployment Insurance Program. To lead this work and to present is Janice Cho, who is a designer who has at least 15 years of experience in very similar work, including working on other benefits programs in other states.

[22:13](https://youtu.be/S7ECHtu0smM?t=1333) We are very fortunate to have Janice working with the state of New Jersey now for a little over a year and a half to help us study not only how to improve communications in unemployment insurance, but also start to take some of those lessons and expand them to other benefits programs as well and she's joined by Noah Marcus, who is an engineer on our innovation team who has similarly been working on behalf of the unemployment insurance modernization efforts here. So Janice, I'll hand

**Janice Cho**

[22:37](https://youtu.be/S7ECHtu0smM?t=1357) it over to you. Thanks. Can everyone see my screen? Yeah. Okay. Yes. All right. So we'll just get started. All right. So within user-centered communications, I wanted to just set the precedence of what we're doing and why. The main reason why we're even going as far as creating all of these designs is because government communication has not accounted for the needs of claimants and employers. And as a result, it jeopardizes the ability for claimants and employers to be successful in their unemployment insurance journey. And so I wanted to show a couple of the ways that we've been trying to make this experience more successful within New Jersey. I wanted to share a couple of quotes. You'll see the quotes at the end of what people are saying after our design, but because we wanted to focus this around the people that it's serving, we definitely wanted to show those quotes of how people are experiencing the communication before and after. So before, these are the quotes. This makes me want the opportunity less. It's a reminder to me that they need to incite control. It's violent. It doesn't feel like they're here to help. And it took me a full day to understand it, and I sent it to someone else to understand it. So I'll be sharing the other flip side of this once we get through all of the principles. But I wanted to talk about three things about what we considered, and then we'll show you exactly how you'll be able to use this as well. Action-forward language, visceral reactions, and selective filtering. So we'll start with action-forward language. I know that everyone here has probably heard about plain language, but we wanted to just kind of push plain language a little bit farther into this action-forward language for a couple of reasons. One, it taps into mobilizing

[24:39](https://youtu.be/S7ECHtu0smM?t=1479) the reader. So a lot of our notices are requiring our claimants or employers to do something in order for us to process something else, or there's an action that they need to take. And so as much as plain language is allowing people to understand, just pushing it a little bit farther into this action-forward arena really helps people take action. And the way that we kind of craft action-forward language is that it uses active verbs and it also deletes filler language. Active verbs meaning it starts the sentence with complete, submit, log in, words that people can kind of get behind understanding and feeling the action that they need to do. And delete filler language is a little bit more of a permission, if you will, where our community and our public has given us permission to take away our pleases and thank yous and, you know, like all of this stuff that we like to kind of put in there to be more formal as an organization. They just want to get us, they want us to get to the point, right? And so it reduces the amount of content. Actually at NASWA last week we just saw a statistic on how the second or third most used word within notices within a particular state was the word please. So there's something there about taking away filler language. The third one is that it repositions the interaction experience of the content to tie closely to success. So a lot of times when we're writing plain language from a particular notice it's either done in an office with people who are, you know, policy folks or those who are trying to just focus on communications. But when we are trying to pull in the claimant and really interact with their service journey along their claimant or employer pipeline, it really ties us to their successful journey of the things that they need to do. The next one is a little bit abstract,

[26:41](https://youtu.be/S7ECHtu0smM?t=1601) but I want to get through this because I want to kind of prove why the design of the way that we have crafted our emails, for example, and just our paper as well, means so much. But if this was an audience where I would ask a question, I would say if anybody knows this designer, this is a very famous lemon squeezer from Alessi. And although it's incredibly functional and very efficient, there is a very creepy vibe to it. I can't see many of your faces, but I can sort of sense the creepiness through the internet, right? And so when we feel that creepiness, even if we interact with this thing, sure, we can interact with it, but we're going to be pretty nervous or we're going to be on our tiptoes like wondering, you know, it's just not going to feel great, right, while we go through the process. It might also inhibit us from going as fast as we can because of those feelings. So what I'd like to just propose is that the way that we present ourselves within our notices and communication, it really sets the tone. There's a lot of workers, our agents do an incredible job, you know, trying to represent the energy of the state and trying to provide that great customer service. And so what really great design will do is that it'll set the tone, it'll also, oops, it'll determine future interaction. So if you meet a friendly face, the next time you go in there, you'll also kind of expect a friendly face and you won't expect kind of horrible service, if you will, right, if you walk into any sort of service model.

[28:15](https://youtu.be/S7ECHtu0smM?t=1695) So because of that, if we pay attention to the way that things are designed, it's actually an opportunity for us to solicit patience from our claimants and employers. I know all of us are doing a lot of work right now to kind of transform and modernize and make things a lot easier from the state's perspective for the people that we serve. And if we are able to just, you know, visually show at least that we're doing something to change, what it's going to do, it's going to signal towards the people that we're communicating to that we're trying. And because of that, that will give a little bit more patience over time, over their length of time, for example, of things that they expect from us as well. The third thing is selective filtering. I'm not sure if any of you have seen this before, but it is true. You'll read this first, and then you'll read this, then this one, and then you'll read the top one last. I just want to point this out because in our brains, when we look through any type of notice, the amount of information that is out there in the world right now, especially for something that you're trying to understand, our brains are taught to do this. There's a reason why marketing exists and, you know, why things are laid out the way that they are. And so focusing on composition on the way that we created our communication was really key because we wanted to make sure all of the content was there, but that through primary, secondary, and tertiary ways of being able to present that content, it was easy to navigate.

[29:43](https://youtu.be/S7ECHtu0smM?t=1783) And so when the first time someone would read it, they would understand the primary. If they wanted to dig in for more information, the secondary was there. And for example, if they wanted even more information about other background policies or other material that they can look into, that's also. Tertiary information. So because of this, typography with dimension was critical. So that's what you'll see in our examples. And user testing was also really critical because it sort of proved why these compositions were working and also, through the design, what people were filtering out initially. And also, you know, when showing them the original designs, what they're filtering out. So I'll show an example of that. But from here, I wanted to show a couple of examples of the emails that we had redesigned within New Jersey. And we're going to walk through how we can make this available to your state. And we're going to actually change one to a state. So you'll see that this is an overpayment waiver application where you'll see a desktop version and a mobile version. The mobile version is very important because more than 60% of people actually access their emails via mobile. So in New Jersey, at the time, there wasn't a mobile friendly version that was available for claimants.

[30:59](https://youtu.be/S7ECHtu0smM?t=1859) And employers. And so now we're doing that due diligence to make sure that that's available to them. And here's another example of an e-adjudication process where we needed more information because of an issue on their claim. This example actually has some of the issue codes written out as well, because we also rewrote those in plain language. I wanted to show this particular example, because if you look at the old email, it just looks like a block of text. And you're not even sure if it's some kind of communication that you need to do something on. And so when we've really read through it, we realize that it's just kind of an FAQ. So making that very clear in this email as well, where it says your application will be sent in a separate email and that it's just rules and regulations. You know, it allows, it gives people expectations on what to spend their time on. And if they are, they won't get frustrated after reading this whole thing and being like, what, this is just an FAQ you didn't need me to do anything. Right. So we're really trying to pivot out of those moments that we cut that cause frustration a lot for our public. And this is an example of that filtering process that a user test really showed us. So on the left and right, where the pink boxes are surrounding are your rights. And on the left is obviously the old email, the second is the new one. And just by creating that background color and rearranging it in a particular way, a user actually said, this one gives me my rights and the other one doesn't, even though the text and all of that content was very obviously there. Right. So there's a moment here to really think about why this works and why this is really important.

[32:47](https://youtu.be/S7ECHtu0smM?t=1967) These are some examples of our appeal tribunal printouts that we've also changed. The left is an old notice of phone hearing where everything about it tells you to register when it's really the appointments already set for you and you just need you to confirm it. So a lot of that language change and that attraction to make sure that you're going to do what you need to do. And this is on the back of that notice where it prepares you for your hearing. And so we've kind of changed all of our preparation to register, confirm, prepare and be ready. And then this is also an insert that gets placed on how to prepare for the hearing. On the left hand side, you'll see everything is written out in paragraphs, where on the right, we gave every single section a one, two, three. Of what to do and when to actually submit whatever it is they need to submit by what time because we were seeing a lot of people missing their deadlines and also just not showing up and giving the right material. Here is that slide where we see people react to the newer changes and it feels kinder. I feel a lot better knowing I'm being paid attention to. Thank you so much. Appreciate it.

[33:59](https://youtu.be/S7ECHtu0smM?t=2039) I took four years of Spanish, but I'm not about to butcher this in front of lots of people. So I'll just read the translated part for this one. It's, I trust your emails because they are from the state and they are correct. The next one is they are understandable. I wanted to really put these in here in the way that they came into our form. The fact that somebody who speaks fluent Spanish felt comfortable enough to write that kind of feedback for us. It's been like this wonderful side effect because we know that as English as second language speakers, they are grasping onto these words that we've made very simple. It's not that they don't understand English at all, right? They have the ability to navigate a world that is not their primary language, but this is just a really beautiful side effect of what we've seen based on the changes that we've made. This makes me feel like I have to get it together. I think this group would really appreciate that because we need our claimants and employers to also feel like they need to act and that they have responsibilities. And, you know, it feels more solution focused. There's just a partnership that feels better with this kind of communication that's going out.

[35:07](https://youtu.be/S7ECHtu0smM?t=2107) Just some quick stats. The average time to read an email, we've reduced it by over 75% going from about 28 seconds to 6.4 seconds. Average time to complete tasks. These are based on our user tests. Also, more than 75% reduction. So it used to take about two minutes. Oh, I'm sorry. This is flipped. The task one is 28 seconds to six and then the read email is, it used to be about two minutes, but now it only takes about 25 seconds to read an email. And right now what we're seeing is that, on average, we've reduced the amount of time from sending the email to somebody logging into the portal by over 10 hours. So that's huge because we need somebody to at least get the process started because once you actually get them in the door to start their form, the more likely they are to finish that form. Okay. And then this is like a little bit more of an innovative test that we created where we're asking our claimants and employers to score the designs that they see from a scale of one to 10. One being not at all and 10 being very much. And our scores have been jumping. So from the older communication we've seen go from a 4.5 to a nine, which is almost 100% increase in clarity.

[36:29](https://youtu.be/S7ECHtu0smM?t=2189) In urgency, it went from 5.7 to 8.7, which is 53% increase. And then a 4.3 to 8.5 in confidence, which is a 98% increase. I wanted to kind of comment on that star here for urgency. People felt urgent, but there was a shift in urgency, where before they felt very kind of tired by the urgentness of it, almost like they didn't really want to, but they knew it was important, towards a more positive sense of urgency, where it's like, I know I can do this, I can get this together, or this will benefit me if I do this, if I do it faster. So that was a really nice moment to see. Kind of, you know, going back to like this resource that I've been referencing, we can put the link in the chat as well. But this is now available on Figma. And I'm going to walk through all of that right as soon as I stop sharing my screen on this particular presentation. But these are some examples that you'll be able to find, and I'll show you exactly how to find those within the email template. And we have a library of over 50 emails that we've allowed you to access, of the ones that we've already changed from New Jersey as well. So that's available to you.

[37:50](https://youtu.be/S7ECHtu0smM?t=2270) I am going to stop sharing and then share my screen on the Figma file. Does anybody have any questions so far? Okay, I will just keep going. Um, I also can't see any chats right now. So if there is one, if somebody...

**Dave Cole**

[38:15](https://youtu.be/S7ECHtu0smM?t=2295) Yeah, we had just one come in here. In addition to qualitative user feedback and email stats, do you use additional user tracking like Google Analytics to track the user journey across digital

**Janice Cho**

[38:27](https://youtu.be/S7ECHtu0smM?t=2307) communications? Yes, we are tracking a lot. We're actually doing A-B testing within e-adjudication emails. And we just launched our e-monetary emails where we're doing A-B testing because we figured out how to do a preview text. So whenever you access your mobile phone or when you're looking at your email inbox, there's like two lines of content that are usually below the email subject line. So we've actually understood how to impact that and to intercept that. And so right now we're also learning that that's kind of upticking people and reducing time even more when you're giving them some preview text to log in. We've actually gathered data from since March till now for the adjudication A-B tests and then the e-mons email just kind of launched a few weeks ago. So we're gathering a lot of that data right now and those will be available too. Yeah. And one more for you. How does the template

**Dave Cole**

[39:29](https://youtu.be/S7ECHtu0smM?t=2369) kit relate to USWDS, the web design standards? Many of the components look similar. We'd love to see USWDS reference. Yeah. Thanks for noticing that. We actually derived it purely

**Janice Cho**

[39:41](https://youtu.be/S7ECHtu0smM?t=2381) from USWDS. Yeah. We took it from there and we took all the components. We did a little bit of work on the typography because I'm a typographer and I am a letter presser so that like I just kind of changed a little bit of the letter spacing. But beyond that, I think all of the components are from there. Yeah. Okay. So really quick and we'll take more questions at the end too. But I realized the amount of times that we've been doing this that people need context on Figma rather than me just saying Figma. So if everybody knows GitHub is to engineers, Figma community is to designers. So I can go in here and I can basically write government and see all of these resources that designers all over the world who are designing for government are sharing for other designers who work in government to kind of see what they're working on and to also kind of build off of that knowledge.

[40:46](https://youtu.be/S7ECHtu0smM?t=2446) It's been a really beautiful community to be able to kind of see and grow. Even when we were starting this project, we didn't have Israel involved. Japan wasn't in here. Italy was not in here. So within the past year, there have been a lot more countries kind of like understanding how this works and also using our work and other people's work to do this. Going back to USWDS, this is their resource that we pulled from here and were able to very quickly rapidly create email templates. Fun fact is that the email templates or there were actually no email templates on here besides Russia's government UI kit which had two emails in there and that was more of like a login one. If you scroll down over here, this is our little resource that you can see and if you go into it, you can download it and open it within Figma. If you feel so inclined, you can bookmark it and maybe leave a rating. But I'm just going to take you into the file itself. One beautiful thing about this is that you do not need to download the program. You can actually just use it on any of your browsers so that there's nothing that you need to do there.

[42:00](https://youtu.be/S7ECHtu0smM?t=2520) When you open it, you'll see a bunch of these little menu items on the left-hand side. The cover, getting started, these are very specifics. If you're somebody who's going to go in here and change a lot of the content in here, you can download the font. All of the links are available to you. The styles and components are here but besides but more importantly, the building blocks of the emails are here. So it's all built within nine components and each one has variations of what it would have. It also has step directives, which are limited to three. The way that this is kind of constructed is linear and there's linear restriction. So it must be placed in the order that it's shown. But let me get to the fun part where we are building, let me see your state example. So this is just a really, we made this because we wanted you all to feel like this is achievable, okay? So if you go into this area where it says state examples and if I click on this email and I go over here right now, it says New Jersey. Let's say I am from New Mexico. Now I have brought all, like we have brought all of our, your links, your logos. If it's not an updated logo, please let us know because we'll update it for you. We've had a couple of requests like that. We've found a lot of these links through the U.S. DOL site but these are available to you, yeah, and you can just see it almost automatically. We pulled in the colors that your particular DOL site also had too. So they're kind of moving and morphing the way that it is. The rest of this, you'll see a lot of resources within here. There's like user research things, tools that you can use when you're doing your user research. There's a full report that you can see as well that's, you can download it in different forms. There's a PDF version, a text

[44:02](https://youtu.be/S7ECHtu0smM?t=2642) only just to read more about it. But let me do something real quick where we're going to go into this. Oh, this email library also has those 50 plus email examples. They're sectioned off by programs. So there's dependency, adjudication, id.me, MEUC, right? So there's all of these emails that are available to you the way that we rewrote them, except they're branded with Avengers if anybody is a fan. And let me show you how we're going to demonstrate something as I pass it off to Noah. So in build your own email, we have built the building blocks in the way that most of our templates are constructed. So this one just has a status. This one has a one button directive. This one has a one, two, three directive. This one has a status with the directive. And this one has a Status with the Directive and Appeal Rights. The last one also has some login information on the bottom, which we all love. So I am going to take this one, and we did this webinar a few months ago now, is it already? So I'm just going to demonstrate for you in here that I know that we were able to change this to New Mexico, but we can also do it on any of the emails that are in this file. So I'm going to go in here, you'll see that it's purple, which means I can change it.

[45:30](https://youtu.be/S7ECHtu0smM?t=2730) I'm actually going to go find California where I was born and raised. So we'll find a California there. I'm going to change this to California as well. And then we're going to do a title here. We'll do Welcome Webinar Crew. So that is the way that we are going to do that. And, you know, more questions around how this works, but basically all of these emails are available for you to manipulate and change and edit on your own. The link is there. And I'm going to pass it off to Noah, because this part right here where you can request GitHub access, we're making the code also available to everyone, but he'll demonstrate how you can build it with all the stuff that we have. Thank you.

[46:34](https://youtu.be/S7ECHtu0smM?t=2794) Yeah, thanks, Janice. So I'll be going over the process of how we take those designs that Janice

**Noah Marcus**

[46:38](https://youtu.be/S7ECHtu0smM?t=2798) just showed and convert them to an email we could actually send to claimants. So let me share my screen here. Okay. We should see pretty much the same thing we were just looking at, which is the Figma file. But the first thing I want to focus on here is just commenting on how difficult it is to actually write HTML for email. It's a very tricky subject for two main reasons. Over the last 20 years, there's been a lot of advancements in web design and HTML standards that have helped with creating more intuitive and responsive digital designs. But for better or for worse, email has not caught up. And under the hood, email is more reliant on code that looks closer to something from the 90s than 2023. On top of that, how users are actually interacting with the email is unpredictable. Of course, with any kind of digital product, we don't know if they're viewing it on their phone, on their computer, or their tablets. We have to account for that. But also, there's many different email clients. There's Gmail, Yahoo, Apple Mail, etc. And all of those clients are actually interpreting the code slightly differently. So what that means is the code that I write to display an email on Microsoft Outlook is actually a little bit different than the code I write for Gmail.

[48:05](https://youtu.be/S7ECHtu0smM?t=2885) So what we need is one main HTML file that's going to account for all these little slight differences. And that HTML file is relying on really old code standards. So this feels like a very impossible task. And when we were first translating these email designs to HTML, we struggled a lot with these kind of email-specific HTML problems. So one tool we've created as we've learned and designed these is something that we're calling the component library. And this tool, the component library, is meant to be a companion to this Figma file. So as Janice went over, we have these building blocks over here, or these components that make up the email. And we've gone through and we've defined a standalone HTML that follows all the correct standards that I've talked about. And it can be easily copied and pasted into a pre-made templated HTML file. So this means instead of starting from scratch every single time you want to build an email, we can quickly build an email in minutes and rely on all this testing and configuration that we've previously done. So currently, the component library is living in this GitHub repository. In this repository, we have a template file which has all of this configuration at the top. And if we scroll down, we can see clearly marked places where we copy in each of these components. And then each of these components correspond here into their parent folders to all these different building blocks. You'll see we have headers, we have main content, and over here we have our headers, our banners, our main content, etc. So let's actually do the fun part and let's build the email that Janice was showing. So today we'll be designing this email.

[49:54](https://youtu.be/S7ECHtu0smM?t=2994) I'm going to be using this tool called Litmus to kind of demonstrate this. I want to stress this is not a tool that you need to implement these emails. It's just something I'm using today to help with the demonstration process. But Litmus is essentially just a marketing tool that specializes in email communications. It helps us with building, optimizing, and testing these emails. And as you can see here, it gives us this beautiful view. We're on the left-hand side, we have the code, and on the right-hand side we can see a live visualization of what's happening. So to start, in the interest of time, I've already kind of templated out a little bit of this file. So I've copied in the template and I've copied in a few components. I've also changed some of the features to match the California designs. We have the colors and the logo already imported in. But you'll see there are some things missing from the design here, so let's just quickly compare it. It's looking like we got a title change and I think we're missing this component down here. So yeah, it kind of just ends right there. So the first thing I'll demonstrate is the title change. So in the code, I'm just going to do a quick lookup to see where this main title is. And it's right here. And we're going to change that to Welcome Webinar Crew. It's as simple as just typing it in, if I knew how to spell.

[51:20](https://youtu.be/S7ECHtu0smM?t=3080) And there, we saw the live change reflect there. Now, the next change is a little bit more complicated. We're missing a whole component here. So first, let's figure out which component we're missing. It's looking like the secondary action. So let's go to our building blocks and find the right one. So here we go, secondary content. And there's two options. It looks like we have the default option and a benefit breakdown. So we want the default. So I'm going to head over to the component library. I'm going to drill down into secondary content. And I'm going to select the default. And now from there, I'm just going to copy this whole component that's already, you know, like I said, all the configuration, everything's been updated. It's going to render correctly as expected. And I'm going to bring that over to the email. I'm going to look up where to copy that in by searching for a secondary content. And here we go. It's telling me to insert it right here. I'm going to paste it in. And there we go. It's right there. Now, from there, this is something that's litmus specific. And why I like it so much, we can start to see what that's going to look like across all these different email clients. We could pass it through quality assurance checks. It looks like my internet's running a bit slow. But from there, we can test it and then eventually send this to the claimants. And I'll stop there for questions.

[52:46](https://youtu.be/S7ECHtu0smM?t=3166) So just one note about how to access the code. We'll post a link to the forum. And the code is

**Dave Cole**

[52:51](https://youtu.be/S7ECHtu0smM?t=3171) available open source. We do make it available through a request in a forum just so we can filter and make sure it's getting to folks who are actually working directly with governments on this kind of thing to help limit potential abuse of it. Other questions for Janice or Noah? I think there was some great feedback and chat along the way. Very good. Anything else on your end, Noah? Good. Well, thank you so much to both of you for sharing the work and the resources that folks can use to leverage what you've done here in their own projects. For our next section, we wanted to move on to one more working example. And in this conversation, Naman Agrawal, who's an engineer with the Office of Innovation, has been working on UI modernization as well as other benefits programs for a couple of years now, is going to walk through some tactics that we have used in partnership with the Department of Labor to modernize existing legacy applications. So rather than replacing the full application in a so-called Big Bang, how we can incrementally improve the content, the design, and the functionality. So Naman, I'll hand it over to you.

[53:57](https://youtu.be/S7ECHtu0smM?t=3237) Thanks, Dave. Hey, everyone. So yeah, I'm going to start sharing my screen. And as Dave mentioned,

**Naman Agrawal**

[54:04](https://youtu.be/S7ECHtu0smM?t=3244) we're going to talk today about how to redesign a legacy application. I have an asterisk here that we don't want to touch it. Ideally, you do, but sometimes there's problems that prevent you from doing that. And this is inspired by work we've done closely with the Department of Labor in our office on unemployment insurance, but also other programs like Paid Family Leave. So I'm going to walk through two approaches that have worked well for us, but I want to start with approach zero, which is not one that I'm going to actually go into, which is why it's zero. And that's just updating the legacy application directly. So maybe we don't want to rebuild it from scratch yet, but we do want to update some content, change a word, update the design. That can be tough. I'm sure you've run into scenarios where it takes a long time to make a simple change, or it takes a lot of complex coordination to do that. And this can be for very reasonable reasons, like the owner has limited bandwidth, IT teams are doing a hundred things at once. Maybe your team doesn't have access to the code you want to change, so you can't do it yourself. The team also might not understand the legacy system well enough to feel confident in making that change. There's other reasons, like the legacy system may be a little fragile and you may be scared to break it, or maybe you want to use certain tools in the changes, but the legacy system can't accommodate them. So when you're dealing with these sorts of problems, we have a few approaches that have worked well. The first is injecting a script. So what this means is you can update the website's front end without directly touching its code. The script acts as a mask, basically, where you keep all the code the same, and the mask is just going to change the presentation layer, move around maybe content, layout, styling, to address the changes we want.

[56:03](https://youtu.be/S7ECHtu0smM?t=3363) So I'm going to talk through the UI intake application where this worked well. And in this case, like, of course, there's work to rebuild this from scratch, and there's a lot of work ongoing. But from doing user research, we found a lot of quicker fixes that we could make to improve the experience in the short term. So one thing that Janet started touching on was around mobile accessibility and how a majority of our users use mobile accessibility. There's also an angle around equity where those underserved communities are less likely to be on desktop or have broadband access. So they're going to be the ones more likely to be accessing your webpage from a phone. So that's where this becomes extremely important. As you can see on a mobile view in the original intake application, it can be hard to navigate, everything is zoomed out, tough to read content, you need to zoom in manually and scan through it. So I'm going to show you the after now. And with the mask, so we now have this script on top of it that's changing some of the design. One great thing we can do is change the scaling so you don't need to zoom in to read content from your phone, as well as stack the information vertically, rather than having two columns here and having things more spread out. When you stack it vertically, everything can fit nicely in a better mobile experience that users are also pretty used to on other websites.

[57:37](https://youtu.be/S7ECHtu0smM?t=3457) Another thing that we were noticing was around language. So this application can be harder to understand. And also when things are hard to understand, they can take longer to complete. So with the script, we're able to find specific pieces of text, and then actually update the content. So we're not going to be changing the existing website at all. But the script is able to find those pieces and update the sentences. So as you can see here, we've changed Do you currently reside in the US? Or have you been employed in the past 18 months to simpler language such as Do you currently live in the US? Or have you had a job in the last 18 months? You can do similar changes, like changing the content of options people have to collect, as well as like which options people even have. So you're not necessarily changing the back end. But by changing what users see and what options they're presented, you can hopefully give them easier options and then do the work in terms of the back end or the script to translate that into what the old application was expecting. And then one last thing to talk through is the hierarchy of information and generally the design and styling. So with the script, we were able to pull in the US web design standards, which I know some people were mentioning with the emails as well. And this helped us in terms of sectioning content. So using text styling to break up questions into easier to read sections, as well as certain design patterns around buttons and inputs. So you can you may notice that the inputs are actually bigger here on the mobile version. And this can be important for people with motor impairments to more easily select options. And the US web design system and our New Jersey web design system, which is just a

[59:38](https://youtu.be/S7ECHtu0smM?t=3578) certain theme on top of it, has been extensively researched to account for accessibility and use best practices in terms of the design. So we're able to get that work in for free and then pull that in in the sorts of buttons and components that we're using here. And the impact is huge. So it may feel like it's just surface level changes without a full rebuild, but on average, we saw that it takes about 48 minutes less time to complete an application, which for an individual person who is strapped for time, that can be huge, and this is a 19% decrease from the original application time. And I did mention analytics. That's another thing that you can add with a script to get a baseline sense of like how people are using the application and then start measuring if your changes are actually useful or not.

[1:00:39](https://youtu.be/S7ECHtu0smM?t=3639) And then I'm just going to highlight technically why this can be so successful one more time. So in this diagram, you can see you have the legacy site. So this was the original UI and take application. The owner only needed to add one line of code, very simple on their end. And then, and from this one line of code, we're downloading a script, which can be managed by your team. And that script is then going to update the presentation layer in the ways I was showing you. So because we're managing the script separately, you can incrementally add changes to it. You can update it without necessarily deploying and redeploying the intake application. So this can allow for a faster and more frequent workflow. So to summarize, when would you want to use this? If you can't make the changes directly and your changes are more surface level related to content or design, this may make more sense when you don't want to use it, if you're doing a full redesign, it's possible, but it may be more complex.

[1:01:39](https://youtu.be/S7ECHtu0smM?t=3699) And if you need new data, remember that we're simply adding a mask to existing content. So we're not going to be pulling in any new information. I'm going to pause there for a second. If people have questions around this approach before I move on to the next one. Simon, Reagan had two questions related to our use of

**Dave Cole**

[1:01:59](https://youtu.be/S7ECHtu0smM?t=3719) design system and components. The first is how we manage the web design components that you're showing in the web templates versus what Noah had presented in the design system. What the kind of connection is there. And then if we have a new version of the web design system, we can use that as well. What the kind of connection is there. And then if we have had to contribute any improvements upstream to the US WDS or how we manage that at the state level. Yeah.

**Naman Agrawal**

[1:02:24](https://youtu.be/S7ECHtu0smM?t=3744) So that's a good question for, in terms of how we manage the design system, we have a GitHub repository, which is going to be forking the US web design system. So we're constantly getting any changes upstream from them. And then we have a file that overrides the theme. So design system allows you to customize your primary button colors, link colors, different interactions like that. And we have that in the GitHub repository, which can be built and served as CSS files and JavaScript files, which you can then pull in to your website. So that's what we're doing here. We're pulling in that CSS directly. I don't think we've had any opportunities yet to send improvements upstream to the US design system. I might be wrong, but we're definitely paying attention to it as we use it more and more, we might find some issues. Yeah, I think that that covers it generally. And just to say that in terms of how this relates to the email, we start with the same New Jersey web design standards theme, which is based off of the state web presence guidelines as Janice had mentioned, we have made some changes specifically for the topography and the presentation and email that we use in email templates, the things that we present on the web, largely just taking the components directly out of the library that Naman was just referencing.

[1:03:45](https://youtu.be/S7ECHtu0smM?t=3825) So links are in chat there. We'll follow up with them as well.

**Dave Cole**

[1:03:50](https://youtu.be/S7ECHtu0smM?t=3830) One quick follow-up here, Naman, and then let you get ahead to the next section. Do we use any custom events in Google analytics to help designers have more granular understanding of the use of the service and the journey? Yeah, we certainly do.

**Naman Agrawal**

[1:04:05](https://youtu.be/S7ECHtu0smM?t=3845) I think by default, Google analytics is giving you information on the views on each page and what kind of devices people are coming from. But when you want to get into certain interactions, like which options are people selecting, which page are they getting to, are they kind of running into any stumbling blocks along the way? That's where custom events can be really useful. So we definitely use that to understand what the usage is of different interactions and see if we can improve on them. Okay. So I'll move on to the second approach and go through it relatively quickly. But what we're doing here is in the last example, I was mentioning how you might need a full redesign, and that can be cumbersome when you're just dealing with that mask or that script on top. So what you can do is actually create an API around your entire legacy website and use that to provide data in a structured way to an entirely new front end.

[1:05:12](https://youtu.be/S7ECHtu0smM?t=3912) So it's a bit confusing when I say it like that, but I'll show you an example and then a diagram as well. You have your, in our case, we were doing this approach with the claim status website for UI. We noticed that a lot of people are calling in and have confusion around where their claim is once they've applied, if it's been processed or not. And this is a big place where we can allow people to figure out the answers themselves without necessarily needing to wait to find an agent. So we had an old claim status website, and I'm going to show that soon. This was stood up in the pandemic. It was extremely useful to take in an inflow of questions automatically online, but we had difficulty in making changes to them directly, and we wanted to do a relatively involved redesign based on user research, so what we figured out we could do is create this API that is going to essentially treat this website as our database and scrape the website to get the data that we need rather than integrating directly with the claim data, which is on premise and relatively difficult to do.

[1:06:25](https://youtu.be/S7ECHtu0smM?t=3985) And once we're able to scrape that data and provide it in a structured way, we can then consume that in an entirely new website that aligns to the standards we want and can change it in the way we want. So let me demo what that looks like now. This was the old claim status application. One thing that we were learning from our research was that it was a bit painful to log in and get the information you needed. There were a lot of repeated fields where you need to confirm information as well as fields that weren't necessarily needed for getting the claim status. And that was before you log in, this is the view once you have logged in. Once you do log in, there's this content added at the bottom where you can see your claim status, date of claim. Some issues that we were noticing from talking to users was that the most important information, like whether something was filed or not, was not necessarily the most prominent, a bit hidden here, as well as it was a bit unclear around how to initiate certain actions on the page, like resetting your pin.

[1:07:37](https://youtu.be/S7ECHtu0smM?t=4057) And that takes us to the redesigned version where we had full control over the front end by using this API wrapper method. And as you can see here, in terms of logging in, we were able to simplify that process as well as use best practices around how to take in user information informed from the U.S. web design system. And once you do log in, you can see that we're able to display that information, the same exact information and hopefully a more clear way that gives the answers that people are coming for, as well as understand better how they can take actions and get feedback on what to expect with them. And in terms of how this actually worked, our API would be accessing this web page, so not through the browser, but through code. And when it goes on to this URL, which is fully public, we don't need to connect to some mainframe database. We're just connecting to this page. We're getting this HTML file, which is essentially like what's visually shown by the browser and simulating a login based on what the user is inputting.

[1:08:47](https://youtu.be/S7ECHtu0smM?t=4127) So they gave their SSN once, but we can put it in twice with our code and then we can pull this exact information that we need, like the claim status, the date of claim, and return that in a structured format. So as you can see here, this is the structured format that we got back from our API directly from this web page, and then now we can show it in the way we want. We can make it mobile accessible. We can use the web design system we want, as well as change the hierarchy and do a much more involved redesign. And in terms of impact, I could go into metrics, but I think there was more clear impact just from hearing what users thought about the change. So this is some information that users gave on their own from a feedback form that we have. And we heard things like this website's easier and faster than before. Oops, I love the new layout, just looks nice in general.

[1:09:48](https://youtu.be/S7ECHtu0smM?t=4188) It works well for people that may not be as literate in terms of the Internet, and it's quicker to access things and less confusing. So from taking the exact same information and then just displaying it in a more user friendly way from our research, we were able to make a pretty solid impact. And just talking about when to use and not use this approach compared to the last one, in this case, if you want full control over the front end and to do a more full redesign, this can be useful. It also gives the opportunity to bring in new data sources that might have been harder to integrate with the old website now that we're working with a separate front end, as well as bring in a new tool set. So we could use the web design system library. We could use automated testing, GitHub, some best practices that we like on our team. When not to use it, sometimes that scraping process can be a bit difficult depending on the login system you're working with.

[1:10:50](https://youtu.be/S7ECHtu0smM?t=4250) There could be CAPTCHAs. These are things that can be overcome, though, and another thing is if you do, of course, need to use the back end or underlying data, this is out of the scope of this work. So these are two lightweight ways where you can make hopefully a big impact, and we can definitely share with you more around how that looks for your specific cases. Very good, thank you, Naman.

**Dave Cole**

[1:11:20](https://youtu.be/S7ECHtu0smM?t=4280) For folks who have questions, drop them in chat.

**Audience**

[1:11:23](https://youtu.be/S7ECHtu0smM?t=4283) Do you do anything to detect changes on the site that you're scraping that might cause issues? That's a really good point.

**Naman Agrawal**

[1:11:31](https://youtu.be/S7ECHtu0smM?t=4291) So we are making a huge assumption that this old page is never going to change because we rely on it in order to find exactly the stuff we need. In this case, this website is no longer updated, so we're kind of making that assumption, but we could definitely add in some monitoring or work closely with the team managing this to set up something more automatic. Very good. Well, thanks again, Naman, for the presentation, sharing your work here.

**Dave Cole**

[1:12:05](https://youtu.be/S7ECHtu0smM?t=4325) I think we should move on to our last section here, which is going to be back to Jill to share some of the strategies that she and the Department of Labor have used to help kind of transform the organization in how it approaches this kind of Agile human-centered project management. So, Jill, we have about 15 minutes left, so I'll hand it back to you. You are muted, by the way. Can you hear me? Yes.

**Jill Gutierrez**

[1:12:34](https://youtu.be/S7ECHtu0smM?t=4354) OK, sorry. Well, thanks, Naman, thanks, Janice and Noah. So I'm going to talk about three sort of I'm going to quickly talk about three ways that we've worked at NJDOL around human infrastructure. And this may not be what you came in expecting to hear about, but I really think it's one of the keys when you're thinking about what are the what are the people impacts, what are the people processes, what are the people feelings that go into making big changes? So the first I'm going to talk about has to do with growth, and Dave, you can switch the slide. So our NJDOL CIO, Sharon Pagano, had this idea that both we need to think about how we use our internal DOL staff, our internal staff here are not the idea of Agile development is new to them, right?

[1:13:35](https://youtu.be/S7ECHtu0smM?t=4415) It's not how typically the state has done its development or its procurement. So what do we need to do in order to grow those folks? And the idea being that there is a learning curve, right? There's so much that they could learn from this method of development. There's so much growth potential, making it sort of an exciting place to be when you're thinking about working in state technology. But it's also a sustainability issue for the state in terms of when this Agile project may finish, who is going to maintain it and how do we build that human infrastructure underneath it to make sure that we are creating a path to both understanding what is the thing that was built, as well as giving ownership to the staff about I know how to lead those changes. I know how to think about human centered design and the personas in order to create updates and continue editing the code and making changes as policy or processes need to change.

[1:14:41](https://youtu.be/S7ECHtu0smM?t=4481) And so the idea is to both to hire the staff. Yes, it's challenging, but you're embedding them into this exciting process right where they have becoming the experts, they are learning how to develop and maintain in this new way. And I think one of the most exciting things. One of the first things that we heard, right, was that one of our younger developers said to us, you know, I really love this work. I love it because I see the responsibility, I see my partners who are going to review my code, and I like the transparency in the work. It's really impactful when you think about this is a young coder. He is going to grow up in this department, if he chooses to, and spreading these ideas about how is it that one should be developing, what is our responsibility towards technology, and how can I take responsibility for how it grows and how the code is developed. So that's on growth. And kudos for sharing the thinking in this way.

[1:15:46](https://youtu.be/S7ECHtu0smM?t=4546) The next slide talks a little bit about energy. Bringing forth an intake application to launch this fall with the idea that we should be reflecting what our primary language uses are. And our number one language use in NGDOL's UI system is Spanish. And while Google Translate exists currently, that's not how we wanted to present the online experience of translated UI application. And so working with USDR, we came up with a process in which, yes, we had a professional translation done of our application. But that professional translation is not enough. And so we took the UI call center agents, the ones that answer the phone calls in Spanish about what's going on with my claim, what do I need to do next? And we included them in the process of reviewing this translation and asking them, you know, what are the top things that you end up having to explain in Spanish?

[1:16:48](https://youtu.be/S7ECHtu0smM?t=4608) And what are the words you use to describe them? Right? Really bringing along the staff in terms of you have this really deep knowledge of where people get hung up and really thinking about the energy that they could put forth and that they were excited to share with us because no one had ever asked them, how do you go about explaining last day of work? Or how do you go about explaining why someone was let go from their work in Spanish? And so we've worked with USDR through focus groups, usability tests, also with Spanish speakers from a diversity of countries in order to develop a professional and community-based reviewed Spanish application that serves many different populations within our state. And you can see this, this work. Next slide, Dave. This work is reflected in a glossary that's available online for other states to use about this research. USDR has made public for it. And there's ways that you can understand, right?

[1:17:54](https://youtu.be/S7ECHtu0smM?t=4674) In some cases, we need to show, we may be able to do an explanation of your benefit year in English in just one sentence, but maybe in Spanish, because of different ways that people might understand that term, we may need to offer that in three sentences, right? Try different ways. The research also showed that sometimes you need to have it in English and a translation in Spanish. So really thinking about not only the plain language impacts of people understanding that Janice talked about and how that helps English language learners as well, but really thinking about a one-to-one translation is not necessarily what we need. So that's one of the things that we're looking at. And then the other thing that we're looking at is how do we make sure that we're providing an equity and involving our staff in getting to those results and getting to that sort of understanding of what it is we should be offering to get us towards an equitable access to these benefits.

[1:18:57](https://youtu.be/S7ECHtu0smM?t=4737) So that's one of the things that we're looking at. And then the other thing that we're looking at is how do we make sure that we're providing an equity and involving our staff in getting to those results and getting to that sort of understanding of what it is we should be offering to get us towards an equitable access to these benefits. So that's one of the things that we're looking at. And then the other thing that we're looking at is how do we make sure that we're providing an equity and involving our staff in getting to those results and getting to that sort of understanding of what it is we should be offering to get us towards an equitable access to these benefits. So that's one of the things that we're looking at. And then the other thing that we're looking at is how do we make sure that we're providing an equity and involving our staff in getting to those results and getting to that sort of understanding of what it is we should be offering to get us towards an equitable access to these benefits.

**Dave Cole**

[1:20:14](https://youtu.be/S7ECHtu0smM?t=4814) Thanks again, Joe, for sharing. Yeah, I don't know if there's anything else you want to say, or we can open up to questions here. So folks, really appreciate all of you spending the time for the last almost hour and a half with us getting to see some of the work that we've been doing. We will follow up with the resources that have been posted in chat and follow up email as well as the recording so you can review them on your own time, keep the discussion going. Our goal here is to simply share some of the things that we think are promising and working at the state level in New Jersey, so that you may apply as you see fit in your own programs, whether it's here in the state, different programs or in other parts of the world doing similar work. Are there any other questions for any of the panelists, or if anybody has a particular case that they are working through want to share and get some feedback from the panelists, please feel free to raise it now. I'll also note while we're waiting for any questions here, Hannah has dropped in a link a few times on a survey so please do take a second to let us know how this format works for you and any feedback that you have.

[1:21:23](https://youtu.be/S7ECHtu0smM?t=4883) I know that that will be very helpful for improving the content as we go forward with other sessions as well. Very good. Well, I'm not seeing any final questions here, so I will just reiterate again how thankful we are that each of you have spent the time with us. Do reach out if there are follow up questions or anything that we can answer to help each of you in your modernization journeys independently and thanks again to the panelists for sharing all of the great information the work that's been going on for the folks across the state, who have enabled this work, many of whom are not even presenting today but have contributed substantially to this work both at the OL, OIT, the innovation team, and others across the country that we have derived influence from. Thank you everyone.

**Laurie Harrington**

[1:22:15](https://youtu.be/S7ECHtu0smM?t=4935) Thank you to you Dave thanks to the all of the panelists, this has been such a pleasure for us to be able to hear about the incredible work that New Jersey has done in this in this area so thank you all, and thanks to everyone for spending the afternoon with us we'll be sharing the resources. After this session. Thank you again to everyone who participated. Have a good afternoon.

**Unknown speaker**

[1:22:39](https://youtu.be/S7ECHtu0smM?t=4959) Bye.

* * *

Original page: https://innovate-us.org/modernizing-unemployment-benefits-a-hands-on-workshop-for-states · Licence: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org
