---
id: "workshop:how-to-apply-human-centered-design-to-government-ai-projects-"
type: "InnovateUS workshop"
title: "How to Apply Human-Centered Design to Government AI Projects"
url: "https://innovate-us.org/how-to-apply-human-centered-design-to-government-ai-projects-"
date: "2024-02-29"
duration: "90 minutes"
language: "English"
instructors: ["Elham Ali"]
recording: "https://www.youtube.com/watch?v=SEUu9GkbnxE"
transcript: "automatic, from the recording"
source: "InnovateUS (innovate-us.org)"
licence: "CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org"
---

# How to Apply Human-Centered Design to Government AI Projects

InnovateUS workshop · 29 February 2024 · 90 minutes

Join us for this 90-minute virtual workshop with Elham Ali, Researcher at the Opportunity Project for Cities with the Beeck Center for Social Impact and Innovation at Georgetown University. In this training, participants will learn about approaches to integrate the principles of human-centered design and equity to Artificial Intelligence (AI) design, use and evaluation. Elham will share best practices for public sector professionals to apply human-centered design in AI to their work projects, team, and community collaboration. Participants will also learn from a city government case study where government advisers, community nonprofits, and a technology company worked together to successfully apply human-centered and generative AI design to a civic technology project.

## Learning goals

- Learn about approaches to integrate the principles of human-centered design and equity to Artificial Intelligence (AI) design, use and evaluation
- Learn best practices for public sector professionals to apply human-centered design in AI to their work projects, team, and community collaboration
- Learn from a city government case study where government advisers, community nonprofits, and a technology company worked together to successfully apply human-centered and generative AI design to a civic technology project

InnovateUS offers weekly virtual trainings on innovation, data, and digital skills for the public sectors. Check out and sign up for our upcoming workshops [here.](https://airtable.com/appdAyP4l2W9gxQdc/pagx9sXF2F53dI7kn/form)

## Speakers

- **Elham Ali**, Researcher at Beeck Center for Social Impact and Innovation, Georgetown University

Recording: https://www.youtube.com/watch?v=SEUu9GkbnxE

## Transcript

Automatic transcript of the recording: names and terms can be misheard, and the times can be a few seconds off. Each time links to that moment of the video. The transcript alone: https://innovate-us.org/how-to-apply-human-centered-design-to-government-ai-projects-.txt (plain text) and https://innovate-us.org/how-to-apply-human-centered-design-to-government-ai-projects-.json (every segment with its start and end second).

**Unknown speaker**

[0:00](https://youtu.be/SEUu9GkbnxE?t=0) I'll spotlight you and turn it over to you Ilham.

**Elham Ali**

[0:06](https://youtu.be/SEUu9GkbnxE?t=6) Thank you so much Hannah for that kind introduction but also to you and for Stephanie Kane for inviting me to the space and also to everybody who attended. I know this is a lot of your time so I just want to honor the time that you have given us today and hopefully you can come out of it with something that's useful and helpful for for your work. Again my name is Ilham, my work intersects in design research, data science as well as program evaluation and for the past decade I've had the honor to collaborate with local and state governments from the town of Amherst to Wisconsin Department of Health Services on projects like HIV education and COVID-19 response and at Johns Hopkins my doctoral research focuses on using decision science to evaluate and forecast the benefits of climate action strategies from state governments on health equity and in this space I'm representing as a researcher with the Beck Center for Social Impact and Innovation at Georgetown where we're an experiential hub offering training technical assistance and support to government practitioners to improve the design and delivery of critical services through data, design, technology, and policy and so our goal is systematic improvements for equitable societal change but also the the work that I do advises and coaches the opportunity project for cities or Topsy if you've heard about us we're a design sprint where local governments community-based organizations and google.org pro bono technologists partner with the communities to create solutions for local challenges so past projects we've had from our last cohort included making bibs online permitting roadmap we also had Akron's housing compliant website and Detroit's mobility data tool which we're going to talk about today with a wonderful panel of three people and Miami Dade's

[2:07](https://youtu.be/SEUu9GkbnxE?t=127) commute and extreme heat data sharing app so I'm really excited and very grateful for our collaborators the Center for Public Impact as well as the Knight Foundation including our funder and I'd love to hear from you today just like Hannah said feel free to share your thoughts your questions and feedback Hannah will be monitoring the chat I'll try my best to read it as I go through the the the presentation but please let me know if you have any other questions we have dedicated 20 minutes at the end also for for Q&A with our panelists so for today we're going to learn a little bit how to integrate human-centered design with AI and I'll share one example from my work that's relevant to sentiment and large tech analysis for public comments with a government partner and I recognize that the bulk of this presentation revolves around the synergy of people and teams and the the kind of work that they need to do to support practitioners of human-centered AI how to build models which is our foundational layer but for human-centered design to truly succeed in AI projects especially in government settings we need to touch on the role of the organization so this is a larger bubble which means we need to invest in this foundation of leadership operation measurement and core values so that this collaboration of human-centered AI is embedded into the fabric of how we do things and for that I've invited a panel of speakers today from the city of Detroit including google.org pro bono technologist and ECN a community-based organization who will expand our conversation beyond just like how to do the thing and the team dynamics and into more how do organizations and communities collaborate to set us the stage for systemic integration. I know that the policy consideration like trust safety and privacy is important but they definitely warrant their own spaces and sessions so I won't be touching on them deeply but might speak to them here and there if it's

[4:10](https://youtu.be/SEUu9GkbnxE?t=250) relevant and I'm hoping that you can look into this information with curiosity and inquiry and I also invite you to hold me accountable today throughout this journey if there's something that feels off or you would like to get to know a little bit more as well. So a quick heads up I'm going to use these acronyms today HCD for human-centered design AI for artificial intelligence general which I'll tell you why I'm differentiating that from just generative AI gen AI and then HCAI is human-centered AI and then lastly top C for the opportunity project for cities and I'll clarify as we go along. So before we get started into diving into how do we integrate I want to give you a very very brief history and rapid fire history of AI just so that you know the foundation for today's discussion and why I'm differentiating between general AI and generative AI but I'm very curious in the chat how far back do you trace AI's origin? Feel free to add to the chat any time period I'm curious to hear. In his book literally theory for robots right we have the 1900s 1950s yep Dennis E. Tennant highlights AI's deep roots actually could potentially go back to the 14th century with Arab scholar Ibn Khaldun's prophecy wheel and AI moved from rigid rule-based systems like the 1980s I'm not sure if anyone here remembers chess masters that followed a predetermined response and then we moved swiftly to adaptable learning systems that can interpret and respond to human language like dragon naturally speaking that use machine learning to translate languages from French from vast text data to voice assistants like Siri and Alexa leveraging deep learning to navigate through noise and deliver precise response absolutely I don't know I wasn't sure if anybody remembered dragon naturally speaking I'm glad somebody did the

[6:12](https://youtu.be/SEUu9GkbnxE?t=372) real breakthrough though came with transformers and GPUs which accelerated you know AI's capacity to process and generate conversational text and so this is what's epitomized open AI's GPT series and an early example of generative AI is a much simpler model known as a Markov chain it's introduced by Andrei Markov in 1906 and it's been long used for the next word prediction task like the autocompletion function that you see in your emails that's part of it and anyone who plays the game wordle like myself we intuit each of these probabilities all the time and the reason I'm giving you this foundation is because when we go into our conversation with the city of Detroit who has integrated Gen-AI with HCD they're going to touch a lot on the foundations of how they built their model and all that Gen-AI does is predict what word comes next in a sentence it's like you're playing a guessing game and Gen-AI isn't about certainties it's about prediction and so it looks at certain words like ate apple child and tries to arrange it into a coherent sentence and it scores it guesses kind of like the child's the apple eight is unlikely so we but the child ate the apple makes sense both in the way that the words are put together the syntax and in what the words are trying to say the the world knowledge and so generative AI is already out there you've probably seen it in tools like Grammarly where it's using AI-based assistance in your writing and your work it's a series of algorithms where we're combining larger AI to understand how sentences and paragraphs are written it's not just classic spelling it's how to be a better writer Khan Academy uses its own generative AI chat bot called Conmingo and every day it's being used by millions of people through Khan Academy we also see it in education with Harvard's flagship computer science course CS50 where they use a GPT-based personal instructor instead of

[8:17](https://youtu.be/SEUu9GkbnxE?t=497) additional TA office hours to help students through that class but we also see it in health care doctors at Beth Israel Deaconess which is a hospital affiliated with Harvard Medical School are using a GPT-4 chat bot to teach current medical students how to create an illness script from patients and presenting their their issues and then arrive at a diagnosis but most importantly we see this already both general AI which I'll explain what the difference is from gen AI in public service in the city of Amarillo they have what they call a digital human that makes city hall accessible in many languages to serve both residents and newcomers and then across 27 states we see AI enhancing the process of verifying mail-in ballot signatures which is actually painstakingly painful manually integrates very seamlessly into their existing systems we also see it in Maryland for folks who are in Maryland AI's controlled traffic light tackles congestion and the social security administration has these predictive models that expedite disability claims using historical data from application forms that are completed by millions of previous applicants to generate the predictive score that the administration uses to prioritize new applicants so this comes at a point for us to understand the difference between general AI and generative AI is that one of the key significant challenges with generative AI is that we have to distinguish it from the well-established field of machine learning deep learning there are actually these FDA approved algorithms to detect skin cancer which can be used and those traditional machine learning algorithms both have probabilistic but also deterministic outcome so you can see what happens you know when it happens and every time you make a credit card transaction there's a machine learning model at work that determines whether or not there's a fraudulent credit card transaction so that's the terminus kit and we can follow the path and the logic of what's happening

[10:20](https://youtu.be/SEUu9GkbnxE?t=620) which is very different from generative AI and it has a lot of possibilities it's not deterministic and it also has different innovation potential but it lacks predictability and has this black box where we don't really know what's happening and it cannot produce the same output given the exact same inputs methods and circumstances that aren't guaranteed to repeat so this unpredictability especially is a risk for generating or confabulating the false information which is a challenge when we need high accuracy and reliability in places like governments but also healthcare so one of the key takeaways is that machines alone cannot become moral agents this is why both systems require human intervention and that's why we need human centered design or other approaches like reinforcement learning with human feedback to ensure accountable applications and so with that in her excellent op-ed if you've read this in the journal nature Pratyashree Kaluri stated don't ask if AI is good or fair ask how it shifts power and her key point is that most AI is built on a surveillance model taking data from a wide range of sources without direct consent so performing limited anonymization and using that data to build powerful accurate models of prediction and so these practices of surveillance tend to reinforce long-standing marginalization of women, black, indigenous, LGBTQ, poor, disabled individuals and AI research and corporations need to address this marginalization and bring members of these communities into the core of building new models and tools which is I think one of the essence that human centered design does and so multiple harms stem from the exclusion of marginalized communities and training data sets and reinforcement feedback and we when we continue to exclude we only continue to reinforce current systems of power and we rely on these tools as we grow including our

[12:24](https://youtu.be/SEUu9GkbnxE?t=744) concentration of power and so Jack Clark who is an AI policy expert who also works for Anthropic who are the makers of the gen AI called Claude also said in a testimony in the UN Security Council that AI is a form of labor that's being developed by one narrow class of actors which is companies so we should be clear-eyed about like the immense political leverage that this affords and if we create or substitute or augment for human labor and sell it to the world we have to become more influential than time and so the more concentration that we have of this power and we don't know how accountable it is we also have the counterpart of human centered design which you might have seen this before it's a structured process that helps us increase our confidence and our competence about our communities their needs centering them so we don't rely on guesses and we rely on this wavelength approach where we go wide to understand the right problem so we can go narrow to to define the right solution built on these layers of iterative feedback and we have already seen this in public service both at the local state as well as federal level in Akron including in Topsy our program HCD has helped tenants to vet properties by their eviction history so they can choose safe and affordable housing Allegheny County uses personas to improve access to drug and alcohol treatment and for my real client experiences and then the VA uses journey mapping to personalize care to veterans to embed trust and value as a core service principle but the problem with a human centered design on its own is that it has a problem with scaling especially with large volumes of data when it comes to the human experience in the context so cue in human centered AI which combines both worlds and it argues that responsible AI

[14:26](https://youtu.be/SEUu9GkbnxE?t=866) is human centered and human centered AI is about ensuring that we would we build benefits people in society and how we build it begins and end with people in mind it also means that we are thinking about people early from the research decision makings and behaviors and using that understanding to drive all technical decisions of a system so that's where you have these terms called like humans in the loop or in the center we emphasize ux design and measuring human performance to celebrate the new power that people have, but it also differentiates AI by shifting that center from algorithms and AI systems and replacing with humans at the center. And what that looks like, it's about augmenting human capabilities. It also helps us to extend human rights and underpins the government's duty to protect, fulfill, and respect its community. And AI doesn't undermine the basic principles of human-centered design, which is people-centered, iterative. It enables us to overcome the past limitations in scale, also in scope and learning, of human-intense design processes. So both fields have the power to shape each other. And AI is a deeply human endeavor that emerges from a long history of broad collaboration. So good news is, human-centered AI, these two different processes are actually quite similar. They're very aligned and compatible. And I know the processes that you see on my screen, like empathize, hypothesize, define, iterate, prototype, and test, they look linear. But in my experience doing this for the past 10 years, it's actually not always this way. In fact, I'd love to actually pivot now to a real-life example from my own work to show you what the process looked like. And I'm going to share with you some examples from Google's People and AI Guidebook, but also Microsoft's Human AI Experience

[16:30](https://youtu.be/SEUu9GkbnxE?t=990) Playbook that operationalize human-centered design practices for creating responsible AI. So let's just dive in. The Washington State Board of Health, or I'm going to refer to it as the board, was considering mandating the COVID-19 vaccine for school entry. And it sparked widespread public engagement. And this proposal led to forming this technical advisory group in December 2021 to deliberate on this policy. So during this process, the board was inundated with this deluge of feedback, over 30,000 emails and 50,000 community public comments in town halls and surveys, et cetera, both coming from genuine inquiries about health decisions, but also we had orchestrated misinformation campaigns. And so, you know, we're facing with this overwhelming volume of data. And then you also have the constraints of a very small team in the board with only two people responsible for media relations and public records and customer service. The board found itself at a crossroad. The challenge was the team did not have the capacity to analyze the responses. And a number of the inquiries were constituents who were actually seeking information that could help them make informed health decisions. But the comments were muddied by the oppositional commentary and without external help, the team will not be able to identify like key areas to focus future information campaigns to the public. So I worked with U.S. Digital Response after my tenure as a technologist in residence in their health program as a mixed method researcher. And I collaborated with a brilliant computer and data scientist, Dr. Timobi Ucha, to solve this problem. And we also collaborated with the board, the communication manager, their policy team, and also USDR's scoping team. So one early takeaway

[18:34](https://youtu.be/SEUu9GkbnxE?t=1114) is that we paired design, research, and data science very early on for people centered problem solving. So part of us scoping this project, we had to ask what challenge are we solving together? Why does this matter? Why now? Who are the people we wanted to help? What do we know about the people and from existing research, from even adjacent research from other states, as well as what does success look like? But this was a really good time for us to also examine our biases. For example, I asked what makes an email trolling because I heard that word multiple times. What words or phrases are used in this context and in Washington State's context? And what value do we assign comments and public comments that we receive from the public? And between myself and my colleague, we wrote our own hypothesis of how we think part of this project fills in. We believe by doing this project for this people and by this people, we mean for the board itself, for decision making, for residents, but also for me because I'm making decisions as a researcher of where to focus our efforts. And this is a template that you see here on my screen from Google and from their playbook, which is useful, that focuses on AI. And I want to give a caveat.

[19:50](https://youtu.be/SEUu9GkbnxE?t=1190) We actually never used the word AI or machines who are doing the work in this project. And I think the language that we use was extremely important for us to frame how we communicated and thought about our process to our team and our community government partner, which highlighted to us that we needed to do two things. One is that we needed to automate the task of analysis and pinpointing us where to actually focus the insights, but also number two, to augment human capabilities, especially for me as a researcher, because I didn't have time to look at 80,000 comments. I had to figure out a way where we can create a process to funnel on the right approach for clusters of comments and words to help us make core decisions and also understand that the context helps us understand lenses of truth that surround the parameters of the data. So a few tools that I wish I had, to be honest, access to, but we did it informally and in other spaces of this project. These are tools called, from Albert Maiz, AI Canvas and the AI Prompt Card Deck. They help you understand the people we serve early from their values, their principles, decision-making, who would drive the technical system. And that designing interactions means that we have to design every interaction with people around the context in which the person will be interacting with your system, your model, including their environment, the social context. It also helps us set the right expectation to our government partner about what we understand about our participants, including the residents who provided the comments. And it helps us also plot assumptions on the AI Canvas.

[21:31](https://youtu.be/SEUu9GkbnxE?t=1291) We visualize the risk, so low risks for low uncertainty, and then you have high risk for high uncertainty. And it clarifies to you the meaningful use cases to apply AI capabilities by leveraging people needs. What's funny and interesting to me is that human-centered design is about embracing uncertainty, where we embrace living in the gray area, but AI is about reducing that uncertainty. So deciding between automation and augmentation is really important. Automation means the machine will take over tasks, and augmentation means that we're improving human-machine collaboration. So if you do it correctly, you can both streamline complex processes to drive solutions where AI contributes unique value. And that was our case. We wanted to pair both of them. And that meant that we landed on recognition as the first approach to automate task and sentiment analysis, and drive this funnel of decision-making towards text analysis. And this would help prepare me as a researcher to analyze key themes when I recognize key patterns. And that's the most desirable, viable, but also technically feasible collaborative approach we thought about at that time. And we adopted a large-scale sentiment analysis, and then text mining, using a technique called natural language processing, or NLP, to recognize patterns in the data. But also know that there are these biases in the data, which can propagate biases in their output, because we did have a lot of targeted campaigns from misinformation, disinformation, which can also skew the training data. So our goal was to also mitigate these biases. And we'll show you some of the techniques that I was thinking about with our team. So our first milestone was sentiment analysis and text mining using PDF records. We had tons and tons of PDF records from town halls. We also had emails that we received from residents, and also survey

[23:32](https://youtu.be/SEUu9GkbnxE?t=1412) responses. And we also created, eventually, a list of names and keywords to understand if these comments were positive, neutral, or negative. And we also did a phased approach to understand where they are when used R as a free open source tool for us to analyze. And we did exploratory research. The first thing we did is that we had to work with our communication manager at the board, so we can understand the nature of these public comments and how they were used and integrated. And one of these tools is called a data landscape canvas to help you get to that level of understanding this collaboration between your organization, the people, and the data insights. And what we asked is that, how was the data collected? From whom? What language was it shared? If it's only in English, did people with other languages had an opportunity to share it that way? Is this verbatim or proxy data? And what are the typical organizations who are providing the comment? Because some of these are people who are affiliated with certain organizations.

[24:33](https://youtu.be/SEUu9GkbnxE?t=1473) And who gets to provide comments? And at what time do they provide them when they attend town hall meetings? And how accessible these are? And those types of questions help us understand whether the data is also public or private, that's helpful for user needs, and how it would influence AI tuning for precision and recall. And most importantly, it helps us understand that non-verbatim records, especially handwritten notes that we receive from phone records, we had to exclude them because there could be potential biases in terms of who's filtering that information and how they understand what the person who's communicating on phone giving to them. So it opened up an opportunity for us to improve also the data collection pipeline in a different project. But also, it helped us understand that the data assets teaches us that if AI is supposed to work on a broad range of people, the data should be representative of all those people. That means that if people don't exist in the data, they will not exist in the model or the insights.

[25:39](https://youtu.be/SEUu9GkbnxE?t=1539) And so that's something that was very key for us, because it helps us also understand, do we have control over that access and representation in this project? And how we might increase that in future projects. So early on in this process, when we're ideating, is that we try to collect this funnel approach, and we had this instincts and hypothesis, potentially we might use negation. And I'll show you what that means, like using words like not, no, without. And we found that the data is imperfect, especially because the sentiment analysis had issues with the reliability of sentiment of certain words, especially considering there are these clusters of campaigns that are targeted at certain times and days for misinformation, potentially some disinformation. But we also had these echo chambers that are right for manipulation. Either they're consciously or unintentionally, which explains how it skewed a lot of the data. And this is just an example that you can see that we've received from multiple types of similar campaigns that are copy pasted via emails that we hadn't understand. So early responses, when we're thinking about like, what's our initial sense of this model, is that we did exploratory just to figure out what the general keywords we're hearing. And you can see here, we use words like not, without, just to get a sense. But to be honest, as a researcher, I'm looking at this when I first got into this project, like, this doesn't really make sense to me, because human speech, not everybody's going to think about opposition against vaccination mandates using this type of lexicon, like not, without, please stop, without stopping, or do not prevent, etc. There are some of that.

[27:21](https://youtu.be/SEUu9GkbnxE?t=1641) And so initial hypothesis for me, I was like, I don't think I think there's something has to be said for understanding that for every statistics that we see, the truth lies somewhere in the parameters around that numbers, and we need to dig deeper into that context. And so we needed a different approach. And some of these approaches we had to test very early on, this is something that we didn't do. But if I had more time and resources, one approach that you could do is for you to experience the data and see the world through its lenses. And I would have done data ethnography, where you can simulate an AI experience, like Wizard of Oz testing, with team members, where testers simulate the interactions and responses of an AI system to empower communities to identify and address potential issues really early on. But one other, one of the things I've talked to with other practitioners in the space is that most of them test for the MVP scenario, which is typically at the end, and they focus. And the spaces that I've been in is that I don't typically see testing with humans in the loop before they deploy the model, especially in AI based solutions. And if they do, they're often testing for this ideal design scenario that they've already picked. So if error cases were tested at all, it was often done in an ad hoc way, which I have to admit, this is something that happened to me, and I've experienced it, and trial and error. And to be honest, it's really hard to know what the errors are until like bad things happen. And that's because AI is probabilistic. It's designed to generalize new scenarios and can learn over time, all of which can make it difficult to know when it could fail. But now there are tools that I learned about to help us proactively and systematically explore some common foreseeable type of failures that we can plan for and test in advance. And on my screen is Microsoft's open source tool for natural language processing, or NLP,

[29:22](https://youtu.be/SEUu9GkbnxE?t=1762) like search information that we can use early in our development process to explore different choices and help us better design our user experience. So the pipeline initially was for us to scrape PDFs, emails, figure out the sentiments, help us funnel through where the concentration of these sentiments are, and then use text modeling to understand where the clusters and the common words that we're learning, and for us to manually create thematic analysis using Atlas TI and MIRA to generate insights. So we did extensive research, not only in the state of Washington, but in general. From all the past work that I've done from other states, in terms of COVID-19 crisis response, is that we focus our keyword analysis on a broad spectrum of categories, ranging from organizations, influencers, including those who are involved in organized trolling to understand what are some of the keywords that we've learned, and early testing from sentiment analysis helped us get to some of the signals and the noise.

[30:26](https://youtu.be/SEUu9GkbnxE?t=1826) And this work is grounded in collective labor. And one of the key takeaways is that we should recognize and speak about AI tools as products of human collaboration and effort, rather than as independent entities and their own intention or abilities. And so this is one way where we're creating collaborating with the board as well as our team to start building what we understand of human language. So instead of saying like, the AI figured out all these keywords, we typically say that the design engineering teams were behind the selection of these keywords and the approaches, which have led to X, Y, and Z. So an example of this is from a public comment here from one town hall in February to March 2022, where we got to see like where exactly some of these nuances and words happen. And we learned that our prototype model, what we built was only 34% relevant.

[31:26](https://youtu.be/SEUu9GkbnxE?t=1886) So it's kind of low. We know that there are things that could eventually go wrong and we wanted to understand where will AI fail in our model. And what are the design interactions so that people understand why we made certain decisions and so that's where we included a relevancy model. And when you're designing for AI, the model will have to be tuned for precision and recall. That basically means when you're creating the model, you have to decide how accurate you want the model to be in identifying true results. That's what we call precision versus not missing any important results, recall. And so this decision, it's also known as designing the reward function, which involves the collaboration between UX design research product and engineering, and it's so important because it greatly impacts how well the AI works and the user's experience. So it's kind of like, imagine an AI diagnosing cancer, it could either really identify real cases of cancer, that's precision, but possibly missing some cases or it could aim to catch as many cases as possible, recall, which could risk some false around.

[32:37](https://youtu.be/SEUu9GkbnxE?t=1957) So that was like one of the things that we had to think about. And one way for you to think about it is when researchers and designers pair with data scientists, their job is to help them decide what to optimize for, which experience will be less prejudicial to the end user. And providing meaningful insights about human reactions and human priorities can prove the most important job for us researchers and designers in an AI project. And this is a another template from Google's reward function playbook that based on the person's attitude towards automation augmentation, the team with the rest of their colleagues have to map out all the scenarios, especially one that can be false positive and false negative, and list out every instance of each reward function. And so you can consider looking at false positives and false negatives and evaluate, does the service that you're providing offer the most useful benefit for fewer false positives, so you have to consider optimizing for precision.

[33:43](https://youtu.be/SEUu9GkbnxE?t=2023) But if your service is offering the most useful benefit for fewer false positive, false negative, sorry, consider optimizing for recall. And so our model was optimized for recall because it was really important for us to capture as many relevant comments that were muddied in the opposition. And also given that we had a lot of public health implication, and we understood that the trade offs for choosing this method was to make sure that we possibly include as many relevant comments but also ensure we don't miss any important sentiments and perspectives, and people are seeking care. And so, to identify trolling in large data sets like this, we had to do is text analysis and understand where these clustered happen from the keywords that we identify and created patterns and find some detect any anomalies. And so, the process involved us to have this preliminary analysis to find these keywords, figure out where the clusters to isolate the trollings and the common copies and pasting that we see in emails, and we actually manually reviewed thematic analysis using stratified sampling to understand where these clusters happen and what the core insights that generally came from the content and so our precision recall score actually from our testing improve significantly it jumped from 34% to between 91 to 92%, which was actually the best that we could do with our approach.

[35:14](https://youtu.be/SEUu9GkbnxE?t=2114) And through this process we actually dived into a spectrum of beliefs, values, attitudes and behaviors, which underpins vaccine acceptance, and it's actually a model called the health belief model that shows that this trajectory as we make decisions as human, and we shared our insights with the board, including our process, our GitHub repository, our survey analysis, and we found four important things. One is that there are these several barriers to vaccine uptake, including belief systems that's relevant to economic, cognitive, health, technical, relational, normative, and even power barriers for example like difficulty to secure vaccine appointments because of work, religious constraints, the influence of social networks like your friends on your decision making. But the thing that really interested me the most is that we uncovered values related to, of course we had risk aversion, and people wanted personal agency for vaccine and attitudes towards vaccine misinformation that range from skepticism to outright refusal.

[36:20](https://youtu.be/SEUu9GkbnxE?t=2180) What was interesting is that we found that there's a spectrum of people engaging, interrogating, and critically thinking with resources that they provide for questions that they had, irrespective whether you're on different end of the spectrum of acceptance or refusal. And that showed us that people were engaging with materials that could have conflicting sources, and have information overload. But we also saw that a lot of these behaviors towards vaccine acceptance were also influenced by needing more evidence but also the need for reciprocity for the board to engage with them in a conversation. So we were able to run these authoritative mandates for future decisions about like what's the future of their kids. So what, like, why is this really matter, and it opened up three opportunities for us to chat with the board about some communication principles, one of which this reciprocity bias in which people needed to feel assured that they were heard, and that they can examine the sources, the information that they have with the board.

[37:24](https://youtu.be/SEUu9GkbnxE?t=2244) It also offered opportunity for people to reduce this context switching of going to multiple sources and various information for them to have an opportunity to discuss where the board can be that source of truth for them. And the third opportunity is that how do you communicate with empathy so that people are seen, heard, and acknowledged with their concerns, their rights, their agencies, and have full ownership of their freedom in a way where we can tackle that. And so that offered us an opportunity for topics where we can, where we aggregated from the text analysis where the clusters happen of the key topics where we saw opportunities for people to discuss for residents had questions, comments, feedbacks on including risk, how do you evaluate risk, especially for a child. What's the role of a child in being an anchor for vaccination. We also saw that there are key communication values, we've seen in the data, where role models, show me the evidence that you're talking about before you make those decisions.

[38:30](https://youtu.be/SEUu9GkbnxE?t=2310) Empathy were reassure me that this is not a paternalistic demand and acknowledge me with plain languages. What happened at the end is that this helps the board reconfigure how they communicate with their team, and how they design future communication campaigns and help them make those kinds of decisions. So you can see here that they wish they had, you know, more funding to this type of work. They're aware actually of reciprocity concerns and opportunities but they did not know the extent of that language. And so, this is a great example of how this can be applied to different data sources, and can be abstracted to different issues, and also opened up that opportunity for a different project to improve their data collection pipeline to increase to improve the reliability of their data. So the main takeaway here before we head on to the, to the panel discussion is that language and learning is deeply human, it goes beyond syntax of where we started from.

[39:32](https://youtu.be/SEUu9GkbnxE?t=2372) Don't stop without, and involves the meaning that comes from our own experiences and human centered design is grounded in collective labor where intelligence is distributed, and it requires, don't even require me to do some manual work. So we have a few resources, if you want to nerd out about the project, we can do it in the q amp a or one on one after the workshop. And there are so many resources in terms of books, community slack channels to explore this in your own time. And I'm excited to shift now our attention to our panel speakers with the city of Detroit for their gen AI and HCD collaboration project. So I'll go ahead and introduce the panels. We have Connor pets, who is a senior mobility strategist with the mayor's office of mobility innovation at the city of Detroit. He has so many years of experience working and leaning innovative projects in the municipal world, ranging from automated vehicles to robotics projects to policy development that can lead to better quality of life for Detroiters and Connor has a master's in urban planning and a bachelor's in environmental science with over five years of experience in the public sector, working with the Southeast Michigan Council of Governments suburban mobility authority for regional transportation, and most recently with the city of Detroit.

[40:56](https://youtu.be/SEUu9GkbnxE?t=2456) We also have arena Johnson, who is the climate equity project coordinator at the east side community network on the east side of Detroit, where she has strong ties to community and her work she actually focuses on climate and mobility project that centers the communities in place and she's currently working in her master's in community development, to better understand the unique ways that communities contribute to improving the quality of life of other people through equitable solutions. Last but not least, we have an order by who brings extensive experience in cloud commuting product management and entrepreneurship to the forefront of AI innovation and as the pro pro bono technologist and product manager at google.org she spearheaded the top cities Detroit project, which successfully developed an AI driven automation solution that will touch on that streamlined survey data processing in collaboration with the Detroit office of mobility innovation, the Center for public impact and the And I wanted to just give a heads up what, what is the problem in the project that we're starting with.

[42:17](https://youtu.be/SEUu9GkbnxE?t=2537) And then we're going to head off and start with arena and Connor first. So the project we've learned is that a lot of the residents in the city of Detroit were inundated and overwhelmed by the amount of times that were surveyed to to learn information in terms of their experiences, as well as their attitudes and beliefs for mobility challenges in the city. The other end of the spectrum is that you have city administrators, employees, staff who are having difficulties, making informed decisions about the future of community mobility, because they don't have access to this data so it's not centralized. So going into this project human centered design is about putting humans in the loop and I'm curious to start with you, Connor.

[43:23](https://youtu.be/SEUu9GkbnxE?t=2603) When you first started this project, what did you first expect when you started doing research with arena and the ECM team.

**Connor**

[43:32](https://youtu.be/SEUu9GkbnxE?t=2612) Sure. So thanks for the question. And I think the initial thing that was thought about when it came to expectations was definitely thinking back to anecdotal experiences that we've had during community engagement efforts, as well as just general anecdotal experiences throughout its city administration. So our expectations were essentially that we were finding a lot of redundancy and frustration with redundancy amongst community members, as well as growing frustrations of the non centralization of data in administration. And with that, we kind of developed this effort which will definitely get more into as the panel continues but my expectations were definitely that we needed to develop some type of solution that bridged the gap between this confusion of non centralized data, which ultimately leads to the redundancy and does lead to redundancy and community engagement efforts which is what we were trying to avoid and essentially what we a solution that we ended up developing throughout this, the process of the opportunity project for cities.

**Elham Ali**

[44:49](https://youtu.be/SEUu9GkbnxE?t=2689) Yeah, and it sounds like also trying to reduce that type of fatigue of continuously engaging and extracting from the community in a way that we're not. able to give them back the information. And I'm curious also from that lens arena is that ECN has these deep roots to the community and you led majority of a lot of these interviews with mobility advocates and city workers so that you can understand their experiences and needs with accessing mobility data. And this is just an example of a quote that we had from one of those interviews. Can you walk us through what did we learn from that research as in what did people

**Arena Johnson**

[45:30](https://youtu.be/SEUu9GkbnxE?t=2730) need to accomplish their goals? Yeah, so when we interviewed some of our advocates we learned a lot about what was missing from their perspective and their ability to get access to mobility data. So we learned that yeah so in order to get access to some of this data it was confusing a lot because some of this data just wasn't easy to find and it wasn't digestible for folks. Some of the data that folks were looking for included things about budgets for transportation and mobility projects or just data on how many people are using a vehicle in their neighborhood and that would help advocates better understand what projects to implement. So yeah finally we just kind of understood that from previous community engagement this just wasn't the data and the plans and the projects that were conducted weren't easily accessible nor digestible for folks so that definitely helped us in going forward with this project.

**Elham Ali**

[46:39](https://youtu.be/SEUu9GkbnxE?t=2799) Yeah and I remember the team has created these personas to understand the different lenses of not only needs and experiences and journeys that people go through to make with the data but also in terms of their work and their lives. And Connor when you heard a lot of the insights that we learned from the research how did your expectation about the project the problem and your understanding of it maybe change or remain the same after learning about the research insights?

**Connor**

[47:14](https://youtu.be/SEUu9GkbnxE?t=2834) They changed definitely by understanding just how large and complex the issue actually was and I think how built out the solution needed to be. The problem with it being so multi-faceted and basically being like a domino effect where there's confusion on the administration side about certain engagement efforts and leading to the redundancy which leads to frustration and then additional confusion in residence. That was obviously like the main thing but I think really understanding just how large the problem was is the biggest thing considering the solution that we had developed. We had developed in a very limited time frame and we're still working on it to this day and it will be a continuation and iterative process as we keep moving forward. This is going to be something that continues to be built upon

**Elham Ali**

[48:14](https://youtu.be/SEUu9GkbnxE?t=2894) and we'll definitely see that in the coming months. Yeah the scale of it I remember learning about that was really interesting but also shocking at the same time and also not shocking at the same time to know the kind of work that we have done in the community and the type of collaborations that it took multiple times and iterations. On your end, Anuradha, when you and the team of technologists at google.org that worked on this project when you were working with the city and ECN to brainstorm solutions, how did their research

**Unknown speaker**

[48:49](https://youtu.be/SEUu9GkbnxE?t=2929) insights inform you that GenAI was the right tool for the job? Yeah sure. Hi Adam and hi everyone. So like Irina and Connor mentioned there was lots of extensive research that was already done and the inputs what we received was that there is frustration in residents that same services are being repeated and for city team there is lots of data but how to get the meaningful data out of those large reports was a problem. So for us the challenge was that how do we address both of these problems and when we were working with city team with Irina and everybody in the team, we figured it out that if we build a centralized dashboard where residents can see which service were done, what actions were taken and what was the summary of those service. So that might address some of the problems and at the same time for city team that will be beneficial because they can see the information what service were done and what actions were taken so that the service will not be repeated and further actions can be taken on top of that. So when we figured it out that this information is available in large pdfs, service are done by different organizations so these pdfs are not in a consistent format. Every pdf has different information in a different place so that is when we all had this time where we figured it out that this is a problem that we can solve with generative AI specifically LLM where we can feed the pdf and we can extract the relevant information out of those data and then we can present it on the dashboard.

[50:35](https://youtu.be/SEUu9GkbnxE?t=3035) So this was the time when we thought that AI would be the right tool for this.

**Elham Ali**

[50:41](https://youtu.be/SEUu9GkbnxE?t=3041) Yeah and what I'm also hearing is that you also figured out the jobs to be done, the core jobs that the residents and the community advocates as well as the city administrators will need to do and how they read that information in a way that they can extract it to make core decisions and I love that because that's one of the core needs that we're looking at also in human-centered design is to understand these core jobs people need to do to improve how they think, how they behave, how they work and collaborate and when you're we're going to get into how we explored the model and ideating and I remember that you touched on Anuradha as well that we had to look at different types of tools before GenAI so we looked at chatbots, we looked at chat pdf, we also looked at existing solutions like this dashboard from another solution to provide us with the same opportunities to look for information and centralize it and we eventually landed on like you've mentioned an AI automation interface that extracts key information from long form pdfs and I'm curious Irina to start with you what were your initial reactions coming from the

**Arena Johnson**

[52:00](https://youtu.be/SEUu9GkbnxE?t=3120) community side about hearing GenAI as a driver for the solution? Yeah that's a great question I think from the community side and as a community partner on this project as well our knowledge was limited on this being the driver for the solution but I think that we had a lot of trust with the folks that we interviewed and then also just internally we had a lot of trust on this team, this project team, the dynamic project team that kind of like facilitated this, that helped facilitate the solution and so we just had an understanding that this was a complex problem and needed a creative solution and so I think together as a team we did that and kind of like thought outside of the box to come up with a solution that would solve like we've been speaking

**Elham Ali**

[52:54](https://youtu.be/SEUu9GkbnxE?t=3174) about such a complex problem. Yeah and I'm curious to get back to it as well when you've done testing with the communities how you thought about that in just a minute because I want to touch on that and in terms of Connor from your perspective when you've heard at least from the city side of potentially GenAI as a solution from what Anuradha was mentioning how did you expect such a solution would change your work in terms of mobility data I mean even of your teams like

**Connor**

[53:24](https://youtu.be/SEUu9GkbnxE?t=3204) would it improve it change how you look for information etc. Oh most definitely those last two points that you hit on it would definitely it is improving our work and it's improving our work by allowing us to be able to go in and understand what has been done and seeing it on a visual digital platform considering that this AI system that was developed doesn't just extract the information but you know the second leg is of course putting it into a visualization tool that was developed in ArcGIS and I think that understanding the geography of where some of these things have or some of these engagement efforts have occurred as well as understanding like how widespread the information is and where we're missing some information and understanding gaps and seeing where we need to prioritize some of our work that is probably some of the most important things that when when it comes to changing the way that we work changing our dynamic as well as understanding what some of the other departments and or counterparts have done so it's not just about mobility it's about building it's about building first with mobility and with transportation and then understanding again as this continues to be built out we want to add in other facets we don't want to just keep this in transportation and mobility we want to expand this to other I think you know data points that include things like maybe housing economic development other sectors that still intersect with transportation and mobility that would still be extremely useful and helpful not just not just for us in the city administration to know but also the community and community leaders.

**Elham Ali**

[55:25](https://youtu.be/SEUu9GkbnxE?t=3325) Yeah I love that scale and I think it touches also on this idea of trust that I see also here in the chat which we're going to jump right in and hot second in terms of scaling a lot of the work that we're doing here to other departments so that they can benefit. One of the things that I've seen in terms of developing AI supported prototypes is this belief in this black box which is I'm going to give you this model and you need to trust it without knowing what's going inside of it and it can have the potential to deliver a lot of mistrust in the long run but with the communities but it also means I'm very wary of outsourcing all decision making to AI systems and I want to make sure that we focus on the role of humans and AI augmented system and this is a picture where the team in this project was able to collaborate in the open and share their code to the team and I wanted to first start out Anuradha if you can walk me through how did you balance using Gen AI with the need for human oversight. Maybe give us an example how human judgment played a key role in decision making process that was initially designed to be AI driven.

**Unknown speaker**

[56:52](https://youtu.be/SEUu9GkbnxE?t=3412) Yeah sure that's a great question. So when we analyzed different tools and we found that for prompting we need to provide multiple prompts to get the right answer and sometimes it happens with AI that the answer is not going to be consistent what it gave earlier might not next time it might not get the same answer and sometimes it might extract information which is not intended. So in all of these scenarios to address all of these scenarios what we thought is that if we put a human validation after AI extracts the information that way we can manage the information that will be finally available to see for everyone. So with CTDM this was something that we all agreed that first the AI tool will extract the information and then like in this interface we can see that users can change and update any of this information. So some of the information might not be even present in the PDF and that can also be added but whatever will be extracted can be modified and then only after a human confirms then it will be fed into the pipeline and to the dashboard. So this is how we are adding the feature to make sure that human-centered design and information is available. Yeah and I see here just a couple

**Elham Ali**

[58:18](https://youtu.be/SEUu9GkbnxE?t=3498) questions in the in the chat and this is maybe for you Anuradha and Connor is was there any

**Connor**

[58:25](https://youtu.be/SEUu9GkbnxE?t=3505) personal identifiable information that was used as part of this project? There was not any personal identifiable information that was part of this project nor was there any PII collected during the community engagement process. I believe putting any type of PII and something that is essentially you know essentially working towards a somewhat open data platform is a big no-no especially when it comes to government. So I'll leave it at that.

**Elham Ali**

[59:00](https://youtu.be/SEUu9GkbnxE?t=3540) Yeah absolutely and to that point Connor like what measures did you implement to make sure that your team and stakeholders that you work with understand the limitations and capabilities of Gen AI used in this project?

**Connor**

[59:19](https://youtu.be/SEUu9GkbnxE?t=3559) Yeah of course so I think the biggest thing when it comes to understanding like the limitations like when we were at the community engagement event a lot of the participants expect expected this to be straight away like a perfect solution a perfect catch-all solution I think at first and I remember Irina and I continuously iterating that this is just that this was initially just a draft product. So I think managing the expectations that this is a that this again like is an iterative process of continuing to build it out and that AI is a very new tool that is being introduced into government and we're still exploring not just How necessarily we can use it as a tool, but also how to regulate it. So this project definitely helped, I think, with understanding a little bit more on the capabilities currently of AI and what it can be used for, as well as feeding into general AI policy and ordinances that will be developed by the city in the future. And for context for folks here,

**Elham Ali**

[1:00:27](https://youtu.be/SEUu9GkbnxE?t=3627) the community event that Connor is alluding to is actually a community demo day, where they demoed the prototype to the communities, including advocates and folks in the city, to get their reaction, which we're going to talk more in a bit and understand what trust means at that level. And in terms of prototyping, can you walk us through how the prototype works? I have a couple slides here, and I blasted it, the screenshot, just so that folks know how it works. And maybe also touch on, is the product fully gen-AI powered, or how much do humans

**Unknown speaker**

[1:01:04](https://youtu.be/SEUu9GkbnxE?t=3664) manually operate it, if you can walk us through it? Sure. Yeah, so this solution has three parts. First part is reading PDF, extracting information out of that, and this is done using the AI tool. The second part is the data pipeline, which takes these metadata generated by the AI tool and the summary of the report to the dashboard. And the third part of the product is the dashboard, which is a centralized dashboard, which provides this information. It presents this information visually to users. So basically, to build the first part of the project, we used the AI tool, which is extracting information out of the PDF file. And then there are certain things that might not be available in the PDF. Like we found that zip code, which is very much required to do the geomapping on the dashboard, this information was not available in most of the sample reports.

[1:02:08](https://youtu.be/SEUu9GkbnxE?t=3728) So we added a couple of fields where the information can be fed manually, because when we show it on the dashboard, we want to map every report with the geolocation to make it more relevant for the neighborhood, which studies were done and what actions were taken, or any other information relevant to that survey. So for these kind of mapping, like zip code neighborhood, this information, if available, it will be provided using the AI tool. But if not available, then we are adding over here a provision where the information can be fed manually. So it is

**Elham Ali**

[1:02:45](https://youtu.be/SEUu9GkbnxE?t=3765) paired with three different parts to complete the solution. Yeah, and I see here, Anuradha, that there are these fields that says manual input required. Can you talk a little bit more about, it sounds like there's this hybrid between manual inputs and Gen-AI operated.

**Unknown speaker**

[1:03:06](https://youtu.be/SEUu9GkbnxE?t=3786) What are these for? And how do people interact with them? Sure, yeah. So basically, yeah, you were right that this is hybrid between AI and manually provided information. So what happens is that we parse the PDF using the AI tool, extract the metadata out of that, including the summary of the report. But there could be certain information like, for example, zip code, which we saw most of the reports were missing this information. But this is very critical to put it on the dashboard. So these kind of information we are taking, we are asking for manual input. Other things might be like list of reports, which might be in the next section. So list of reports relevant, for example, one study was done on a particular topic, and then there are relevant reports on the same topic. So the PDF, what we are scanning at one point of time might not have that information, will definitely not have that information. So we are asking user to provide that information so that on the dashboard for users, we can link those related reports. So these are the things that we are asking like zip code and neighborhood and report title will be generated automatically, but like council district number. So these are, yeah, like we are feeding it beforehand and users can select and add that information.

**Elham Ali**

[1:04:33](https://youtu.be/SEUu9GkbnxE?t=3873) That's really helpful to understand that there is this cross collaboration, like we continually mentioned that it is collective labor for that cross cutting linking. And we've had that demo day that we've just recently talked about. But I'm curious, from Irina, your perspective, when you're looking at this tool, how is this prototype beneficial for like the average person?

**Arena Johnson**

[1:05:02](https://youtu.be/SEUu9GkbnxE?t=3902) Yeah, I think the great thing about the prototype is that it's beneficial for the average person, but that's across the board, ranging from like the city staff member, the resident and the advocate. So in our community mode, we did speak a little bit about how that helped each persona, each person. And so for example, like the resident can now be able to look on the map visualization feature and find one of the existing projects or plans that they got feedback on. And now they can know what's going on in their neighborhood pertaining to mobility and transportation. They can understand the feedback that was given during that time. And then the city staff member can now like easily access all of this data in one place, right? And so they can now make some informed decisions on what mobility and transportation projects to implement. And then as well as the advocate, the advocate can now see what unique approaches can they implement that falls in line with the existing projects and plans in the neighborhood. So I think all of that works together. And so it can be beneficial for all of these folks. And then lastly, mentioning like the features of it, the features of it, the map visualization, the search and filter function, right? All of that makes it easier for each individual to be able to navigate this tool. So I think that definitely helps.

**Elham Ali**

[1:06:40](https://youtu.be/SEUu9GkbnxE?t=4000) Yeah, absolutely. Improve the way they work, they collaborate, they communicate, is all things that I'm hearing. And while new technologies have made it easier for us to develop AI, it's still pretty hard work. And we don't like taking any shortcuts, especially when it comes to collecting data, developing algorithms and testing it because it can undermine the integrity of the product and the process, but also the integrity of the trust that we have built with our communities. Can you walk us through Anuradha about the rigorous data collection and algorithm development aspect of the project? How did you make sure that these processes were rigorous

**Unknown speaker**

[1:07:25](https://youtu.be/SEUu9GkbnxE?t=4045) and reliable? Sure. So basically, when we are addressing this kind of solution, for us, one of the challenge was that what kind of data will be fed and can we get a diverse set of that data so that we can make sure that we are able to address most of the scenarios? Second part is that which AI tools should be used, like which Gen AI or LLM tool should be used? And for that, we analyzed a couple of tools and we provided that analysis to our CT team. And from there, the information, so once the tool was identified that we are going to use chat PDF, after that, we used different samples that were available for the PDF. And then we started analyzing the output that it was generating. And when we were analyzing different tools also, at that point of time, also, we were feeding all these different PDFs and we were extracting information and comparing the outputs. And we benchmarked the manual input that we received that manually, this is what is expected response from a PDF for all these different fields. So while doing the analysis of different Gen AI tools, we were feeding these PDF files and we were benchmarking against the manual information. So this is how we could build the analysis that which tool should be used and which will provide the more optimized information for the dashboard.

[1:09:03](https://youtu.be/SEUu9GkbnxE?t=4143) And after that, it was more about that what prompt should be given, because when it comes to LLM, we have to provide multiple prompts to get the right answer, which is like follow up, we ask one question and then following up with another question. So when we are building this tool, one of the challenge was that what should be one prompt that we can provide so that we get the answer in first shot. And for that, we had to do our own analysis that how many prompts, like we provided multiple prompts, and then we came up with one single prompt which could address all of those questions. So this is how we figured it out that one prompt can extract that information. But again, it is a continuous process. The prompt needs to keep on refining. So this is the initial version where we provided that prototype where multiple tools were analyzed and multiple prompts

**Elham Ali**

[1:10:04](https://youtu.be/SEUu9GkbnxE?t=4204) were analyzed to get the right answer for the questions. Yeah, thank you for that. And I'm also recognizing I want to we want to leave 15 minutes for the audience to ask us questions. So we have more questions, but I also wanted to give you that time. My last question actually will be on the demo day that we held with the community. And my last question before we hand it off to Q&A is that the community actually interacted with the prototype. And I'm curious, Arena, how did the community react? And how did they know that they could trust the prototype?

**Arena Johnson**

[1:10:45](https://youtu.be/SEUu9GkbnxE?t=4245) Yeah, so I'll answer the first question. So in regards to how did they know they could trust the product, I think it depended a lot on the relationships that already existed. So a lot of the folks that were at the community demo day, we had previously worked with, we had previously had relationships with. So that definitely kind of like helped that trust aspect for them to kind of like, really lean into it. And because, you know, the prototype was still in its early phases. And like Connor mentioned, folks had some questions about that. So it was definitely a work in progress. So we kind of like leaned into that, as I mentioned before. And then yeah, we had to go back to the drawing board, we realized that a lot of the things that we needed to work on were like adding different reports to the dashboard and to the tool for it to be just a little fleshed out more for folks to understand and also specifically doing some more in depth work with folks to kind of like go through this tool and kind of to help folks understand how it could be helpful to their workflow and how it can be helpful just in general to like the residents and the city staff and workers there. So yeah, I think we had to kind of go back to the drawing board and kind of understand like how we can tweak this to also to make it better for folks and easier to understand. But definitely looping back around to say that that trust was there for folks to like lean into it with us. And to kind of like understand like, it's been a work in progress, but we're going to continue to work. We're going to continue to work on it. So

**Elham Ali**

[1:12:42](https://youtu.be/SEUu9GkbnxE?t=4362) yeah, I appreciate that and that loopback. And I want to give the audience an opportunity to ask any of our panelists any questions. And Hannah, if you could help us funnel this question or feel

**Unknown speaker**

[1:12:53](https://youtu.be/SEUu9GkbnxE?t=4373) free to also raise your hand and go on mute. We'd be happy to do that. Absolutely. Let me just stop the recording.

* * *

Original page: https://innovate-us.org/how-to-apply-human-centered-design-to-government-ai-projects- · Licence: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org