---
id: "workshop:prediction-isnt-intelligence-how-predictive-models-really-work-in-government"
type: "InnovateUS workshop"
title: "Prediction Isn’t Intelligence: How Predictive Models Really Work in Government"
url: "https://innovate-us.org/prediction-isnt-intelligence-how-predictive-models-really-work-in-government"
date: "2026-01-21"
duration: "60 minutes"
series: "Prediction, Automation, and Decision-Making with AI: Risks and Opportunities"
instructors: ["Arvind Narayanan","Rebecca Cai"]
recording: "https://www.youtube.com/watch?v=nJxKODRJ_sI"
transcript: "automatic, from the recording"
source: "InnovateUS (innovate-us.org)"
licence: "CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org"
---

# Prediction Isn’t Intelligence: How Predictive Models Really Work in Government

InnovateUS workshop · 21 January 2026 · 60 minutes

AI is not one thing. Government agencies are being offered a rapidly expanding ecosystem of tools from workflow automation to document classification to chat-based assistants, enterprise search, recommendation engines, and predictive models. But these tools differ in capabilities, requirements, risks, and appropriate uses. This workshop offers a clear conceptual map to help public servants understand the differences and distinguish real functionality from marketing hype. Participants will learn why prediction is not intelligence, why predictive systems have hard limits that matter for government, and how confusing automation with prediction could lead to costly or harmful deployments. Through case studies in public safety, benefits administration, and inspections, the session highlights how misidentifying the problem—rather than technical failure—often explains why systems underperform.

## Learning goals

- Distinguish predictive models from automation, retrieval systems, chat-based agents, and rule-based tools.

- Understand why prediction has inherent limits and why those limits matter in government settings.

- Recognize common points of failure such as biased training data, feedback loops, and drift.

- Build a skeptical, evidence-first mindset toward claims that “AI can predict X.”

## Speakers

- **Arvind Narayanan**, Professor at Princeton University & Director of the Center for Information Technology Policy
- **Rebecca Cai**, Chief Data Officer, State of Hawaii

Part of the series [Prediction, Automation, and Decision-Making with AI: Risks and Opportunities](https://innovate-us.org/workshop-series/risks-and-opportunities).

Recording: https://www.youtube.com/watch?v=nJxKODRJ_sI

## Transcript

Automatic transcript of the recording: names and terms can be misheard, and the times can be a few seconds off. Each time links to that moment of the video. The transcript alone: https://innovate-us.org/prediction-isnt-intelligence-how-predictive-models-really-work-in-government.txt (plain text) and https://innovate-us.org/prediction-isnt-intelligence-how-predictive-models-really-work-in-government.json (every segment with its start and end second).

**Arvind Narayanan**

[0:00](https://youtu.be/nJxKODRJ_sI?t=0) I know the bios are in the chat, but very briefly, she is the Chief Data Officer for the State of Hawaii, and previously she was the Chief Data Officer for New York's Office of Information Technology Services. She has 24 years of experience in digital data analytics, etc., in the private sector, and we're delighted to do this together. The title here is Prediction Isn't Intelligence How Predictive Models Really Work in Government. As a professor, I don't know how predictive models really work in government. I haven't worked in government. That's why Rebecca is here. She's going to tell us all about that. But what I will start with is what my role has been over the last few years, is studying AI failures, and that is perhaps what my research is most known for. And I have been compiling dozens of examples over the last few years of AI failures, especially in the public sector, that I will primarily draw from in presenting 10 kind of rapid-fire case studies, just spending two minutes on each of them, and what lessons we can take in terms of how to avoid pitfalls, how to deploy AI better in the public sector.

[1:12](https://youtu.be/nJxKODRJ_sI?t=72) Rebecca will talk about how to create positive impact based on her experiences, as well as data governance, and maybe we'll have a little bit of a discussion, the two of us, based on the points we each made. We can just play that by ear. It depends on how many questions there are, but I'm really looking forward to all of your questions, and then we'll do Q&A at the end. Rebecca, is there anything you want to add at this time, or should I jump in? Okay. Great. Please jump in.

**Rebecca Cai**

[1:37](https://youtu.be/nJxKODRJ_sI?t=97) Thank you, Arun.

**Arvind Narayanan**

[1:37](https://youtu.be/nJxKODRJ_sI?t=97) Thank you, Rebecca. So let's look at two examples of public sector AI deployments going wrong in some way. The first one is from Toronto. They wanted to know if beaches are going to be safe to swim in. They used a machine learning model which looked at various aspects of the weather and other factors in order to predict whether the following day, the bacteria levels in the beaches should be too high, and so whether the beach should be closed or not. And when they deployed the system, it turns out that it was very error-prone. The accuracy wasn't much better than random, and in fact, it was worse than the previous method of simply checking the water quality each morning. And this was a bit of an embarrassing failure for the city of Toronto. Let me give you one more example. This one is from New York. The city deployed an AI chatbot to help small business owners understand various aspects of, I think, dealing with regulations.

[2:42](https://youtu.be/nJxKODRJ_sI?t=162) But as you might know, these chatbots can hallucinate, and in many cases, it gave them wrong advice that would have constituted breaking the law. And this was also an embarrassing incident. Now, we can look at failures like this and try to look for lessons, what went wrong, how can we do things better? But the critical point I want to make looking at these two is that these are two very different The term artificial intelligence is an umbrella term for a set of loosely related technologies, kinds of AI. and we have to talk about them somewhat separately so that we can be clear about what the potential and what the risks and limits and what the mitigations are. So the Toronto example that I showed, that's predictive AI. It's making decisions, whether to open the beach or not, based on predictions about the future, what the bacteria levels are going to be. And the chatbot example is generative AI, which is the more familiar kind of AI that is in the news all the time. I don't need to tell you much about this.

[3:42](https://youtu.be/nJxKODRJ_sI?t=222) And there are a few other types of AI as well. But let's focus on the difference between predictive and generative AI for the moment. So I want to claim that these two types of AI vary greatly in all the ways that matter. When it comes to these predictions, it turns out that when you look at the cutting edge research on this, and this is what a lot of my research has been about, it turns out it's hard to do better than decades or a century old statistical models. Simple things like regression that many of us may have learned a long time ago in statistics classes, that's still mostly the technology that's used. As opposed to generative AI, which is based on foundation models, large language models, these massively powerful computation infrastructures, technology that's getting rapidly better by the day. And certainly at the level of the tech, they're quite different. What they're used for is also often very different. Generative AI is used for generating text that can answer questions for people, help them find stuff, improve their creativity, give them ideas, et cetera.

[4:43](https://youtu.be/nJxKODRJ_sI?t=283) Whereas predictive AI is often used for decision making. Should we open or close the beach? It's used in criminal justice. Is this defendant too risky? And so it might involve decisions that affect that person's freedom or even their life in healthcare settings. And thirdly, there are inherent limits to accuracy. So yes, it's true that chatbots generative AI can hallucinate, but the accuracy level is getting much better. There isn't actually an inherent reason why chatbots can't accurately answer questions. The way that they're built today based on these statistical models, yes, there are limitations, but things are getting better. As opposed to trying to predict what the bacteria levels are going to be tomorrow. We just don't know. The future has not been determined yet. This is a very key point. Is this defendant going to commit a crime or not in the criminal justice system? We don't know. It doesn't matter how much data you throw at the problem. So predictive AI has inherent limits to accuracy, and this is a critical point that's often underappreciated and I will come back to.

[5:43](https://youtu.be/nJxKODRJ_sI?t=343) And in terms of how they fail, again, there are important differences. When a decision making system fails, unless there is due process, unless there is an opportunity to appeal the decision of the system, you might be messing up someone's life. Whereas when a chatbot fails, there is still a human in the loop who ideally is in a position to take responsibility to review the output. Yes, sometimes they are careless and that's an issue, but I think fundamentally there's a difference in how these two types of AI systems are deployed. So for that reason, we want to focus on predictive AI today. It's a particularly consequential type of AI. It can be useful. It is deployed across the board in so many different kinds of systems, but there are inherent limits to accuracy. And when these systems fail, the consequences can be severe. So with that preamble, let me do a rapid fire of these 10 case studies, starting with the Toronto one. So what went wrong here? A few things. One of the things that went wrong, and we did an analysis of this in our newsletter, is this is a common pattern where the vendor involved in this case, the way they pitch their product is often based on, oh, you know, why have someone who is doing this time-consuming work, why not automate the process?

[7:01](https://youtu.be/nJxKODRJ_sI?t=421) And so the client, in this case, the government, you know, believes that claim, deploys it without human oversight. You know, it wasn't clear if anybody was reviewing the output of this machine learning model that was predicting the bacteria levels. But then when things go wrong, when there is bad press or some other negative consequence, the vendor points out in the fine print that the tool should never have been used on its own. But if there's always going to be a human in the loop, it's not even clear that it's saving you time or on what basis the human would second guess the statistical model's prediction of a certain bacteria level the next day. It's not like the person is going to mentally calculate it and then double check the output So it's a bait and switch. of the tool. We've observed this over and over again. Do you want the efficiency benefits or do you want proper accountability? It seems like it's hard to have both in many cases. Let me go to another example. This one is from a university in Maryland. They were using an AI tool to predict which kids might struggle.

[8:01](https://youtu.be/nJxKODRJ_sI?t=481) Seems like a really good use of AI. You can devote extra resources to helping those kids do well in school. And that's how the tool was supposed to be used. As far as I understand, the tool is called EAB Navigate. However, in a faculty meeting, the president dropped a bombshell. He said the way that he wanted to use the tool is to identify struggling students and convince them to leave the university. Why? It turns out because if they left early enough, instead of struggling and then dropping out, they wouldn't be considered part of the enrolled student body to begin with. And that will actually boost their retention numbers. And so the university would look better in the statistics. And of course, the faculty protested, this is not a responsible use of the tool. And I'm quoting the president's words. You think of the students as cuddly bunnies, but you can't. You just have to put a Glock to their heads. This might be an extreme example, but this does represent a pretty common pattern, which is that a tool might be built with the best of intentions, and the vendor might have done nothing wrong.

[9:05](https://youtu.be/nJxKODRJ_sI?t=545) But even a tool that is built for good purposes, like helping struggling students, can be used in a way that harms that very population. And so ultimately, the responsibility lies with those who are deploying the tool. So this example was from the private sector. But when we come to the public sector, I think the responsibilities are heightened even further. So here's an example in this political article. This was from a decade ago in the Netherlands. This was a big scandal. They used a machine learning model to identify suspected, I think, tax fraud or welfare fraud, And this model made a lot of errors. something like that. It was also biased. People who were from certain nationalities were assigned a higher risk score purely on the basis of being from that nationality, even if everything else was equal. And I think thousands of people were falsely accused of fraud, and it led to great mental anguish and financial penalties and other consequences for citizens.

[10:09](https://youtu.be/nJxKODRJ_sI?t=609) This became such a scandal that I think the president or the prime minister and the entire cabinet resigned. This was a really high-profile and embarrassing failure. And there have been other similar examples from many other parts of the world. There was one in Michigan, one in Australia. The British post office scandal, where postmasters were falsely accused of fraud. All of these are fraud-related examples. The British one became such a big thing that there is even, as I understand it, a TV show about this particular long-running miscarriage of justice. So I think all of these, the key diagnosis is that there was no due process. There was no way for people to appeal the decision. There was no way to ensure the right data quality that was going into these fraud predictions. And the responsibilities in public sector implementations are much higher than in the private sector. Okay, I'm going to keep a quick pace here, but always happy to come back to these examples and discuss them further.

[11:15](https://youtu.be/nJxKODRJ_sI?t=675) Let me go to another one. This is something that was in the news a lot, still continues to be in the news a little bit, which is false arrests due to facial recognition across the country. And there is a fantastic series of investigations by Cash Hill at the New York Times. She also wrote a book on this, Your Face Belongs to Us, looking at Clearview AI. And the lesson that a lot of people took away from this is that facial recognition doesn't work. The error rate is too high, and it's also biased. And this affects black people in particular. We should stop using this. So the ACLU, for example, has filed lawsuits calling for facial recognition used by police to be stopped on this basis. And when we started reading these stories, my co-authors, Ayush Kapoor and I, we were a little surprised because from our computer science perspective, it is true that 10, 15 years ago, the accuracy of facial recognition was really, really bad.

[12:15](https://youtu.be/nJxKODRJ_sI?t=735) But since then, the accuracy has been improving dramatically. So we wanted to understand what actually went wrong here. And I want to point out that facial recognition is not an example of prediction. This is not a tool that is predicting who might or might not commit a crime in the future. It is predicting, is the person in the surveillance footage, you know, this person or that person That's not an example of prediction. from this database? And there's no reason to expect that it's going to be inherently failure-prone. So are these tools, in fact, failure-prone, or did something else go wrong? And when we looked into it, at the time we did this analysis, we wrote this up in our book, AI Snake Oil, and we knew that this was an unpopular opinion. We were going to get hate mail for it, and we did get hate mail for it. But I stand by what we wrote. When we looked at the six cases that were known at that time of people being falsely arrested, they were all black people. So there is clearly a racial disparity here. But every one of those six involved basic policing errors. So here's an example where the facial recognition tool was only one part of the investigative process.

[13:19](https://youtu.be/nJxKODRJ_sI?t=799) Ultimately, he was arrested based on the eyewitness testimony of a contractor who was not even present at the time of the theft. And this is a person who was arrested in the wrong state. This is someone who was arrested based on feeding a 2015 photo to the facial recognition tool, even though a much more updated and accurate photo was available. And what we concluded is that this whole narrative about facial recognition being irredeemably error-prone and therefore should be discarded, it's maybe targeting the wrong thing here. Maybe what's happening is that people are blaming tech for what are ultimately errors in policing. Errors in policing are ultimately very common. They happen on a daily basis. And even if the use of facial recognition were stopped, those errors will unfortunately continue. And so the focus here is a little bit misguided. And we give an example in the book of someone, this is a case where facial recognition was not even involved, someone who was arrested just by having a similar name to a suspect and faced major consequences.

[14:25](https://youtu.be/nJxKODRJ_sI?t=865) So I'll have other examples as well where this comes up. When we talk about failures of the tech, we have to be careful about, is it actually the tech that was wrong? Can we actually fix this by getting the tech vendors to do better? Or is it a process failure? And to her credit, the aforementioned Cash Hill, who has been doing these investigations, has been pretty clear in all of her stories at putting the focus on the process failures and the systemic failures in policing. And despite that, the narrative that has developed around her investigations is as if it's a And I think that is quite unfortunate. tech failure. Okay. I'm about. Halfway or a little more than halfway through. The next example is from the UK. The National Health System developed an algorithm for matching organs, specifically livers, to recipients. So when someone dies, their liver is available. There's a long waiting list of people. Who should that liver go to? And these types of decisions used to be done by surgeons. Now they tend to be done in most countries for most organs by these statistical algorithms.

[15:30](https://youtu.be/nJxKODRJ_sI?t=930) You can call it predictive AI. People have different opinions on whether the term AI is justified here or not. And there was an unusual source, the Financial Times. But they did a really good investigation based on complaints from patients that this algorithm was extremely biased in terms of age. And younger patients had very little chance of qualifying for a liver. And we looked into this. We read all the papers behind this. And we did a little bit of an analysis ourselves. The citation is at the bottom. And what we realized that, yes, there is, in fact, a severe age bias here. And we're familiar with how machine learning can be biased because it's trained on human data. That human data contains existing societal biases, such as racial or gender biases. That story is well known. But what happened here, and in some other cases I've looked at, is much more subtle. The age bias that happened here was not picked up based on training, based on the past decisions of doctors.

[16:31](https://youtu.be/nJxKODRJ_sI?t=991) This was a new bias that was introduced in the algorithm because of a pretty subtle way in which their data collection went wrong. So that was the first surprising thing. There can be lots of very subtle biases based on how the technical development of the data sets and the algorithm happens. That was the first thing. And a second point worthy of note is that this patient here, Meredith, who is the centerpiece of the story, nobody would tell her how the algorithm worked. The doctors, they themselves didn't know. And when she approached the National Health Service, she got condescending responses. This made everything much worse. The lack of transparency around the system and the assumption from the system that patients are incapable of understanding something so complicated. It's not that complicated. Patients can understand it. That was another unfortunate thing. And the third thing is that before the algorithm was introduced, the decision used to be made in a holistic way. Surgeons might consider, is this a deserving patient? Is there a liver failure on account of their own behavior, such as alcohol abuse, or is it something they were born with?

[17:36](https://youtu.be/nJxKODRJ_sI?t=1056) Things like that. Geographic considerations. Does the donor who died want to help someone within their own community? But I'm not necessarily defending those criteria. That sort of thing. What I'm pointing out is that all of those considerations are thrown away when you move to this algorithm. It's kind of a cold calculation of number of life years saved by picking a particular recipient over another recipient. And people are not aware that the entire moral framework behind this kind of medical decision making has been completely switched to a utilitarian approach because of the large-scale switch to algorithms. And I think there is a level of transparency, not necessarily about how a specific algorithm works. That's important too. But these large-scale structural changes that are taking place and how we make life or death decisions about citizens' lives that are not really communicated to people and don't necessarily have public buy-in. And I think that's a bit of an issue. Okay. Next example. This is from criminal justice. This is a risk prediction tool.

[18:38](https://youtu.be/nJxKODRJ_sI?t=1118) This is the Public Safety Assessment. I believe it's funded by Arnold Ventures. They learned the lessons from ProPublica and Compass and some of those investigations you might have seen from a decade ago where things went wrong in terms of racial bias. They did a lot of things right here. But nonetheless, as this particular investigative piece shows, there are some issues. They looked at one particular area, Cook County, Illinois. And this was built on data from all over the nation. However, Cook County is particularly safe. The base rate of crime is really low, which means that this nationwide tool is not tailored to the peculiarities of the particular regions where it's being deployed. And so they are arguably detaining 10 times more defendants than ideally should be detained based on the base rate of crime in Cook County. And what this is an example of is what worked there might not work here. This is one of the core insights in policy studies that I teach my students when I'm teaching in the policy school. In traditional policy analysis, policy evaluation, we know that you can't just take some general insight, oh, it worked in the Netherlands or whatever country, and then do the same approach here and expect it to work.

[19:50](https://youtu.be/nJxKODRJ_sI?t=1190) However, when it comes to these AI tools, the assumption is that you can throw out those questions of how to particularize the tool to your specific population, to your specific jurisdiction. It's a one size fits all tool. And the whole power of it is supposed to be that you take data from everywhere, you put it into this one system, you train it once, you can go deploy it everywhere. This is not a magic bullet that allows us to escape the pain of having to particularize our policies to specific geographies and populations and jurisdictions. We still need to do that hard work. So that's another common pitfall with AI and machine learning. Okay, so the next quick case study, this is a classic example from the computer science literature. Medical AI researchers built a model for predicting the outcome of patients who were hospitalized with pneumonia. Are they likely to have complications from it? And what the model predicted that asthmatic patients who had asthma are less likely to have complications from pneumonia.

[20:53](https://youtu.be/nJxKODRJ_sI?t=1253) And they went, huh? And they looked into the data and shockingly, this was a true pattern in the data. It turns out this is because when someone presents with asthma, they get triaged and the hospital treats it as a high priority because they have pneumonia and they're sent to the ICU and the care there is so good that their outcomes are actually better than patients who have asthma. And ironically, the system that they had built was meant to replace the existing triage policy. So it would have been a disaster if the model had learned this pattern and therefore decided to deprioritize patients with asthma. And this is just an example of how evaluating these models is really hard. The current practice of having some benchmarks and automatically producing a performance number against those benchmarks, just doesn't work at surfacing these kinds of nuanced failure modes for which you really need domain expertise And that is not yet a standard part and critical thinking.

[21:53](https://youtu.be/nJxKODRJ_sI?t=1313) of many data science pipelines that I have seen. So that's a big issue that I wanted to flag. Okay, the next quick one, this is from predictive policing. I won't spend too much time on this because this is based on an article by my colleague Mihir Shirasagar who I understand is doing the next of the trainings in this series. So maybe he's gonna talk about, I don't know specifically what he's gonna talk about, but he points out why predictive policing has often gone wrong. And he makes a couple of very important points. One is building on the point that I just made, evaluation is hard. One of the ways in which evaluation can go wrong in predictive policing is self-reinforcing feedback loops. It might look like the system is working really well because the data you've collected is based on where you have decided to deploy police based on your algorithm. And that's the first point. The second point is that there are good ways to apply data science and analytics in policing which is to understand patterns of crime and then have human decision-making around what we're gonna do about this instead of having a completely automated feedback loop where the algorithm automatically decides what is the best response to the observed patterns of crime.

[23:04](https://youtu.be/nJxKODRJ_sI?t=1384) I think that decision-making still requires judgment and should be in the hands of humans rather than the algorithm. So the next one, this is a well-known example. Maybe you've seen this picture before. This is from hiring automation. There are many AI tools out there that have the pitch that just have your job candidates upload a video where they're talking about just their hobbies or whatever. And the AI is gonna use facial expressions, body language, things like that to score their personality traits and based on that predict their job performance. It's kind of a crazy claim. There have not been a lot of independent evaluations of these tools because the vendors are not very transparent. I'm on record calling these elaborate random number generators. I don't think they work at all. But in this particular case, investigative journalists managed to do something clever. They have a video of a person talking and then they upload the same video but they've digitally inserted a bookshelf in the background and it produces very different scores.

[24:05](https://youtu.be/nJxKODRJ_sI?t=1445) And you can kind of see why the model has presumably picked up this pattern. It could be that in its training data, candidates who have bookshelves in the background, there's another one, candidate with glasses, without glasses, those might be correlated with performance but it doesn't mean that if you take the same candidate and modify their background, that it's gonna change their future performance. So it's really affected by these superficial features. And so candidates are often aware of this and they can easily game these systems. So that's another possible failure mode. Okay, the last one and then I'll wrap up with 10 different takeaways from all these. Really look forward to hearing from Rebecca. This was a New York Times investigation of a tool in Spain that is used for predicting domestic violence. So what happens is when a woman goes to the police and says she feels threatened by her partner, the police ask her a lot of questions, put the data into the system, and it spits out a risk score. If the person is considered to be high risk, then they'll have police protection and then there's like three or four different levels and they'll decide what to do based on the risk score.

[25:11](https://youtu.be/nJxKODRJ_sI?t=1511) And they have stories like this one. It's a very heartbreaking story. This person was scored as negligible risk but then her partner did in fact murder her. And there have been a few notable misses of this tool. what the risks are and what the risks are not. So I really went into this to try to understand To try to understand what went wrong. I read the technical evaluation and documentation behind the tool as well as the processes behind how the police were using this. And I was forced to conclude, I don't think the Spanish police really did anything wrong here. They're in a tough situation. They can't have police protection. It's a finite resource, you know, policing time. And so there are more people who want this protection than are available. So you have to have some way of sorting them. And this is a statistical tool. It's been validated. It does a decent job. Of course, it's impossible to perfectly predict the future. There are going to be false negatives. That is the tools saying that it's low risk but then in fact a crime happens. You can't get every case of it.

[26:12](https://youtu.be/nJxKODRJ_sI?t=1572) And ultimately I thought that the New York Times reporting here did a disservice to the, you know, to the victims here as well as the police who are trying to do the best job that they can. And it's making it seem as if every false negative here means that the tool is somehow not working as opposed to looking at what is the next best possible alternative? You know, what are the processes around the tool to ensure accuracy, et cetera. So my takeaway from this is with a predictive tool, even if you do everything right, there are going to be failures and that's something you have to be willing to accept. And I think it's better to proactively communicate with key stakeholders and the public about the strengths and limitations of these tools so that there are not misaligned expectations and articles like this one. Okay, so let me just briefly put up these 10 case studies here. I told you about a bait and switch by AI vendors in the case of the Toronto tool. I told you about a case where a university was using a tool that was meant to help students to instead kick them out.

[27:16](https://youtu.be/nJxKODRJ_sI?t=1636) I told you about why due process is so important especially in cases like welfare fraud detection. I discussed a little bit why it's important not to blame the tech when there's a failure of the underlying human and systemic institutional processes like in the case of policing and facial recognition errors, a little bit of bias and transparency issues in the liver allocation algorithm and examples from criminal justice for medicine, predictive policing, et cetera. So I will stop here. And then Rebecca, do you want me to put your slide up on the screen? Great, I think you're still muted, but yeah, I really look forward to hearing from you. Thank you everybody.

**Rebecca Cai**

[27:59](https://youtu.be/nJxKODRJ_sI?t=1679) Thank you, thank you, Arvind. Yeah, thank you Arvind for sharing all that. I have sweated a little bit when I heard all those stories. Yeah, so I think Arvind touched several things. Things could go wrong on the data you get on the model itself and how you use it, the process itself. So for government from Hawaii, I'm wondering how could we use it? Who would benefit from it? One of the questions from the Q&A is, how can we talk about predictive AI from the lens of benefits? So this is a kind of a very high level diagram on the flow, the process flow. We need the data. We need the data for the predictive models. But we all know the reality of the government data. They are collected for compliance, not for analytics. And they are collected in different data silos, different departments, different programs.

[29:01](https://youtu.be/nJxKODRJ_sI?t=1741) They have their own data there. And might not be consistent across the board, might not be shared. And often it takes, there are some delays on the data we get. There's inconsistency across them. And typically we get them through the compliance reasons, for historical reasons. And sometimes they're aggregated at a level that you can't do a detailed level prediction. So just we need to realize there's no perfect data when we do the predictive model. Like Arvind mentioned, instead of just blaming on the tool itself, we need to think about the data that's feeding into the tool, whether it's complete, it's accurate, it has represented everyone. One story I used before is when the AI first came out, I used Gen.AI to generate a picture of me and my two sons, and they said Asian lady, a Chinese American lady in Hawaii with two sons, and they are both all very tall.

[30:17](https://youtu.be/nJxKODRJ_sI?t=1817) So the picture were very tall and skinny, and I'm like, I'm six foot tall, almost six foot 5'11", and my boys are muscular, I'm strong as well. So I said, make it strong. Make them muscular. Well, as soon as they make it muscular, they don't look like me anymore. So the data that feed into the model is not the model creating the bias. It's the data, just not enough data of that. And in the middle, once we feed the data into the predictive model, we need to make sure it's explainable, it's stable, and it's auditable. We don't want to use any mysteries model without knowing how it works and what it does in there. So I don't know how much I can trust it. Eventually, it's up to the human to make decision, right? It's up to the human to take action. So we have to trust it. We have to make sure it's auditable. From the model, what can we use it for?

[31:18](https://youtu.be/nJxKODRJ_sI?t=1878) We can use it for, for example, what benefits would Rebecca qualify for? What federal and state and local benefit program would Rebecca qualify for? And if there is a requirement for, for example, for the HR, new HR1, right, for the working hours, or Rebecca worked for only two hours, this is the last week of the month, Rebecca is at very high risk, and she's going to lose her benefit because she didn't work enough hours for the first three weeks. So let's send Rebecca a reminder, so Rebecca can take actions. Let's even recommend Rebecca certain job opportunities or volunteer opportunities based on her home location, so she can easily sign up and, you know, work the required hours. And also, who has the health? Health is important now, always, right? Especially nowadays.

[32:18](https://youtu.be/nJxKODRJ_sI?t=1938) So who has the risk of falling? Who has a high risk of getting diabetes? So if Rebecca's receiving food from the government or from the food banks, but she is, she has high blood pressure, has diabetes, should we inform the food bank to provide certain And what's the risk of, what would be the risk of tsunami in Hawaii, right, prediction? foods for her? And what would be the impact if there is a certain level of tsunami in Hawaii? And which houses would be impacted? How many people are in there? And what would it be? Where should we put them? How should we host the shelter for it based on the population?

[33:18](https://youtu.be/nJxKODRJ_sI?t=1998) That's one of the questions another person asked about the digital twin, right? Digital twin is great for simulating. UH, University of Hawaii, build a model where you can simulate when the sea level rise, the close to the beach area, the septic tank, not the septic, the drainage system would have the water back up. So which houses would be impacted with that? Yeah. That's our prediction. And if the government want to make a decision on reducing tax or increase tax, who would Is it a high income people, low income people, where do they live? be impacted? So would that mean that we have more people qualified for benefits if we increase tax? So what would be the, all the different impacts? So that's all the prediction simulation you can do, but it's probability. It's never a true hundred percent for sure.

[34:23](https://youtu.be/nJxKODRJ_sI?t=2063) So we human, when we make a decision, we need to understand it. Nobody can guarantee things could change. That could be if data is not real time, if there's a new situation change, like a COVID So it changed a lot of stories. happened. So eventually on the right hand side, it's the human. We human need to review like the policing. Police officer need to review. This is a picture. This is the suspect we're thinking that we're arresting. Do I think they are the same? Oh, you know, AI makes mistake. You know, actually I think they look the same as well. Some people look the same. My sister and I look very similar. So people might mistaken it. So mistake happens and human makes error. That's another question. How does AI error compare to the human errors? I don't know. I make a lot of error myself, make a lot of human error, but Arvind might not. So humans are different. AI models are different as well.

[35:24](https://youtu.be/nJxKODRJ_sI?t=2124) So we just have to make sure we have the governance in place, control the data. Do we know all the data where they are? Are we able to security share the data? So we can use it for predictive analysis and do we know the quality of the data? Can we improve the quality of the data? All that information and do we have the budget to improve the quality of the data? And with a model, is there any bias built in? For example, saying that any women over 50 should not be employed. You could physically put in a model, but that's not right. That's the transparency of it. And do you have the privacy protection if I use Rebecca's data to predict whether she is at risk of falling or at risk of getting certain disease? Do we have Rebecca's consent?

[36:26](https://youtu.be/nJxKODRJ_sI?t=2186) Do we protect her data? And also we need to watch out for the data pollution. There are bad players out there. They would pollute your data intentionally to create a bias. They would have polluted with all like 90-year-old data. Then the younger people would not get an accurate answer or they could do vice versa. So in the middle, we need to make sure we govern that. On the right-hand side, when we make the decision, we want to provide the feedback. There's got to be a feedback loop back into there. We can provide the feedback on the accuracy, on what the issue might be, any bias. Like a New York City example, the second example Arvind used. It works fine now because the feedback, they improved. And also the transparency, when we use the model, normally we want to inform people.

[37:27](https://youtu.be/nJxKODRJ_sI?t=2247) I'm using AI model, but I reviewed it. I made a decision, conscious decision as the human being. So that's pretty much it on my side. So governance, understand what data you have. I hear a lot of people saying that when there's a business problem arise, either there's a bird flu or anything, how do we know who has the data I need? And how do I easily get access without spending months to sign a data sharing agreement? That's the data governance platform. If the state or the federal, whichever level of government it is, have the data governance platform, have the access control predefined, then you can easily see what data is available, who has what data and what access control is there so I can get it, so I can use for my predictive model.

[38:29](https://youtu.be/nJxKODRJ_sI?t=2309) In the model itself, I need to figure out who is using it. Is Rebecca secretly using it, without letting anyone know? And she just let it make decisions for her. We need to detect all that, govern it. And eventually nobody can blame on the machine. No matter what tool, I have cell phone, I have computer, all those days, right? And I can't blame, it's my computer making that mistake. It's not. It's eventually it's me. The responsibility is on the humans, because we're augmenting the process. We're not automating, replacing humans here at the government. All right, thank you. That's all the sharing I have. Thank you so much. Any other questions?

**Arvind Narayanan**

[39:15](https://youtu.be/nJxKODRJ_sI?t=2355) Thank you, Rebecca. Can I take one minute to very quickly answer two of the questions here in the Q&A? And I know there are some for you as well. So one question is what differentiates age old predictive modeling from predictive AI? Not a lot. But in my mind, one important distinction is that traditional predictive modeling was rarely used for automated decision making. You would have someone like a statistician who was in the loop, they would look at the data and then, you know, you'd kind of have a meeting to decide what to do based on that. A new thing with predictive AI is that you can automatically make decisions with no human in the loop. And that's where a lot of the due process and accountability issues come into play. And then a question is how can an AI amateur police a particular use of predictive AI? Is there a checklist or other process with specific steps? Yes, there are some checklists. We have released one ourselves. We released this accompanying a paper called Against Predictive Optimization.

[40:19](https://youtu.be/nJxKODRJ_sI?t=2419) I know the title is a mouthful, but if you search for that, you will find our paper. And no need to look at the paper, but accompanying that is a 27-question checklist that goes through a comprehensive list of things one might want to check with the development and deployment of a predictive AI system.

**Agueda Quiroga**

[40:42](https://youtu.be/nJxKODRJ_sI?t=2442) Okay, thank you so much. I will put the link to the article, Arvind, right now in the chat, but there are also I don't know if you prefer me to read them so you can decide which is going to answer. other questions. Anyone? Any preference? So I think there's a very interesting question by Natalie and she's asking, could you explain the difference between predictive modeling decades or statistical methodology from predictive AI? Is there a meaningful difference? It's a very technical question.

**Arvind Narayanan**

[41:21](https://youtu.be/nJxKODRJ_sI?t=2481) Sure. Just to quickly repeat my answer, the main difference in my view is whether or not we're trying to take the human out of the loop or have the human have a minimal role, such as reviewing the AI decision recommendation, versus in traditional predictive modeling, you use statistics, you look at the data, and you have a discussion with a set of decision makers who decide what to do.

**Rebecca Cai**

[41:49](https://youtu.be/nJxKODRJ_sI?t=2509) I'll just add on to that. One example in my daily life is master data management. Like all the days 20 years ago, master data management, the matching is using the modeling to try to match the data. It takes a lot of time and that's a lengthy process. But it takes time. But right now, when gen AI is available, they use the gen AI models to do the matching of the master data. It cut down the time significantly. Yeah. And you don't have to have so many programmers or statisticians to custom build it and maintain it.

**Arvind Narayanan**

[42:40](https://youtu.be/nJxKODRJ_sI?t=2560) Can I answer a question? Yeah. There's a couple of related ones about AI errors versus human errors. So I think the, how do the error rates compare? It really depends on the particular application. In a lot of these predictive AI applications, generally what the research says is, if you're trying to predict, let's say, where crime might occur or one of these things we've talked about, generally speaking, there's not that much, you can't really be that accurate, regardless of whether it's a human or a machine learning system doing the prediction. And the machine learning systems tend to be maybe a few percent more accurate than the human. But it really depends on the specifics. It gets more interesting when you're talking about generative AI with applications such as using AI to create software or other harder tasks.

[43:48](https://youtu.be/nJxKODRJ_sI?t=2628) And there is an interesting complementarity between AI errors and human errors. So what AI is often able to do is avoid a lot of the errors that people might make due to carelessness or tiredness or things like that. But since AI still in many cases lacks common sense, it might do something that a person would never do. For instance, I lead a research team evaluating the performance of AI agents on helping people with tasks like shopping. A lot of the time they can navigate websites, add stuff to the cart and buy things for you. But sometimes they will do things like use your business credit card for a personal transaction. That's a common sense thing. We kind of automatically know to avoid things like that. But unless the AI model maybe has an explicit reminder not to do that, it might make that kind of common sense error. So we're seeing that the errors, types of errors made by people in AI in many applications tend to be very different, which is actually good news because it means that when people are reviewing AI outputs, they can...

[45:01](https://youtu.be/nJxKODRJ_sI?t=2701) They can catch a lot of the errors since it's not the kind of error that a person would make.

**Rebecca Cai**

[45:07](https://youtu.be/nJxKODRJ_sI?t=2707) Yeah, I would like to answer a question on how governments manage data governance in predictive AI for indigenous people data. So every single person's data, the privacy is a key thing. We need to make sure the personal data is protected. And normally we use the data aggregated at high level up so it wouldn't expose any privacy of any individual citizen. But in the example I gave, if you want to predict like a give me warning, I want government to give me a warning on whenever I am having a risk of getting a disease or having a risk of losing a benefit or whenever I could possibly qualify for a benefit, I need to give the government consent in order to do that. And with indigenous people, I think one potential risk is do we have enough representation in the data pool? For example, if the school has only one person, even when that aggregated data up at the risk level, that person's GPA actually first, you disclose that person's private information because everyone knows there's only one student there. Second, it's not representative, representing the entire risk because you only have one sample size. So those are the things we need to pay attention to, making sure the data is complete. Another question is how an AI amateur can police the predictive AI usage. Lawmakers, they don't have to be a technical expert to set the laws.

[47:01](https://youtu.be/nJxKODRJ_sI?t=2821) You don't have to be a technical expert. I'm not a technical expert. I'm not at the same level as Arvind. So it doesn't impact me performing my job. So we need to make sure that what we watch out for, the different potential 10 things like Arvind mentioned, we need to watch out for. And we need to make sure that a predictive AI model is explainable, it's auditable, and it has the transparency. And when we use it, we want to make sure that the data feed into this model. We can say that the vendor can tell us the predictive model is 99% accurate, just like self-driving cars, like 99% accurate. But what if that 1% thing happen? So that could be fatal, right? So we need to understand that consequence. We need to know the data fit in into that model. I know, categorize it, classify those data and understand the quality of it. If I know the data it use, it's just not clean. I know even though the model is 99% accurate, I cannot use it. It's a human judgment. Yeah.

**Arvind Narayanan**

[48:11](https://youtu.be/nJxKODRJ_sI?t=2891) Thank you, Rebecca. Maybe I'll take a turn and do a couple of questions. One is, how do you guard against gaming opportunities, people who might game opportunities for public comment without chilling citizen engagement? I have maybe a little bit of a depressing answer to this. I think the traditional model of public comment, where anonymous people anonymous people can add comments and then the government agency is required by law to read and perhaps even respond to all of those comments, that model is dead. You can very easily have AI systems that write thousands or any number of seemingly legitimate public comments. And you can't really detect AI written text, especially for short texts. It doesn't really work. So I think it's necessary to move to a slightly different model. I think one obvious way is for people, for comments to be required to be linked to real people's identities. Of course, there are concerns around privacy and anonymity and surveillance that come up with that. One interesting out of the box idea that some of my colleagues here at Princeton have been experimenting with, Varun Rao is someone whose name I should mention is, in addition to, or instead of the formal public comment process, why not look at the comments that people are already sharing with each other on Reddit or social media or whatever it is? Because people are having everyday conversations about a lot of the issues on which government is seeking to solicit public comment. So they've come up with actually a language model based pipeline with a human being fully in control to take a set of thousands of comments and then extract the key themes from them. So that can be a way to augment traditional processes. If I can quickly answer one more, the question was, there was a case study I presented of the data not matching the place

[50:13](https://youtu.be/nJxKODRJ_sI?t=3013) where the system was used. How can that be addressed? The end users don't have control over the training data. One word answer to this, pilots did not take the vendor's claims at face value because it was evaluated on a population that's different from your users. You have to test it on a subset of your users in a scenario that simulates as closely as possible the way in which the AI system is going to be deployed. There is really no other way. And while you might not have control over the training data, I think you can certainly ask for transparency over the training data, a description of where that data came from and what the population was. Thank you.

**Rebecca Cai**

[50:55](https://youtu.be/nJxKODRJ_sI?t=3055) I'll try to answer one question about what are the different type of use cases for AI? How can I think of when to use AI? That's beyond predictive modeling itself. Normally, you could have conversational AI. For example, it doesn't have to be chatbot. It could be automatic text or even voice calls to answer questions on text, business, how to get a building permit, things like that. And customer service uses it a lot or case managers to support case managers, a supported service. And also content generation where you can create memos, create reports, create any communication materials. There is a natural language processing where you would use it, for example, on invoice verification to look at the invoice content on that invoice to figure out whether the signature is right, whether getting data from it and do they have complete data, did we miss certain fields. And computer vision is our department of transportation here use it to use the pictures to detect the potholes on the roads. And the AI-enabled process augmentation, that's a big now. Any process, actually a lot of them can be augmented using AI, like instead of manually compare signatures, manually review the documents and getting the data from here, getting data from there, and then I prepare the information.

[52:36](https://youtu.be/nJxKODRJ_sI?t=3156) You can automate the process. And then the predictive modeling AI platform where you can predict high crime area for better police resource allocation, all that.

**Arvind Narayanan**

[52:55](https://youtu.be/nJxKODRJ_sI?t=3175) I think we've answered most of the questions.

**Rebecca Cai**

[52:58](https://youtu.be/nJxKODRJ_sI?t=3178) Oh, the last question is interesting. When AI makes a mistake, who is responsible? Humans are responsible. We never replace humans. They just act as our assistant to help us. Can't blame AI.

**Agueda Quiroga**

[53:18](https://youtu.be/nJxKODRJ_sI?t=3198) Okay, Rebecca and Arvind, this has been an amazing, amazing session. I mean, incredible, thoughtful. Thank you so much for sharing your knowledge, your wisdom, and also for answering all those questions. You will find, all participants will find the next information on the next session into the chat. So we hope to see you soon for the next session in the series. And we also, all the information about our upcoming workshops. But again, thank you so much for this one. And we, every participant will have, I'm sorry, we will, you will find the recording to the session tomorrow on our website. So you can record and share. So thank you so much. Everybody has been a pleasure to have you here. I hope to see you very soon.

**Audience**

[54:11](https://youtu.be/nJxKODRJ_sI?t=3251) Since we have four minutes left, I'm going to use the last, I'm going to use 10 seconds to ask you each to spend two minutes, just giving your final advice, really focused on, we have, this is a room full of public servants. Can you give us your final wrap up advice on one thing they should not do and one thing they should do to take advantage of the power of prediction where possible?

**Arvind Narayanan**

[54:37](https://youtu.be/nJxKODRJ_sI?t=3277) That's such a great idea. And I just want to say on my part, thank you, everybody. Thank you, Rebecca. Thank you, Agueda, Eileen, Beth. And thank you for everyone for participating. This has been so much fun on my end. Yeah. Thank you. Same here. And any last advice though,

**Audience**

[54:56](https://youtu.be/nJxKODRJ_sI?t=3296) for people that there's something practical that you less than you want them to make sure they

**Rebecca Cai**

[54:59](https://youtu.be/nJxKODRJ_sI?t=3299) take away. Yeah. Governance, governance, governance, garbage in, garbage out, definitely work on the governance and also focus on the business impact of the business problem, not the technology itself. It's always having the understand the business problem, what impact you want to create, and then look for the tools and don't trust the vendor a hundred percent. You use it, try it, and then find the issues and always have this critical thinking, be a critical thinker, always question, is this trustable and can I use it? Always ask the right questions. Thank you.

**Arvind Narayanan**

[55:40](https://youtu.be/nJxKODRJ_sI?t=3340) Arvind, do you want 10 seconds? Sure. You have one and a half minutes. Yes. I will share something that's maybe a little bit different from everything we've talked about. Oftentimes, in my experience, the real innovation, whether it's predictive AI or generative AI, any kind of AI, is less than the technology itself and the workflows that we come up with in order to integrate that technology into our existing processes. And that innovation is something that the vendor cannot do for you. That's something that every government, every agency, every organization, perhaps every team even, has to do. And we should make sure to really value that and not try to outsource that part to the vendor.

**Audience**

[56:29](https://youtu.be/nJxKODRJ_sI?t=3389) Thank you both. Thank you, Agata and Eileen.

**Agueda Quiroga**

[56:33](https://youtu.be/nJxKODRJ_sI?t=3393) Thank you, everybody, and hope to see you soon for the next session. I am going to close the session now, if it's fine. Yes. Thank you so much.

**Arvind Narayanan**

[56:41](https://youtu.be/nJxKODRJ_sI?t=3401) Thank you. Thank you.

* * *

Original page: https://innovate-us.org/prediction-isnt-intelligence-how-predictive-models-really-work-in-government · Licence: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org
