---
id: "workshop:prediction-isnt-judgment-what-predictive-ai-canand-cannotdo"
type: "InnovateUS workshop"
title: "Prediction Isn't Judgment: What Predictive AI Can—and Cannot—Do"
url: "https://innovate-us.org/prediction-isnt-judgment-what-predictive-ai-canand-cannotdo"
date: "2026-10-20"
duration: "60 minutes"
language: "English"
series: "The Good, the Bad and the Ugly of Predictive AI"
instructors: ["Manoel Horta Ribeiro"]
moderators: ["Mihir Kshirsagar"]
recording: "https://www.youtube.com/watch?v=GoMKi0N1LLw"
transcript: "automatic, from the recording"
source: "InnovateUS (innovate-us.org)"
licence: "CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org"
---

# Prediction Isn't Judgment: What Predictive AI Can—and Cannot—Do

InnovateUS workshop · 20 October 2026 · 60 minutes

Predictive systems are increasingly used to forecast risks, identify priorities, allocate resources, and support public-sector decision-making. But prediction is not the same as judgment. This session explores how predictive AI works, what different kinds of predictive systems actually predict, and where prediction can improve government performance. Participants will learn how to distinguish between problems that are well-suited to predictive approaches and those that require other forms of analysis, expertise, or public deliberation.

## Speakers

- **Manoel Horta Ribeiro**, Assistant Professor, Princeton University

## Moderators

- **Mihir Kshirsagar**, Tech Policy Clinic Lead, Princeton University

Part of the series [The Good, the Bad and the Ugly of Predictive AI](https://innovate-us.org/workshop-series/the-good-the-bad-and-the-ugly-of-predictive-ai).

Recording: https://www.youtube.com/watch?v=GoMKi0N1LLw

## Transcript

Automatic transcript of the recording: names and terms can be misheard, and the times can be a few seconds off. Each time links to that moment of the video. The transcript alone: https://innovate-us.org/prediction-isnt-judgment-what-predictive-ai-canand-cannotdo.txt (plain text) and https://innovate-us.org/prediction-isnt-judgment-what-predictive-ai-canand-cannotdo.json (every segment with its start and end second).

**Mihir Kshirsagar**

[0:00](https://youtu.be/GoMKi0N1LLw?t=0) And thank you to Innovate US for organizing this, and thank you all for attending. So it's great to see all of you here. For those of you who have not come across Princeton's Center for Information Technology Policy before, CITP, we are a center that's housed between our policy school and our engineering school, and we're a group of researchers committed to working to better understand the relationship between digital technology and society. And the question that we're taking up today is actually the second round of predictive AI that we've done through Innovate US, and we have this amazing cast of speakers, Manuel, who I'll introduce in just a minute. But I think just to pause on it for a second, because all of you, I think, when you hear AI, you're all thinking generative AI, right? That's the latest, shiniest thing. But there are lots of AI systems that are already in place that make very consequential decisions that use old-fashioned predictive systems. And a lot of, actually, generative AI is duplicating some of that same work. So the pitfalls and the opportunities of using data responsibly and well is a big challenge that cuts across any use of AI. And so very, very grateful that all of you from your different backgrounds are here to bring questions to us. So that's the second part of this seminar sessions that we're having, is that we want it to be interactive. We want to hear your real-world problems and try to talk about what are the issues that are coming up in your lives. And so please use the chat extensively. Please use the questions extensively.

[1:55](https://youtu.be/GoMKi0N1LLw?t=115) We want to get to understanding what are the challenges you face? What are the bottlenecks you face? How can we help you understand what the issues are? So with that, we'll leave plenty of time for that. But I'm absolutely delighted to have my colleague, Manuel Horta-Ribeiro. He leads the Human and Machines Lab at Princeton. He seeks to understand and improve socio-technical systems in the age of AI. He's written an enormous number of award-winning papers and conducted cutting-edge research on a number of different topics. But one of the things you will discover as you listen to him is that he also is committed to making his research accessible and really useful to decision-makers like you. And so with that, I'll turn it over to

**Manoel Horta Ribeiro**

[2:48](https://youtu.be/GoMKi0N1LLw?t=168) Manuel. Thank you so much, Mihir, for the very nice intro. Can everyone hear me fine? Hopefully. I just started sharing my screen, so let me see if I can see the chat. Can everyone see the screen and hear me? Yes, everyone can. Okay, awesome. Sorry, just because when you share your screen, you lose track of everything. Thank you so much. So yes, thanks, Mihir, for the nice intro. My name is Manuel. Today, I'm going to be talking a bit about what predictive AI can and cannot do. And I'll try to draw a couple of examples from the scientific literature. As Mihir mentioned, while generative AI is kind of the shiny new thing, we've been struggling with the question of how to introduce artificial intelligence in society, in the public sector, in healthcare for a while now. And if anything, the advent of generative AI has sped up this process, because now generative AI allows you to create zero-shot predictions without having training data. I'll come back at this point a bit at the end of the talk. But the goals of this session are that I hope that by the end of the day, you can try to separate what is prediction and what is decision, and having clarity between how these two things differ. It's crucial for someone in a role that has the possibility to adopt these AI tools to detect when prediction is appropriate. And even when it is appropriate, what are the consequences of introducing predictive AI into your workplace, for example? Maybe to get things rolling, it's worth noting that predictive AI is already in the government. There's already a lot of places and institutions

[4:55](https://youtu.be/GoMKi0N1LLw?t=295) that are using predictive AI systems. For example, Chicago has this system that does forecasting to detect which restaurants are more likely to violate sanitary rules. This system, as many of the other ones that are listed here, they work in a similar way. They use some predictive AI system to assign a score to each unit. Here, the unit is a restaurant, but it could be a section of a forest, or whether someone will receive bail or not, will be released before the trial. This score is used to produce a ranked list, and this ranked list is used to determine in which order a series of actions are going to be enacted upon. So essentially, sometimes Chicago has a limited budget of the amount of inspections, it can make, and it can distribute these inspections across this ranked list. So that's the idea there. There's many cases in which these systems already exist, in healthcare, in fire prevention, in the criminal justice system, and so on. Maybe it's worth taking a step back and thinking about the anatomy of these predictive systems, especially when they're built with traditional AI systems. We can briefly talk about how this would be different.

[6:31](https://youtu.be/GoMKi0N1LLw?t=391) But within this blue box here, you have perhaps past data. Let's use the Chicago example of the food inspection as a running example. You have past data of what are different attributes for different restaurants, and which restaurants have been found to be in violation of food safety rules, for example. You use this past data to train a model. This model could be some sort of linear regression, it could be a random forest. There's a variety of models that try to use this training data containing past decisions or past inspections into a likelihood that any sort of restaurant will be in violation of the rules, and this allows you to make predictions. So this part is separated from how these predictions are used into the real world. Typically, these predictions help in deciding whether to do something or not to do something, or which order to do something. They lead to some sort of action, and this has real-world implications. It might be that a restaurant is shut down because you use this prediction model.

[7:48](https://youtu.be/GoMKi0N1LLw?t=468) So in the world of large language models, everything that is in this blue box has been compressed sometimes. We just get predictions. So if in the past, maybe someone would create a classifier with past data around restaurants' food safety, maybe right now, they wouldn't train this model at all. They would just use some sort of large language model like GPT 5.6 or CLOD, OPUS, to generate this data. Again, this model is also being trained with data, but data that most of it is unrelated to that specific outcome, it can generalize quite well. But what this distinction here outlines is that there's a difference between prediction and judgment. So if you're doing some sort of predictive model, you're trying to answer the question of if things continue as they have, what is likely to happen in this case? So if you continue, if the world keeps as is, what is the likelihood that this type of restaurant in this neighborhood with this history will have a violation of the rules versus what you should do with that, which was the right-hand side of the diagram, which is given your goals, your options, your obligations, what should you do? How should you use this information? And one way to think about this is that there are some questions where it's very clear what you should do. And in those cases, prediction is often very appropriate. One terminology introduced by this paper by Kleinberg at all is that these are umbrella questions. The action that you must take is clear. So in the food safety case, the action that must be taken is to inspect restaurants.

[9:52](https://youtu.be/GoMKi0N1LLw?t=592) There's no question of what action should be taken. There's this mandate that the city should inspect the restaurants. And thus, there's this clear action. The question is, we don't have enough budget to inspect every restaurant all the time. So how should we do this? And these are called umbrella questions because the analogy is that if you know it's going to rain, you can bring an umbrella. So there's an obvious action that should be done to prevent something from happening. And if you can predict what is the outcome of some sort of expensive thing, of you trying to inspect the restaurant, you might as well use it to take this action more efficiently. But other policy questions, they are really about the effectiveness of specific actions. So one might ask, does food safety inspections actually decrease the number of food poisoning incidents in the city? Or you could ask, does a specific job training program reduce unemployment? And in many cases, people conflate these two things. People think that because we can, for instance, create an algorithm that predicts unemployment, that this will be helpful in trying to answer this question around the effectiveness of action.

[11:21](https://youtu.be/GoMKi0N1LLw?t=681) Because just predicting unemployment here, for example, doesn't tell us anything about your job training program. In a way, what you're trying to do here is to intervene upon the world and change the way that the world works. And what your prediction is doing is trying to predict things as things were, as the world used to work beforehand. So if that makes sense, I'll try to make it a bit more interactive. And so there's two challenges here, right? So one of the challenges is that a child welfare agency must decide whether to send a caseworker after each hotline call. And the second scenario is that a housing authority must decide which households get priority for scarce vouchers. In this case, which of these questions is better suited to prediction? So I'll try to, it's A or B. Let's try to see which one the chat believes is the most appropriate. So I see some disagreements already, which is always good.

[12:36](https://youtu.be/GoMKi0N1LLw?t=756) So it's fairly mixed, right? And this illustrates somehow why there is confusion here. And I would argue that the answer here is letter B, that a housing authority must decide which households get priority for scarce vouchers, right? Because in this case, we do already have like our umbrella. We know what we should do, right? So in this case, we have these vouchers, we are, we know that we must assign them. And the question here is what, like who to assign them to, right? In the first case, you know, like, and this question is kind of a tricky question. So don't worry if you answered A, but in this, in the first case, the point here is that it's like the decision that the agency is facing is whether to send the caseworker after each hotline call. So if this was framed differently, if it was said, you know, like whether they have, you know, like the decision there is whether they do it for, like whether they enact this intervention or not, right? And not necessarily this would be the case, right? It could be that the welfare agency never sends anyone or sends everyone. The effectiveness of this as the effectiveness of the vouchers is outside of the scope of this. And the, if I had phrased A as something like a child welfare agency must decide which hotline calls get their caseworker sent and they must, you know, like they must send at least, they must send caseworkers in case where there is this or that, then it would be correct. But it's worth thinking about whether, what is important for you is that you already decided which action you're going to take and you're trying to optimize the consequence of these actions or whether your question is about whether to do the action at all, right? So hopefully that example can stress this,

[14:42](https://youtu.be/GoMKi0N1LLw?t=882) it stresses this differency. But even, but then for the second part of this short introduction to this theme, I wanted to stress that even when prediction looks appropriate, sometimes it isn't appropriate. And What I want to do for the next couple of minutes is to give you a couple of ways in which predictive AI often fails to deliver what people expect it to do, right? The first cause of, like, failure is that in many cases you have that you're actually trying to predict the wrong target, right? So sometimes predicting what you really want to predict is really hard. So people settle into trying to predict some sort of proxy variable that should capture what they want to predict, but this has sometimes pretty bad implications because the way these proxy variables work encodes all sorts of problems, right?

[15:43](https://youtu.be/GoMKi0N1LLw?t=943) So this is the case from this famous science paper by Obermeier et al., where there was this big predictive AI system which was trying to find which patients would benefit the most from extra care management programs, right? So it's a healthcare prediction task. And the way the model worked, the data that they had at hand was that they wanted to try predicting—they tried to predict what was the future healthcare costs for each person, right? So they took historical data from how much each person claimed in healthcare costs, and they tried to say, hey, let us make a predictive model that, given a person's feature, tries to predict how much they're going to spend in the future, right? And you can imagine that this is, like, something—like, it feels correct, right? Because ultimately the idea of doing this care management programs is to reduce future healthcare costs.

[16:48](https://youtu.be/GoMKi0N1LLw?t=1008) So you're trying to target people that are going to spend a lot of money on healthcare to maybe prevent their situation from worsening beforehand, right? However, it's worth thinking about what is encoded within this proxy variable that we're trying to use to—like, we're trying to essentially use the healthcare costs as a proxy for how much do I need these programs, right? And the problem there is that groups that historically had less access to healthcare were discriminated by this algorithm, right? So for example, the paper found that black patients have less access to healthcare in general. If you live in a place where access to healthcare is more challenging, you will spend less money. So the algorithm would systematically mispredict the needs for communities that perhaps needs this extra care a lot, right?

[17:51](https://youtu.be/GoMKi0N1LLw?t=1071) Because they're not capable of getting access to this care, right? So there's some sort of circularity here that prevents these predictive models from really working because we're trying to predict who needs things the most by using this proxy variable which encodes how much people have used this in the past. But communities that are underserved by the healthcare system will thus be excluded from this predictive algorithm. So the lesson to draw from this is that you should choose what you will predict very carefully. If you're trying to predict a proxy, something that approximates the real construct, the real thing that you're trying to predict, if this proxy is systematically wrong for some groups, your predictions will be worse for these groups, right? And you know, this is particularly dire in the case where these predictions go back and impact the real world because they might create this feedback loop where, you know, because these people don't have access to healthcare, they're never predicted to receive this extra care and their access to healthcare will decrease even more and, you know, and so on and so forth.

[19:03](https://youtu.be/GoMKi0N1LLw?t=1143) So as we often use these predictive algorithms to act on the real world, this is a problem. I'll take questions at the end, by the way, because I'm almost there. But the second case which I wanted to highlight is that in many cases, accurate prediction may actually be impossible, you know? So it is not the case that with bigger models or better algorithms, we will always be able to predict everything, right? There are things that by their very nature are very high variance, right? If I am to ask, like, what is the weather going to be like in 10 years, there's an argument to be made that no matter how good my predictive model is of the weather, there's a level of uncertainty that is just irreducible, right? That I would never, you know, there's too many variables for me to understand how the weather would be like in 10 years from now, right?

[20:10](https://youtu.be/GoMKi0N1LLw?t=1210) That's a lot of time. One example of this that's very fun is from my Princeton colleague, Matt Salganik. Matt did a lot of work with this very interesting data set called the Fragile Families Study. The idea there was that this is a big study that spans many years, and they basically tracked people from birth to adulthood in families that were facing, like, or were at risk of facing financial problems or all sorts of struggles, right? So they were fragile families. And you know, what Matt was interested in predicting or into understanding how predictable it was, was at how children would turn out at age 15, given a bunch of variables, 13,000 variables that were measured per family from birth to nine years old, right?

[21:12](https://youtu.be/GoMKi0N1LLw?t=1272) So basically, people, they receive the income of the family, all sorts of demographic characteristics of the family, all sorts of characteristics of the parents, and they were asked to predict things like what will be the children's, the child's GPA, what will be their grit, like some measure of grit made by psychologists, what will be the household material hardship of this kid, will this kid's family be evicted from their household? So they were asking how random those things were, and how can they be predicted from, you know, this, like, all these environmental variables that were predicted from their family and from this very kids when they were young, right? And the, you know, if, you know, if you consider that zero is a random guess, and one is perfect prediction, all these different outcomes had very, very, very low predictive power, right?

[22:17](https://youtu.be/GoMKi0N1LLw?t=1337) Oh, sorry, all these models had very, very low predictive power. So they all ranged from 0.03 to 0.23, right? So this is a case where social scientists spent millions of dollars, and like tremendous amount of time gathering 13,000 variables for each family, right? In many cases, you know, we will be able to have all these variables to input into our predictive models. But even then, you know, the best models, and this was an open challenge, a competition that anyone could enter, the best models could explain only like 20% of the of the outcomes, right, in this rough metric. So, um, so it is like this, this basically warn us that even with all the data in the world, or even with a lot of data, there's often limits, there are often limits to prediction, right? So, in this, I think that is particularly relevant for this crowd, because there's a lot of companies that sell predictive technologies without proper validation, right?

[23:22](https://youtu.be/GoMKi0N1LLw?t=1402) So, um, for example, like there are companies that hire that that sell you that do, you know, you will speak to a camera, and for like, essentially to assess whether you should be hired or not. And they will analyze your voice, and your facial expressions, and they will say whether you're going to be a good employee or not. And, you know, like, can this even be predicted, right? And often, often cases, these companies don't have a good way to validate these predictions, they don't have a good way of saying, hey, like, these are the people, these are the scores of people that were previously in the companies. And this is how much, I don't know, sales each one of those employees made, right? So they don't have this kind of good, good predictive capabilities. Um, and again, right, you're, you're maybe just capturing some sort of proxy variable that is just encoding other things rather than what you really want to predict.

[24:23](https://youtu.be/GoMKi0N1LLw?t=1463) Um, so, uh, I want, I'm looking forward to have having like a longer conversation about this and to hear your questions. But I wanted to leave you all with four questions to ask yourself when you're thinking about predictive systems, or, you know, if you're considering including predictive systems in your workplace. Um, the first one is thinking about what do you really want to predict? The second one is, um, um, can, can, uh, can that thing be predicted, right? Right. Um, is, is, is there a reason to believe that, um, whatever, whatever construct I'm trying to predict is, is, is indeed capturable from the variables that I have at hand? Um, the, the, the, the, the third thing is saying, if you're settling for proxy variables, right?

[25:25](https://youtu.be/GoMKi0N1LLw?t=1525) Um, like what are my proxy variables really capturing? Are they capturing the right thing? Are they capturing what I actually want to predict or not? And once you deploy these things into the real world, it's worth thinking about what their consequences will be, right? So, uh, I'm happy to have this as an open conversation as a chat. Thank you so much for your time. Um, yeah, excited to present to you all.

**Mihir Kshirsagar**

[25:49](https://youtu.be/GoMKi0N1LLw?t=1549) Thank you. Thank you so much, Manuel. Uh, we, we have some questions already, uh, so I would love to get to them, uh, and I'll, I'll put them up, uh, and I'll ask them again, just for everyone's benefit. Uh, but, uh, please put more questions in. I have some questions for Manuel as well, as, as we go through it, but, uh, let me go with the first one. Uh, Manuel, this is a question from Sherry, uh, asking, would you walk us through your process of creating helpful questions for prediction and who collaborates with you through the process?

**Manoel Horta Ribeiro**

[26:26](https://youtu.be/GoMKi0N1LLw?t=1586) Yeah, that's a great question. So I feel that, um, the, I'll remove this just to see you all. Um, so I think that the, the, the importance, the, the important thing here to in, in creating this, like, what are the targets to prediction is to make this as like a multi-stakeholder process, which includes people that are familiar with what are the ins and outs of the data at hand, right? So for instance, um, um, the, the, the, the, I think it's important to understand what are the, the places like, so for, for, in, in, in, as a good heuristic, right? It's, it's worth for you to think about what are the places that have some sort of scarce allocation of resources or processes that already exist, right? So if you think about some sort of bureaucracy, if you think about some sort of institution, this institution will likely have, um, some sort of bottlenecks and this bottlenecks will involve deciding where to take action in some cases and like taking action in some cases and not taking action in other cases.

[27:43](https://youtu.be/GoMKi0N1LLw?t=1663) And I feel that these are very natural places to, to use, uh, for questions of prediction, um, as you're creating this predictive processes, right? And this is going back to the first part of my answer. I think I inverted it a bit. It's worth for you to consider involving a lot of stakeholders to understand whether whatever you're trying to predict with the data that you have actually captures the construct that you want to predict, right? So given like, for example, if we want to predict, um, healthcare, uh, spending, uh, like who, who, who needs these extra care programs, um, if you include public health, uh, professionals, they'll probably think about the biases that would come from using, uh, healthcare spending as a proxy variable.

**Mihir Kshirsagar**

[28:34](https://youtu.be/GoMKi0N1LLw?t=1714) Yeah, that's great. Reminds me a little bit of, um, a, a challenge that I've heard, you know, where, uh, the coast guard had to go out and find, uh, they used a predictive system to find icebergs. Um, but they didn't, they didn't actually figure out when they went to do this, uh, that they didn't consult any geologists or whoever it is who specialized in icebergs. They didn't realize that icebergs flip and, and change shape overnight. And so when you use a plane that flies over to predict it, uh, or a satellite, you may miss the fact that it's the same iceberg. It just looks different. They wound up, uh, having to, to, to change some of that. And so, you know, in that process of sort of incorporating users in it, uh, in the prediction process, uh, you know, where, where have you seen, I guess this is, you know, you've, where have you seen people do that? Well, uh, you know, do you have some examples of places where you've seen people approach this in a better way than maybe some other place?

**Manoel Horta Ribeiro**

[29:45](https://youtu.be/GoMKi0N1LLw?t=1785) Yeah. I mean, I feel that in many, like one, like cases where it is often works are cases where there is some level of accountability or the predictions that are made or predictions that are not made. Uh, I work a lot with like. social media platforms. And one case where, you know, I feel that predictive systems face intense scrutiny are, for instance, in online communities, right? So if you look at a website like Reddit, where essentially you have predictive systems built within each online community that are governed by the owners of this community, you have this kind of like short feedback loop where actions are constantly reviewed by the community's owners, right? So there's a sense in which someone can look at some prediction and some action that was made on that prediction and get a sense of what is being flagged, right? So I think that maybe one broad pattern here that would be very helpful is to have some sort of transparency, which is ultimately the key way in which we try to understand what are the ways in which our predictive systems are faulty, right? So there's a big concept in predictive systems called error analysis, which is essentially, you know, if you have some predictive system, and for instance, if you're having this good inspection system, it's worth for you to keep track of the times where you're, you know, like, like, if you have an accident, you went there and inspected it, it's worth for you to keep track of where does your system makes mistakes?

[31:25](https://youtu.be/GoMKi0N1LLw?t=1885) And can you find systematic ways in which your system is making a specific type of mistake, right? Because then you can try to reverse engineer, for instance, your data and try to understand why is it the case that your system is making these mistakes, and maybe address this, right? So you can think of like, hey, let me look at the restaurants that get that, you know, they get very high priority. But whenever I try to inspect them, everything's fine. And let me think about if there's something about the way my model works, or the data that I trained, that really explains this.

**Mihir Kshirsagar**

[32:04](https://youtu.be/GoMKi0N1LLw?t=1924) This dovetails to the question that that anonymous participant had, you know, which says, is there a way to create traceability for these predictions that can be saved, so it can be audited by the public? Maybe one thing that I'd add to that, because you've been addressing this, how do you deal with some of the objections from maybe some people, you know, in the same community who would say, oh, but if we give up prediction logic, and we tell everyone it, then all the cheaters and the scammers will take advantage. And so our predictions, you know, the restauranteurs will evade our system, because now all this data is in the public. How do you

**Manoel Horta Ribeiro**

[32:48](https://youtu.be/GoMKi0N1LLw?t=1968) counteract that kind of response? Yeah, that's a great point. And definitely, I think that these two things are at odds, right? So in one hand, and this depends a lot on how the variables are collected, right? So I guess there is truth to the fact that if the variables that you're trying to predict, and you're enacting some action are self-reported, for example, you know, if you're trying to predict things based on whatever people are reporting, there is some level of gamability, right? In the sense that making these traces public would allow people to understand what are the things that they should declare, so that they don't get inspected, for example, right? So this is definitely the case. In other cases, right, it might be that this, whatever you're trying to predict is very hard to cheat, right? So there's some level at which it might be easier to comply with whatever safety measures than to cheat. You could think about like, if it's something around Google reviews, for example, and you know, it's actually like harder for you to do it, you might as well do it. So I don't think that there's like a silver bullet here. But I do believe that it's very specific on like, how do your variables, where do your variables come from?

[34:08](https://youtu.be/GoMKi0N1LLw?t=2048) For example, right, like, it could be that a very simple way of doing this for the food running example is that probably the biggest predictor is past inspections, right? So if you're being inspected all the time, and you know, you're always good, it's very likely that you would be bad, right? The restaurant doesn't gain so much from, you know, becoming dirty, versus, and this is something hard to gain, right? Because you yourself are going there and doing the inspection. And in this case, you know, you could imagine having some sort of public record of like the inspections and how they went. I don't know actually if this exists in Chicago. But you know, like whenever you enter an establishment, you actually do see the assessment of the latest

**Mihir Kshirsagar**

[34:53](https://youtu.be/GoMKi0N1LLw?t=2093) prediction of some sort, right? Yeah, I remember when that was rolled out in New York, when they started putting up the letter signs. Initially, a lot of restauranters complained, but then after time, they understood the value of this transparency. So there's a lot of value to transparency, even if it can be gamed on occasion, as you said, you know, some of that, you can figure out how to deal with it. Let me ask then, there's lots of questions, which is great. Keep it coming. Now, ask the next question here. It says, you mentioned the danger of predicting the wrong target. What steps should we take to ensure we have the right target? What should we do if leadership is interested in building predictive AI before ensuring we have the

**Manoel Horta Ribeiro**

[35:43](https://youtu.be/GoMKi0N1LLw?t=2143) right target? Yeah, that's a great question. I guess that the kind of annoying answer I would give is that there is no super straightforward way of ensuring that you have the right target, right? So I think this is like, the prediction target is kind of an argument. And the way in which you decide that a target is not appropriate, it's kind of like trying to find a counter argument, right? So the best way to do this is to, I think that the way that you could usually think of like why a proxy is not correct, is for you to come with potential reasons why this proxy might be related to a second variable that can itself explains the outcome that you care about, you know? So it's similar to, you know, when we discuss kind of studies about nutrition or studies that are not run in an experiment, we often try to think of, I guess the technical term here would be confounders, like things that cause both the, like whatever you're trying to predict, the proxy and the outcome that you care about, right? So if you're thinking about, you know, a classifier that uses, like in the example that I gave, you have this confounder, which is access to health care, right? And access to health care predicts both the need for these extra care programs and it impacts the health care spending. And in this case, when you have something that is related to both these variables, you're going to have problems, right? So like I would argue that the best way to do this, it's come up with this hypothesis and try to investigate if this hypothesis holds true. If you have something that affects both the real target and the proxy variable that you're using to predict. And you know, like maybe a way to make

[37:48](https://youtu.be/GoMKi0N1LLw?t=2268) the argument for the leadership would be to stress out this hypothesis, right? So it's kind of like, if you think about how a lot of things were discovered and how a lot of associations were uncovered, it had to do with this, right? So if you think about smoking causes cancer, you know, you had to say, hey, not only smoking causes cancer, but there's no third variable that kind of explains both smoking and cancer. So in a sense, it's a causal argument that you must make. And you could ask for small analysis seeing if these things are correlated, right? If there's a sense in which this variable is correlated with this proxy variable and with

**Mihir Kshirsagar**

[38:32](https://youtu.be/GoMKi0N1LLw?t=2312) the target variable. Great. And I should say, you know, some of these questions, keep them coming. And we'll be doing a wrap up as well at the end to think about what processes need to change internally if you have to respond to some of these questions that are coming up. Okay, well, Alexander has a question, says, you shared four questions that we should ask ourselves when considering predictive AI. What should we do to teach ourselves the technical aspects to understand whether or not the thing we want to be predicted can be predicted? How much of these questions can be answered without a background in predictive analytics?

**Manoel Horta Ribeiro**

[39:18](https://youtu.be/GoMKi0N1LLw?t=2358) Yeah, that's a great question. I feel that a lot of the understanding of whether a question can or cannot be answered from data, it's actually somewhat detached from the methods of predictive analytics, right? So you can be an expert and like, you know, I'm a computer scientist, us computer scientists are experts in being experts in, like, we're experts in predictive models. We have all these machine learning models that we came up with. But consistently, we actually fall prey on not understanding whether someone can be predictive or not. So I think in many cases, this is kind of an empirical question, Alexander. Like, there's a question of like, is this really predictable, right? I feel that this is less of something that you can come up from first principles. And in many cases, it's something that you need to understand in the real world. So my suggestion would be, you know, the most important thing is for you to be able to scrutinize whether the evidence someone is showing you that determines whether something has predictive power or not is credible, right? So if someone if someone comes to you and says, hey, I have this hiring software that, you know, can basically predict whether someone will be a good employee or not, right? I would like maybe like, you know, like, let's be skeptics for a second, right? Like, maybe it is the case, right? Let's not take a stance here. But what I would wonder is, what evidence do you have of that, right? And I would ask for this evidence. And I would use this kind of conceptual lenses that I've discussed to reason about this evidence. Like, do you really have some way of measuring productivity in this case, right? Do you really have convincing evidence that these people that have a higher score indeed went to perform better? In many jobs, this is very hard, right? If you're working a

[41:23](https://youtu.be/GoMKi0N1LLw?t=2483) sales job, maybe that's possible, because there's a very quantifiable way of doing it. But if you're working any sort of other job, it's very, very hard to get like a one number that predicts productivity. So this alone would make me very skeptical, right? That there's that you have no variable even that you could use that would convince me, right? So then you think of like, you ask, okay, so what can you predict, right? And then you can get go on and converse with the person around this. But I really take this as like, an empirical question. And I would put the burden of the evidence on the person who's trying to predict to start trying to sell you something, or who's trying to convince you that they can predict something, right? As a kind of like,

**Mihir Kshirsagar**

[42:07](https://youtu.be/GoMKi0N1LLw?t=2527) strategy. Interesting. Yeah. Maybe another question, along the same lines, you know, how do you deal with, how do you accurately deal with variables that interact with each other, that change the outcome, when the time the variable occurs can increase or decrease

**Manoel Horta Ribeiro**

[42:29](https://youtu.be/GoMKi0N1LLw?t=2549) the probability? I mean, like, I'm not sure if I'm answering exactly the correct question when I'm saying this, but like, so you're free to send another question if I didn't answer your question. But I think it's worth like, there's no problem, really, if the variables are interacting, in some sense, that's fine for the prediction. The question there is basically like, there's two problems here that you can have, right? The first one is that really, you're not doing prediction, you're doing judgment, right? So the first one is that indeed, what you really care about is that you don't want to, you don't want to predict something, you want to take an action and see how the world evolves, right? That would be one case in which this could matter because you're now changing something in the world, and it might, you know, cascade and have some other impact. And the other case is basically where whatever you're measuring is kind of a proxy, right? It's not the real prediction, right? So like, in this case, it's very nasty, right? Like, there's no real way of you making the proxy variable, which interacts with this third variable that you care about. So like, one way to frame this would be that healthcare spending interacts with access to healthcare, right? And this is the problem that you have. And in this case, right, like, there's no real answer, right? Like, you need to find a better target variable, you need to find a new way of getting this data that you need. And in some cases, right, getting accurate data is tremendously expensive, right? So if you think about, like, a lot of governmental agencies spend millions of dollars running surveys around businesses to understand how businesses are. And in some cases, if you don't go the extra mile to actually collect this data, that is really what you want to measure, you will be in trouble if you try to use simple things,

[44:32](https://youtu.be/GoMKi0N1LLw?t=2672) right? If we stopped, you know, if we stopped collecting any sort of economic data, and we relied only on like, trust pilot reviews, we wouldn't be great, right? Like, there would be a lot of problems. So there's a sense in which there's some wisdom in thinking, okay, can we go ahead and, you know, spend money to collect this data or to get some reliable measure of this

**Mihir Kshirsagar**

[44:53](https://youtu.be/GoMKi0N1LLw?t=2693) thing that we care about? Right. And this, maybe this is this is a question my own. But I think it relates to a lot of these questions, which is a lot of these models that have come up—and you've done work in this area—involve pretty large data sets, right? I mean, you're trying to predict across—if you're talking about sort of Internet-scale data on social media platforms, you have a lot of data to work with. But when data is pretty sparse, when it comes in the government context, if you're trying to make certain kinds of decisions where there aren't that many variables, there aren't that many data points, it's expensive, as you said, to go out and collect the data. As you also mentioned, marginalized populations may not be even represented in that data. So how do you account for the lack of data availability into what can be predicted or not predicted? How should we think about this question? If you're a government person trying to employ a predictive system, how much data do you need to have an effective system? How should we be thinking about those questions?

**Manoel Horta Ribeiro**

[46:11](https://youtu.be/GoMKi0N1LLw?t=2771) Yeah, that's a good question. And I think I can piggyback on Hector's question on how to audit like predictive models. And I think that the way to think about this is that you can think of this process of creating and assessing a predictive model as this sort of like iterative process, right? So I think it's naive to think that you build a predictive model and that you have this black box tool that is magical and that will be able to do everything, right? So in a way, right, like I think that the correct approach is for governmental agencies or any sort of institution to have this process where they're able to come up with predictive models, right? So they're able to collect whatever data they have at their hands and say, hey, this is a prediction for this task that we think is relevant. They are then able to assess the effectiveness of this predictive model, you know? So you can think of this by like if it's a very high risk area, you can try to assess this in the background, right? So maybe you're conducting business as usual and you have a pilot where in the background you're running this predictive model and you're trying to see, you know, like maybe, you know, like let's suppose that right now the boot inspection is at completely at random. This is how things always were. It's lottery and I'm running my predictive model in the background at the same time as I'm randomly inspecting the units and I'm seeing how like how well my predictive model is dealing with like the many cases. And then as this predictive model is running in the background and even as it gets deployed is to have some sort of way of continuously auditing this predictive model, right? So maybe one way that one way that you could do this, for instance, is to keep a share

[48:14](https://youtu.be/GoMKi0N1LLw?t=2894) of your decisions random or done in other ways. And you're always comparing the performance of your predictive model with these other ways in which you're running your system. Or, you know, that you're you have some sort of additional checks for each of the decisions that you take and you see how your predictive model is working or not working. So it's worth for you to think about continuous mechanisms to audit this predictive model. If you care specifically about, you know, specific groups in your data or specific populations in your data, you can always continuously run analysis that is trying to see whether there is some sort of disparate impact being enacted by this predictive model, right? So you can try to think about, you know, are restaurants from all the different neighborhoods equally represented, right? And if you think about this, you can try to start creating some sort of, like, limits through your predictive model that they have to confirm to such boundaries to ensure some notion of fairness, right? That, you know, we need to audit restaurants in Manhattan and in the five boroughs of New York equally or something like this. There needs to be some sort of minimal amount of restaurants per borough. So it's worth for you to think of both these things.

**Mihir Kshirsagar**

[49:37](https://youtu.be/GoMKi0N1LLw?t=2977) Yeah. And, you know, we'll be, the next seminar will be, our colleague will be Alexander Korovo, who's a collaborator and a colleague of ours. We'll be talking about how to audit such systems. One question we have related, I think, almost directly to what you were just saying was, you know, do you create ecosystems, maps to understand the system so that wrong proxies are not chosen to increase the accuracy of the prediction?

**Manoel Horta Ribeiro**

[50:08](https://youtu.be/GoMKi0N1LLw?t=3008) Yeah. Yeah. So I think that that's a great point. And like the way for you to think about this is basically that the way, I think that the most straightforward way of doing this is to come up which, like what is often called in the literature as protected variables. So these are variables that you don't want to be unfair towards, right? So maybe you want the different neighborhoods of a city or, you know, people from different socioeconomic incomes to be equally impacted or that resources are equally distributed towards them. And you could think of, continuously evaluate how your system is doing with regards to this variable, these groups that you're creating, right? And there's a vast literature, Alexandra, who is presenting next week is an expert on this, of like how do you adapt these predictive algorithms to also conform to some rules that you care about, right? So maybe, you know, for like the sake of the business, because there's a lot of work, you don't want to inspect people two months in a row, right?

[51:17](https://youtu.be/GoMKi0N1LLw?t=3077) So you can add this to your decision, like either to your predictive model directly by downweighting this or by the decision rules that you do on top of your predictive model, right? And you can create this and it's particularly helpful if you have some sort of what are the variables or what are the things that I want to ensure that are being, for instance, fairly distributed across specific districts or specific groups or whatever is the variable

**Mihir Kshirsagar**

[51:44](https://youtu.be/GoMKi0N1LLw?t=3104) interest here. Right. So it looks like we're done with all the open questions. Manuel has answered all of them fantastically. I think, you know, as we continue through this series and really appreciate sort of the very active engagement from all of you, you know, one thing I would request those of you who are listening is to also come up with very specific examples of problem areas that you're running into. And I will say that at CITP, one of the things we're very committed to is to helping governments with some of these hard problems. And so if there are particularly difficult challenges that you're facing, I'll speak for Manuel, for myself, I think we would be very interested in sort of helping you to think about. So and the last question coming in is actually was the one that I was going to close with anyway. So you predicted my question good.

[52:47](https://youtu.be/GoMKi0N1LLw?t=3167) You narrowed the variables and you understood the ecosystem that I was coming from. So Manuel, the last question for you is, you know, what's the one thing the speaker would like us, the audience members, to take away from this presentation? Yeah. So, I mean, I think that

**Manoel Horta Ribeiro**

[53:04](https://youtu.be/GoMKi0N1LLw?t=3184) like in my view, you know, I'm a technologist. I'm excited about this predictive systems. I think that there is a lot of cool things that can be done by incorporating prediction into government, into institutions. And but you know, like what it takes for this to be successful is for us to be clear minded of both, what are we doing, you know, like with these predictions? And, you know, like, are we trying to learn about the world or just predict the world? And then if we're indeed just, if we're not worried about learning about the world, but indeed predicting the world, which is the case where predictions are appropriate, what are like, what is the way in which this prediction is being done? And, you know, is this a convincing story that, that, that, you know, that we should try? And if not, you know, how can we, how can we make it so right?

[54:05](https://youtu.be/GoMKi0N1LLw?t=3245) So I'm, I'm, I'm, I'm an optimist. So like, you know, like the, the take here, I don't want you all to think that, you know, there's, we should be, there's no place for, for prediction in, in, in, in, in, in, in my agency, in my job. I think that the, the question is, like, how can I be ensured that this is done the right way? And in a way that will positively impact like whatever, whatever I'm doing. Right. Wonderful. Well, thank you so much, Manuel. Thank

**Mihir Kshirsagar**

[54:36](https://youtu.be/GoMKi0N1LLw?t=3276) you Innovate US for, for hosting this platform and we will see you next week. Thank you.

**Audience**

[54:46](https://youtu.be/GoMKi0N1LLw?t=3286) Thank you so much, everyone. And thank you, Manuel, for sharing your insights. I'm here for such a great conversation and thank you for everyone engaging in the chat and submitting great questions. On your way out, everyone, please take a moment to complete our post workshop survey. It's how we determine what sort of things you want to learn next and plan future workshops. And it's hugely, hugely, hugely helpful for us. Today's recording will also be posted on our website as always. So you can revisit the conversation or share it. Our next section, as Mahir said, is going to be next Tuesday, October 27th. It'll be from 2 to 3 PM Eastern with Alexandra Korolova, also from Princeton. And then similarly the week after that, 2 to 3 PM, November 3rd with Mahir. And I believe Christine will be posting links in the chat for registration. And that's all we've got. Thank you so much, everyone. And we hope to see you next Tuesday. Have a great rest of your day. And with that, we'll close today's session.

* * *

Original page: https://innovate-us.org/prediction-isnt-judgment-what-predictive-ai-canand-cannotdo · Licence: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org
