---
id: "workshop:debiasing-decision-making-in-government"
type: "InnovateUS workshop"
title: "Debiasing Decision Making in Government"
url: "https://innovate-us.org/debiasing-decision-making-in-government"
date: "2024-07-18"
duration: "90 minutes"
language: "English"
instructors: ["Emily Cardon","Erin Britton"]
recording: "https://www.youtube.com/watch?v=8F71LYxf-Jg"
transcript: "automatic, from the recording"
source: "InnovateUS (innovate-us.org)"
licence: "CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org"
---

# Debiasing Decision Making in Government

InnovateUS workshop · 18 July 2024 · 90 minutes

- Learn about how our decisions and judgments are affected by cognitive biases, such as confirmation bias, group reinforcement, and optimism bias
- Discuss how cognitive biases can affect each stage of the policy making and project management process in government, from how we notice a problem, to our deliberations about fixing it, to our attempts to execute solutions
- Gain tools to counter cognitive biases and make better decisions, ultimately resulting in better project planning, delivery, and policy outcomes
- **Register for the workshop [here](https://airtable.com/appdAyP4l2W9gxQdc/pagx9sXF2F53dI7kn/form)**

## Learning goals

- Learn about how our decisions and judgments are affected by cognitive biases, such as confirmation bias, group reinforcement, and optimism bias
- Discuss how cognitive biases can affect each stage of the policy making and project management process in government, from how we notice a problem, to our deliberations about fixing it, to our attempts to execute solutions
- Gain tools to counter cognitive biases and make better decisions, ultimately resulting in better project planning, delivery, and policy outcomes

## Speakers

- **Emily Cardon**, Director, BIT US
- **Erin Britton**, Senior Advisor and Head of Capacity Building, BIT Americas

Recording: https://www.youtube.com/watch?v=8F71LYxf-Jg

## Transcript

Automatic transcript of the recording: names and terms can be misheard, and the times can be a few seconds off. Each time links to that moment of the video. The transcript alone: https://innovate-us.org/debiasing-decision-making-in-government.txt (plain text) and https://innovate-us.org/debiasing-decision-making-in-government.json (every segment with its start and end second).

**Unknown speaker**

[0:01](https://youtu.be/8F71LYxf-Jg?t=1) And turn it over.

**Erin Britton**

[0:04](https://youtu.be/8F71LYxf-Jg?t=4) All right, wonderful. Thank you so much, Hannah, and thanks everyone for joining today. We're really excited to be here. I'm just going to share my screen. Hopefully this will work. Wonderful. And as we're getting started, I know many of you have already introduced yourself in the chat, which is wonderful. I just have one more quick ask for you. We'll be asking you to use the chat throughout, but I would love to have your answers in the chat to this question that you see on the screen. Imagine you could get one metric about a person to assess how generally good their health is. What one metric would you choose? Go ahead. No wrong answers. No right answers. What metric would you choose? Go ahead and put that in the chat. Awesome. I have some great ones coming through here. Income, weight, smiles. I love that number of smiles per day.

[1:05](https://youtu.be/8F71LYxf-Jg?t=65) How do they feel? Yeah. Resting heart rate, stress levels. Hours of sleep. Ooh, car commute time. All right, some really good ones coming through here. Oh, diet, vegetarian or vegan. Time in nature. I love that one. These are great. Amazing. All right. Thanks so much for your engagement of all of these answers so far. I have one more question for you. So I asked you what you thought would be a good metric for assessing what someone's health is. You have a great range of answers. My last question for you before we dive in, what do you think would be a good single metric to assess how good someone's judgment is, how good they are at making decisions?

[2:08](https://youtu.be/8F71LYxf-Jg?t=128) Here's your best guess in the chat again. No right or wrong answers. One single metric to assess how good someone's judgment is. Really good ones coming in, talking about major decisions. Being by example. Sometimes people say, you know, who they married. Track record of past decisions, critical thinking, their reasoning, how they explain their decision making. Ooh, ratings by colleagues, kind of a peer effect. Yeah, great. All right. So that was maybe a little bit harder to answer, although we have some really great responses coming in. But it's a bit of a tricky question. We saw a few people also say, you know, there is no one measure for that or, you know, that's a hard one.

[3:13](https://youtu.be/8F71LYxf-Jg?t=193) And in today's session, we're going to give you one answer for what that might look like. So we're going to dive in. I'll give you the answer, our answer for that in about 20 slides. So enjoy the suspense until then. All right. So diving in. My name is Erin Britton. I'm really excited to be with you all today. I'm joined by my colleague, Emily. Emily, do you want to introduce yourself quickly? Come off of mute.

**Emily Cardon**

[3:38](https://youtu.be/8F71LYxf-Jg?t=218) Yes. Hi, everybody. I'm so happy to be here and talking about this topic that both Erin and I care about so much.

**Erin Britton**

[3:46](https://youtu.be/8F71LYxf-Jg?t=226) Awesome. And as Hannah mentioned, we're from the Behavioral Insights team. So we're a social purpose organization and we use behavioral science and rigorous evaluation to improve public policies and services. We work with cities and the states across the U.S. to help government incorporate an understanding of what drives human behavior into how they design programs and services, building behavioral insights units within government and collaborating on program designs to encourage various socially impactful behaviors on the parts of citizens and residents across a variety of different policy areas. So in this session, we're actually shifting our focus a little bit to look at decision makers inside of government. So looking more inwards at how can behavioral science help us understand how decisions are made within governments and help us to improve decisions and judgments about what policies we pursue, how to implement projects, all with a view towards improving outcomes for residents. So that's what we're going to be talking about today. So let's get right into it.

[4:50](https://youtu.be/8F71LYxf-Jg?t=290) This session is in two parts. In part one, we're going to talk through kind of what we see as a fundamental problem. So in one sentence, if the problem is what you see on the screen, cognitive biases lead to worse policymaking and decision making outcomes. So we're going to unpack a little bit about what we mean by cognitive biases, talk about what that means and some of the implications. And then we'll talk a little bit about some of the solutions that you can take away from this session and put into practice within your team or your organization. And a lot of this comes from a report that we did several years ago called Behavioral Government. So if you're interested, there's a longer diagnosis of these challenges and looking at the evidence across government contexts. So we'll share that link after the session if you're interested. All right, so let's dive in. To get started, I'd like to cast your minds back to 2013. This image may be familiar to many of you if you stared at this smiling face in a frustrated way, trying to register for health insurance on a marketplace.

[6:01](https://youtu.be/8F71LYxf-Jg?t=361) And hopefully many of us are familiar with this example, but this is sort of the homepage of the rollout of healthcare.gov when we were looking at the health insurance marketplace, which went live on October 1st, 2013. And does anybody remember what happened here? Some nods. So it was a big challenge. Despite four million unique visitors on day one, only six people we found out later were actually able to successfully sign up for health insurance on this first day. And according to an analysis done by the New York Times, 20 million people who tried to access the site over the first three weeks, only about 500,000 managed to complete applications. And even a smaller percentage of them actually succeeded in obtaining insurance. So it's still a large number, but by the proportion of the people who are trying to do it, yeah, it was a major crash. I think Lisa put it best.

[7:01](https://youtu.be/8F71LYxf-Jg?t=421) So this is a really nice example of a dramatic and surprising policy failure. We like it as an example of a dramatic policy failure, but policies fail all the time. You know, there's always issues that are cropping up. Why is this one particularly interesting? Because it was a failure of this magnitude, especially from the administration at the time had been really tech savvy, had a lot of success with this. It was quite surprising. People were quite surprised by it. And we find this interesting because it suggests that we weren't really tracking reality the way we should have been in the lead up to the rollout of this healthcare.gov site, giving kind of the extensive process, many, many people involved. It it shouldn't it should be surprising. This was such a calamitous failure. This was surprising to us. And this suggests to us that there was bias in the decision making and the judgment process that led out to the rollout, led up to the rollout of the marketplace.

[8:02](https://youtu.be/8F71LYxf-Jg?t=482) We're going to dig into this a little bit more. But my favorite was also the Monday morning quarterbacks who said, oh, this was doomed from the start. Of course, we knew that it was going to fail. But, you know, of course, where are all those people in in in the lead up to this? Very easy to critique it from the other side. And what we say what we see is that challenges like the one experience with the healthcare.gov rollout are really not unique. Government policies that offer people essentially free money, various programs get really low uptake due to some design flaws. So an example out of the UK, they designed to have this very lengthy process with tons of different review steps for this green new deal, the green deal, which gave people cheap loans to buy energy efficient products for their home. So it would kind of get a loan for the upfront install costs and then they could pay it off over time with the energy efficiency savings. And after six months, they found that only four people across the entire UK had actually signed up for the green deal and it was actually canceled two years later.

[9:11](https://youtu.be/8F71LYxf-Jg?t=551) At the macro level, government also struggles to effectively manage mega projects. So if you're from the Boston area, you might know the big dig. What they were trying to do was take a six lane elevated highway that ran through the center of town and replace it with an underground highway, adding a bunch of new bridges. And so the original project estimates were around $2.4 billion. By the time it was all said and done, the ultimate cost was about $14.8 billion, almost seven times. So these are not unique instances, this actually happens across different contexts all the time. So what's important about these examples? The first is that the policy outcomes were both undesirable and they were surprising. So it is surprising that a government would expect 100 people to sign up and only seven of them do or that a mega project would be seven times costlier to the tune of $12 billion more than anticipated.

[10:17](https://youtu.be/8F71LYxf-Jg?t=617) And that we would get these surprises with so many people involved in every policy and decision that's required to bring that policy to life. And there's one thing that I really want to emphasize, if you don't take this away, it's okay for a policy to fail. Many challenges the government seeks to address are complex, they're hard to tackle, and the policies that we put in place aren't always going to be successful. If you're taking a big risk, and you know you have a 20% chance of success, there's a good chance, 80%, in fact, that you'll have a bad outcome. And when that happens, you'll say, ah, well, it wasn't surprising. We knew there was a high chance of failure. But if you're surprised with a miss quite often, then that's where we see the challenge or the problem. As my colleague says, government should be like a bookmaker, we should always know the odds. So we're really not talking about ensuring that policies never fail. That's not reality.

[11:17](https://youtu.be/8F71LYxf-Jg?t=677) That's not the world that we're living in. But what we're saying is that the tools that we're going to talk about today hopefully will help you better handle the likelihood of policies going wrong to help you use that information to manage projects more effectively. So hopefully that's kind of a clear distinction. So now that we've clarified that, what do we think might cause us as policymakers to misperceive reality in this way? Now, I'm going to give you our answer in just a minute. But let me ask you a question first. I'll ask you to put your thoughts in the chat. What do you understand by the term, the word cognitive bias? So I'll give you a minute. Feel free to put your answers in the chat. Okay.

[12:21](https://youtu.be/8F71LYxf-Jg?t=741) Some good ones coming in. Yeah. So limitation on our perspectives, our life experience shapes the way that we see things. It's like echo chambers, challenges in shifting perspectives, being dependent. Yeah. These are great. Silos. Perceiving things based on your own experience, perception. Amazing. Unacknowledged limitations. I really like that clarifying what information is used to make decisions. Yeah. Absolutely. So our working definition, very similar to what you all are saying in the chat, that cognitive biases are systematic deviations or errors in our decision making. And they're rules of thumb or heuristics that were often helpful to us, but can go wrong and influence our perception of the world.

[13:25](https://youtu.be/8F71LYxf-Jg?t=805) So we're going to take a look at some examples of cognitive biases in action in just a minute. And so just at the risk of belaboring kind of our point here on what we see as this sort of fundamental problem, but cognitive biases among people in government can lead to misperceptions of reality, which then cause surprisingly bad and avoidable policy outcomes. So in the next section, what we're going to do is talk about what does this really mean for us as policymakers and people working in government? We're going to introduce a few of these key biases. And the thing that we're going to drive home today is that knowing about these biases is not enough for preventing them. So we're going to talk about them. We're going to call them out. But then we're also going to talk about some tools that you can put in place to mitigate these biases in your own decision making. Because they're systematic, we have a reasonably good sense of when and where they're likely to occur. So we can put measures in place to mitigate them, which is great news for us.

[14:29](https://youtu.be/8F71LYxf-Jg?t=869) All right. So diving right in, we have kind of put together a very simplified policy model of policymaking that distills all of this down to about three stages to kind of crystallize and keep things simple. Again, this kind of this comes from a behavioral government report. So if you're interested in more on each of these stages, definitely recommend checking that out. The first stage is noticing. This is about how information and ideas enter our realm of consciousness. So we're going to take a little bit of a look at the biases that influence the noticing stage, some examples there. The next stage is deliberating. So this is once we've decided on a problem, what are we doing to discuss how we're going to solve it? How are policy ideas discussed and developed? And how do we make decisions about how a policy should be designed? And then third and finally, the executing phase. So this is where the rubber meets the road.

[15:30](https://youtu.be/8F71LYxf-Jg?t=930) You have the problem you want to focus on. You've designed what you think are some really good solutions. Now you're going into project management and delivery phase. We'll investigate some of the biases that affect success in delivery. Cognitive biases do affect all three stages of policymaking. Don't pay attention to the arrows right now. You don't have to worry about those. But what we're going to do is we're going to talk through one bias in each of these three areas to give a little bit of a sense of what we mean, what this looks like in practice. So we won't be able to cover everything, but that's what we're going to do today. All right, so I'm going to start us out by thinking about the noticing phase. And the bias that we're going to talk about is the allocation of attention bias. So this bias is about the amount of attention a policy issue receives and how it may not match how important that issue is. So I'm going to give you an example to hopefully make this bias a little bit clearer.

[16:33](https://youtu.be/8F71LYxf-Jg?t=993) All right, so I'm going to slowly reveal this graph. There's a lot going on. But this is from an economics paper that looked at 5,000 natural disasters over 40 years. And they looked at how much media attention they got in the U.S. media. And the results in this graph that I'm about to show you, before I unpack it, the graph is that it's a little bit complicated. So let's look at it together. On the y-axis, looking from the bottom to the top, you see different types of natural disasters. And at the top of the graph in blue, it says number of deaths needed to receive as much media attention as one death from a volcano. So what does that mean? It means that, empirically speaking, volcanoes are the coolest way to die. That's a little bit of a joke. But in other words, deaths from volcanoes get the most media attention, or they did in the 40-year period that was examined in this study.

[17:35](https://youtu.be/8F71LYxf-Jg?t=1055) So what the authors did was they used volcano deaths as their reference category. And then they said, okay, how many deaths are needed to happen from these other types of disasters in order to get the same amount of media attention as one death from a volcano? So for example, if we can bring up the first part of this, you'll see that for earthquakes and fires at the bottom of this chart, you need between 2 and 12 deaths to get as much media attention as one death from a volcano. So far, so good? Hopefully yes. All right. So now I want to ask you all, feel free to put it in the chat, how many deaths do you think it takes for storms, floods, and landslides to get the same amount of attention as one death from a volcano? Feel free to put your estimate in the chat. 145. And we had a question about how media attention was defined.

[18:40](https://youtu.be/8F71LYxf-Jg?t=1120) I can check the paper, and I think it was like a number of stories across several sources that they sampled. All right. So far, we have ranging for 500, 100,000. Yeah. Okay. So we have a huge range, which is great. That means it's not totally evident to you all. So yeah, for landslide, flood, and storm, we're looking at between like 300 to 900 approximately. A few hundred. So what do you think about for epidemics, droughts, and cold waves? This is pre-COVID, so the epidemic category is maybe a bit different. Okay. Yep. 5,000, 10,000, in the thousands. Great. Yeah.

[19:40](https://youtu.be/8F71LYxf-Jg?t=1180) So you're already getting more calibrated here. So we're looking in the several thousands of deaths to get as much media coverage as one death from a volcano. Great. So you're definitely getting a good sense of the exercise now. Last question. How many deaths do you think are needed to occur in a famine? It's our last number here. Okay. Got some coming in. 6,000, 5,000. Here's some higher estimates. 20, 10, 100,000. Yeah. All right. Rather depressingly, you need about 40,000 deaths from a famine in order to get the same amount of media attention as one death from a volcano. Now, why does this matter? Why have I taken you through this exercise? You might be able to think of many reasons why this is the case that some disasters get more attention or more geographically closer to the United States, involve U.S. citizens.

[20:48](https://youtu.be/8F71LYxf-Jg?t=1248) We are looking at U.S. media. And what this graph shows is actually only half of the story as far as policymakers and government is concerned, because this graph is showing you differences in how much media attention the disasters got. But the concerning thing is that U.S. government aid decisions were also highly correlated with differences in media attention. So if you are a person in a country experiencing an earthquake, you are much more likely to end up getting U.S. aid than if you were experiencing a cold wave. And the opposite is also true. If you experienced a disaster at the same time as a major media event like the Olympics was going on, you were kind of out of luck because there ended up being less likely to get aid relative to the amount of media coverage that you were receiving. There's only so much media oxygen available. So that's a bit troubling. I think we ideally would want all of these deaths or natural disasters to matter equally and for aid decisions to go where the most need is rather than just what gets the most attention. But it's a really rich example because I think we also can see both sides and understand why it is the way it is that government and policymakers operate in a really complicated ecosystem that includes media and public opinion.

[22:09](https://youtu.be/8F71LYxf-Jg?t=1329) And you just can't ignore it completely. So so we see the reasons why as well that that we see these types of outcomes. But this is a really nice example of that allocation of attention bias operating at the noticing stage, the stage of what actually gets on our agenda, gets responded to. Some curious if others have found similar phenomenon in your work where you you maybe disagreed with how much attention something got in your community or in your government relative to what you thought it's important to us. If you have thoughts on that, even just like a yes, this resonates or no or specific examples, definitely feel free to share that in the chat. As those are coming in. I'm going to hand it over to Emily to talk about the next stage in our policymaking process, the deliberating stage.

[23:09](https://youtu.be/8F71LYxf-Jg?t=1389) Thanks, Emily.

**Emily Cardon**

[23:11](https://youtu.be/8F71LYxf-Jg?t=1391) OK, so now we're going to move on to the deliberation stage, and this is when you've decided which problem you're going to allocate your attention, resources and time to. So let's say you're going to use our strategies that we're going to help share to address a really urgent and important problem. You don't have to figure out what you're going to do, how you're going to approach it, who you're going to talk to, who you're going to involve and what decision you're going to make at the end, like what's going to be your approach. We're going to see three biases highlighted on the screen. I'm going to zoom into the illusion of similarity so you can find out about these other ones again in this report. About illusion of similarity, what we mean by this is when policymakers incorrectly assume ordinary people are likely to share their opinions. Right, so you think basically that other people are going to think the same way that you do, lots of people, you might be sort of thinking about groupthink or a lot of the examples that you had in the chat earlier sort of got at this sort of sort of perceptions being in echo chambers, et cetera.

[24:17](https://youtu.be/8F71LYxf-Jg?t=1457) So this is all very related to this bias. So let's walk through an example again to make this concrete. OK, this is a really nice study from Bergman and Rogers, they're at Harvard. They're education researchers, and we really like this example. There was an intervention in a school that they were evaluating, and the intervention was to get parents to sign up to text messages about their child's progress in school. For all of those who have been in the education system or parents, you might be familiar with something like this. It basically tells you how much homework your kids are doing, how well they're doing in interim tests, et cetera, et cetera. The idea is that you want to make parents more aware of what their children are doing in school, the more meaningful and targeted the supports parents can provide their kids. Right. It's like the idea of more information, the better. This sounds like a really sensible intervention that would work. The researchers, the academics then conducted an RCT testing out three different forms of designing the sign up to the intervention.

[25:22](https://youtu.be/8F71LYxf-Jg?t=1522) So this is, again, you can build something, you have to make people come to it. How are you going to let people know about this new intervention and get them to engage? So the first you see in that like first bar on the left was a standard approach. I believe it was an email, but sort of just think about like normal outreach coming from a school basically said this is an initiative to help children do better at school. Please go to the website and sign up. So that's sort of like the status quo. They then tested two different two different other other ways of enrollment. The second one was the simplified request by text to enroll where parents were told they could sign up just by replying start so you didn't have to click anything. You just had to reply back to the SMS message. The third was to automatically enroll parents in the text message service and give them the option to opt out by replying stop. So this is what we call default default enrollment. Now, the really interesting thing is that Berman and Rogers asked the policymakers who designed the intervention how many parents they thought would sign up.

[26:34](https://youtu.be/8F71LYxf-Jg?t=1594) That's funny because like it's a really easy, low cost thing to do, yet hardly anyone gets anybody to do this sort of forecasting and sort of writing down the predictions. But it is incredibly helpful to us today. So the black bars that we're showing you right now indicate the estimated percent of how many how many of the what the policymakers guessed what they forecasted. Of how many people would sign up and you can see that like, you know, they think that the standard approach is going to be less effective than simplified and the automatic enrollment is going to be the most effective. They thought, you know, paying attention to sort of the bars that like, you know, it goes up, but it's, you know, roughly like one third would sign up if you just send out the standard text standard sign up notification. So that's like the thirty nine percent. Just under 50 percent would enroll to the simplified version of the SMS message, and then they thought one in three people would opt out.

[27:35](https://youtu.be/8F71LYxf-Jg?t=1655) OK, so we're going to ask you again, what percent do you think actually signed up? In each of these categories. So do you agree? Do you think that they got it wrong? Some of you are making it a little bit more a little bit more skewed. Is there good estimates? Yeah, Bonnie, I think, you know, that they probably got it wrong, but the question is, how much did they get it wrong? OK, so let's reveal. You can sort of see here, right, this is a lot. A policymaker is predicted that one third of people would sign up to the standard sign up notification, the experiment revealed that only one percent proactively signed up.

[28:45](https://youtu.be/8F71LYxf-Jg?t=1725) So they were thirty nine times more optimistic about this enrollment mechanism. I think it's worth noting that like people also didn't care enough to sign up, even when they got a simplified version saying just text text us back. You know, this is something that we're often doing. Right. We often have SMS messages coming and it's really easy, but still only eight percent. Policymakers did get it right, that defaults would work. They thought it was going to be the most effective, but the magnitudes are wrong. So they have the default automatic enrollment has a much stronger effect than people predicted. And if you think about it, this isn't surprising, right, we're all invested in programs and policies we have a stake in. And because we've poured so much of ourselves into it, it's hard to imagine a world where it's not important to other people as well. You know, it's a good idea, but no one else is interested.

[29:48](https://youtu.be/8F71LYxf-Jg?t=1788) And when we see the value of a policy or intervention, the illusion of similarity can make us make us think that other people will, too. So it deceives us. But. Put yourself in the shoes of the public, these parents, right? Your program, your intervention, your policy is just one of many that is being presented to them. And unlike you, they haven't lived, breathed, slept this one thing for their workday for the last six months. Okay, they may not therefore simply not behave in a way that you think we would consider rational. So think back to sort of Aaron's examples of people not signing up for free money. That doesn't make economic rational sense. Um, but because they're all the things, you know, they have a lot going on and there's lots of reasons and frictions why people that sort of get in the way of, of somebody following through, this doesn't make mean that policymakers are delusional. It's really about them simply not getting enough feedback or doing enough real world testing to be well calibrated, which is going to be the thing that we keep returning to.

[30:51](https://youtu.be/8F71LYxf-Jg?t=1851) So again, when you know, it's a good idea, but no one's interested, the illusion of similarity is going to make you think more people are going to value your thing, be the sort of idea of, of being off in your judgments. Okay. And then we're going to move on to the sort of this last stage, um, of executing. So this is when you're putting the project delivery in place. So, uh, you're over the sort of launch stage, you're into sort of the running. Um, and this is where we're going to zoom in on optimism bias. Okay. Optimism bias, the way that we define it is the tendency to consistently overestimate the likelihood of, of success. In other words, to be overconfident, um, Optimism bias, we often refer to in our work is sort of known as the mother of all cognitive biases. If you sort of remember in that original chart that Aaron showed all of the arrows that we told you not to pay attention to are basically leading to optimism bias.

[31:58](https://youtu.be/8F71LYxf-Jg?t=1918) So it's a, it's a really big one. It's a theme that shows up in a lot of behavioral science work. Okay. We're going to do another example here. Uh, and, um, there's a lot happening again in this chart. And we keep doing that to you. Apologies. It's just from research by Scheffler and Lauren and Lohan in 2017, and they did research with politicians from the national parliaments in Belgium, Israel, and Canada. And so what you see on the screen is each blue circle is a real national politician from one of those countries. It represents an individual. On the X axis, you're going to see something called, you see something called overconfidence index. Okay. There's a lot of different ways you could measure overconfidence. The way that they did it is by asking politicians if they think they will get reelected and compared that to polling data and previous election results. Does that make sense? So it's, it's an overconfidence in a particular instance here around the likelihood that they will get reelected and then sort of calibrating that against polling data and actual past results.

[33:08](https://youtu.be/8F71LYxf-Jg?t=1988) Okay. On the Y axis, you see something called a risk-taking score and that's measuring how willing the politicians were to endorse a risky policy in a hypothetical situation, just to note, this was done pre COVID, but sort of eerily, they were asked how they'd respond to a global pandemic, and they said, well, let's see, this was done pre COVID, but sort of eerily, they were asked how they'd respond to a global health pandemic originating from Asia. So ended up being reality. Now there are two interesting things that we see in this chart. So I'm gonna ask you, what's the first thing that you notice? Don't overthink it. Just see like, what, what is the pattern that you're seeing on screen, paying attention, hints to that diagonal line in the middle. Any guesses, anybody want to hop?

[34:19](https://youtu.be/8F71LYxf-Jg?t=2059) Yeah, Isabel, exactly right. Okay. So there's this correlation between overconfidence and risk-taking. It's not super surprising. What this graph is showing us is that the more overconfident you are, the more willing you are to endorse a risky solution. You might say, so what, we're just saying here that sort of personality matters. You might already know from your professional experience that the personality of senior leaders you interacted with is going to influence how, what information you present them. You might have already have a sense of the sort of options they're willing to go for and what they won't be. And fair enough. But there's another interesting thing about this graph, and I'm going to give you another hint, ignore the Y axis. So just look at that X axis and the overconfidence index.

[35:20](https://youtu.be/8F71LYxf-Jg?t=2120) What's the other thing that you notice in terms of pattern? Yeah, right, exactly. Yeah. So nearly everyone participated with overconfidence. So if you can squint, I don't know how big our screen is for you. You can literally see there is just one score of zero indicating that among everybody surveyed, they were the only one that was not overconfident. But after that person, there's this enormous gap of empty space from zero to 25, 30, and then this big cluster of everyone who was in the overconfident zone and not like a little bit overconfident, like quite overconfident.

[36:32](https://youtu.be/8F71LYxf-Jg?t=2192) And so sometimes when we present this, we get people say again, like, you know, what's so bad about this? What's so bad about being overconfident? Confidence is great. We want, you know, children to be confident, people to be confident, the desirable trait. What's the problem? So I'd encourage you to just reflect on the over portion of the overconfidence. I think we can all agree confidence is a fine thing. It's really the overconfidence that's the difference. It's this excessive confidence that's not really anchored to reality. And this is the part that sort of affects people's judgment. Does that make sense as a distinction? Yeah, it's, it's, you know, not that confident. We want people to be confident. We want people to speak up for themselves. We want people potentially to do the healthy risk-taking. All of that's really good. It's this sort of this, this overconfidence piece that's sort of not anchored to information.

[37:34](https://youtu.be/8F71LYxf-Jg?t=2254) Okay. Why do you guys think this might be? So why are these, all of these politicians, what would you say is the reason that's leading them to be overconfident? Gender is definitely related to some of these things. Yeah, exactly. Running for office is inherently something that takes a lot of confidence. There's selection bias, maybe potentially get rewarded for being overconfident. There's this confirmation bias, et cetera. So yeah, there could be lots of things that are happening in the system that potentially reward this. Again, a sort of theme that we're going to return to is often the lack of information and feedback. So just hold that in your mind.

[38:35](https://youtu.be/8F71LYxf-Jg?t=2315) It can be something that, that we'll, we'll come back to in a few, in a like two slides, so just hold. I will say that like, Oh, before we move on, I wanted to ask you, so that's about politicians. What about us? People who work in government, people who work in policy, design interventions, work in research, work in innovation, people who work with government. Do you think that like policy makers, civil servants, do we also have this problem? Yeah. It turns out like humans have this problem. So I will say, I'm asking for feedback. I do have some data on that. So I'll show that to you now and give all of us a reality check. So this is data that we've collected from over 500 UK civil servants, basically people who participated in past versions, like the longer versions of this course from 2018 to 2019.

[39:38](https://youtu.be/8F71LYxf-Jg?t=2378) So it's a little, maybe a little bit out of date, but not too long ago. And as I said before, there's many different ways to measure overconfidence. It can be sort of situation specific. The way that we did this was that we gave people simple trivia questions and asked them if they were true or false. So things like was Florence Nightingale born in Florence, true or false. If we had just stopped there, this would be a fairly pointless exercise, but the extra thing that we asked them to do was to answer, how confident are you in your answer? And then we compare two very simple metrics, how correct people are and how confident they are. And based on those results, we categorize them into one of the three categories you see on screen. Now you might remember at the very start of this course, Aaron asked you what you thought would be a good single metric to measure how good somebody was at making judgments.

[40:41](https://youtu.be/8F71LYxf-Jg?t=2441) And I'm going to reveal our answer. It's not the only answer, but it's the blue group on the screen called calibration. We would want to know whether a person is well calibrated. What does that mean? So break it down. It's not an intuitive concept. It doesn't come up often in daily life. But basically it means that their confidence and their correctness are tracking like very closely to one another. So the more confident that they are, that they're right, the more correct they indeed turn out to be. That seems like it should be a really low bar, but it's actually surprisingly rare. So only about one in five people in our sample ended up being in that group. And so returning to some of the questions I see in the chat about what's over confidence, it's pretty simple. Again, it's people who are more confident than they are correct. So even if they tend to be, say that they're 90% confident in their answer, they often end up being wrong.

[41:47](https://youtu.be/8F71LYxf-Jg?t=2507) And then actually, again, seems like a low bar to reach. It's again, super common among this group of 500 civil servants. We found 70% of them were overconfident. And then there's this third group that we'd actually don't think about very much, but maybe you guys are sort of already getting there in the chat a bit, and that's being underconfident. And what does that mean? It means you're being correct. You're correct a lot, but you don't back yourself. So you're more correct than you are confident. It's very humble. If not particularly, could be more effective. Right. And I'll say like, even as a company who's, we specialize in cognitive biases, we care about these things a lot. We struggle with this. And what you really want as an organization is to be developing kind of a portfolio of confidence, right? On average, bit actually ends up being well calibrated. And really the only reason is because we have a greater proportion of people that are underconfident than most organizations.

[42:48](https://youtu.be/8F71LYxf-Jg?t=2568) And they help balance out the overconfident people. I think that I, if I had to judge, I think that's because we have a lot of researchers who look at really depressing data all the time. And so that probably is, is why our, our portfolio mix of judgment casters is a little bit more on balance on average, but in general, still, we find that really only 20% of people are well calibrated at bit. So again, let me say that again, calibration. It means confidence and correctness are tracking each other closely. When you say you're 80% sure you actually does, that does end up to be right, 80% of the time. I'm going to keep emphasizing that. But I'm going to, we're going to go to another example to help make this even a little bit more clear. Our argument is that government should try to be well calibrated. So here's a final example.

[43:49](https://youtu.be/8F71LYxf-Jg?t=2629) I'm going to show you two graphs. This first graph is a study of nine doctors, so small, but they made 1,500 assessments about whether or not a patient had pneumonia based on observed symptoms, and then the doctors are asked, do you think that this patient has pneumonia and then how sure are you about, about that diagnosis? And then that was compared to a patient who had pneumonia. And then that was compared with an objective indicator, which was radiographically assigned cases of pneumonia. They actually checked the tests. So perfect diagonal line here is perfect calibration. The more sure you are, the more right you are. And you'll see here, that's like a little bit depressing that doctors are more sure that you have pneumonia than you actually do, right?

[44:50](https://youtu.be/8F71LYxf-Jg?t=2690) Doctors are, when doctors are 80% sure their patients have pneumonia, this turns out to be true only 20% of the time. This is very bad calibration. I don't mean to scare everybody. Again, nine doctors. I want to compare that with the next graph showing weather forecasters, predicting rain and the objective measure of how much it rains. What do you guys notice about this graph? Basically, weather forecasters are close to perfectly calibrated. When weather forecasters say they're 80% sure it's going to rain, that turns out to be true 80% of the time. Despite what might be your actual experience of the weather forecast often being wrong. This is measured by data, maybe for myself.

[45:53](https://youtu.be/8F71LYxf-Jg?t=2753) The other bit of key information in this graph is the small numbers that you probably can't see on your screen. On the weather forecasters, you can see the next eight points is the number of forecasts they're making. They're making thousands of forecasts. This ends up being a really key point. It's not that weather forecasters are more intelligent than doctors. It's not true. That's not the argument that we're making. It's that they work in an environment where they get frequent and rapid feedback. What evidence shows is that this is really key. If you predict it's going to rain, you're going to know in a few hours whether you were right. You're going to make that judgment tons and tons of times. That is not how things like diagnoses about pneumonia or how policies are made.

[46:53](https://youtu.be/8F71LYxf-Jg?t=2813) Maybe the doctor gives you a prescription, it works, but you often don't call back a doctor and say, hey, that worked great, thanks so much. You call back when there's something wrong. The policy mechanism ends up being the timelines and the feedback mechanisms are even worse. You try a policy and then you have to wait a few years to see if it works. If you were able to get that data in the first place, you have an amount of money to do an evaluation. It's often really likely that everybody's moved on. You may never ever know what the outcome was. Think back to that example of the schools from a few slides ago. The policymakers worked on that intervention for years. In the end, they only got a handful of data points telling them how well it worked. In 10 years of doing work like that, they might only learn from two to three opportunities in their experience to really hone their expertise in terms of predicting how their policies are going to turn out and then getting the evidence about that.

[47:55](https://youtu.be/8F71LYxf-Jg?t=2875) Again, what's the answer here? We're not saying that doctors are stupid and weather forecasters are smart. We're not saying that one group suffers from biases more than another. There's just a really big difference in the environments in which these two groups of people are working, the amount of feedback they get about their predictions and forecasts. We've been talking about cognitive biases leading to worse decision-making, but a key thing is that cognitive biases really thrive in unstructured decision-making environments. It's much harder for them to persist in a consequential way if every time you make a biased, poorly calibrated decision, you quickly get a reality check and learn that that was the wrong thing to do. Okay, that was a lot. We threw a lot of cognitive concepts at you, a bunch of different research. I want to just pause here for any questions, comments in the chat.

[48:58](https://youtu.be/8F71LYxf-Jg?t=2938) I'm going to do a second.

**Erin Britton**

[49:00](https://youtu.be/8F71LYxf-Jg?t=2940) I don't know, Erin, if you've seen any come in that we should address. I saw two just about the overconfidence and underconfidence. One person asked, did underconfidence correlate with risk-averse behavior or less risky decisions? That's a really good question. I don't know that the paper specifically addressed that, but yes, that is how the correlation would work. If you're underconfident, you potentially, yes, become risk-averse. Again, I think what we would want to say is that there are some situations that it will make sense to be more risk-averse. They have big, large consequences. Lots of money being spent can really backfire where you're going to want to be really risk-averse about something. Then there are situations in which, no, taking a little bit of risk is actually healthy. Really, what you, again, want to be aiming for is to be calibrated.

[50:05](https://youtu.be/8F71LYxf-Jg?t=3005) Having your confidence and your risk tolerance and your judgment on how right you end up being to all be in line with one another.

**Emily Cardon**

[50:16](https://youtu.be/8F71LYxf-Jg?t=3016) Then we'll take two more questions, one from Karen and one from Erwin, before we move on. Karen was asking, do demographics play into this? Are there some groups that are just naturally more over- or underconfident? We could get back to you on that. From what I know, I don't know, Erin, if you know more. I think the answer is yes, but that distribution persists in really every characteristic. It's not that there are no overconfident women. I do think gender plays a role, but within women, there can be that same distribution. People are way, way more likely to be overconfident than they are to be calibrated or underconfident. Then we'll just do one more question from Erwin, if you have an answer to this. What should the MDs have done to improve their predictions? Yes, this is a really good one. It's a really hard environment.

[51:17](https://youtu.be/8F71LYxf-Jg?t=3077) How much they're able to learn and grow from that is just a challenging environment. Particularly doctors, depending on how often they're diagnosing and how often you come across something, there's rare diseases. It can be really hard to get calibrated about things like that. Ideally, they're able to aggregate up a bunch of the doctors' decisions so that you're not relying on just one doctor's experience. Be able to map these patterns out. You can think about, in the US at least, some of these healthcare systems. You'd be able to track confidence levels or diagnosis and what ends up being the reality of what the diagnosis ends up being at the end of the day. Then being able to give that feedback back to doctors. That is the feedback loop is often what is missing in organizations. It's really hard to get right.

[52:17](https://youtu.be/8F71LYxf-Jg?t=3137) The more that you can get that feedback loop to be individualized so that you're giving that individual doctor how well they're doing against peers, the better results you're going to get. For your decisions, the more personalized, the better. There's just limits to the data that we're able to get and the time for that to be able to happen. One thing that you could do in the healthcare system is maybe focus on diagnoses that are the most common and are able to be diagnosed most quickly so that you can get that feedback loop going. Hopefully, the idea would be that spills over into other areas of diagnosis and decision making. Anybody who's worked in education knows. This is why test taking and feedback are just so critical to learning. Calibration, I will tell you, is something that can be learned. This is not a personality level thing. It is not immutable.

[53:17](https://youtu.be/8F71LYxf-Jg?t=3197) It is a skill and you can get better at it. We're not going to go into individual level calibration training. People Google this. There's lots of institutes actually and specialized organizations out there that do calibration training specifically and around forecasting. If you're interested in that, I highly recommend. We're going to move quickly through a couple solutions. We don't want to leave you with this depressing, maybe a little bit destabilizing idea that all of our judgments are flawed. There are things that you can do about it. As I was saying, overconfidence, all of these cognitive biases really thrive in unstructured environments. What we want to do is put some structure on them. A recap on where we've come. We've seen how cognitive biases can impact policy outcomes. Hopefully, you guys are convinced that the cognitive biases are important.

[54:19](https://youtu.be/8F71LYxf-Jg?t=3259) We know that simply raising awareness of these biases is unlikely to be sufficient to change behavior. If our example about BITS, again, we specialize in these things. That does not mean that Erin and I do not suffer from these in our own lives. Interestingly, you can also name that you're falling victim to it and still can have it difficult to overcome. People find it difficult to notice and correct their own biases. This is what I'm getting at. Even if you do notice it, it can also be very difficult to correct your own biases. There are lots of tools you can use at a team level to improve your judgment and to help your team make better calibrated decisions that are less affected by biases. This is where, again, sometimes working in a group, having that portfolio of people participating can lead us to the better decision-making at the end of that process than having to do it by ourselves.

[55:24](https://youtu.be/8F71LYxf-Jg?t=3324) We're going to talk specifically about team-level de-biasing and three different solutions. Think groups, red teams, and pre-mortems. Some of you might be familiar with some of these already. First, I want to ask you, why might group brainstorming lead to bad decision-making? Why might group brainstorming lead to bad decision-making? No. Yeah, exactly. Groupthink is a big one. That's a big one. Yes, fear of sharing ideas that you screen yourself. People go with whoever is the loudest voice. Lack of expertise by everybody. Underconfidence here, again.

[56:24](https://youtu.be/8F71LYxf-Jg?t=3384) Just comfortable sharing, being different. People participate in different ways in groups. Some people like talking things out. Some people like to think on their own. Power dynamics, that's what I was just going to get to, Bonnie. Who has the most standing in a group, whether that's by hierarchy or just social influence? All of these can play important roles in leading to bad decision-making when we're brainstorming in groups. Thinkgroups is a way to ensure that ideas get equal hearing. I want to say that group dissections are central to policymaking. They're really important for generating ideas and solving problems. The evidence shows that they can make some decision biases worse. They can hinder creativity, and they can lower sufficient challenge of ideas.

[57:24](https://youtu.be/8F71LYxf-Jg?t=3444) All of the things that you guys are talking about are all things that would fall under what we call group reinforcement dynamics. You focus on what people already know. Extreme views and positions get magnified. Initial, like what happened first, whoever volunteered first, can strongly sway group opinion. There can be this reinforcement of hierarchies. Then, just in general, lack of opportunity for more junior staff or acquired staff, less confident staff, to not voice their opinions and objections. I think all of you probably can come up with examples where some of the most innovative and refreshing ideas have come from new or junior people or people who weren't involved in the problem already. They have a fresh take. Think groups are a tool that you can use during deliberation to ensure that all ideas get equal hearing. It's definitely a play on the term group think that many of you already volunteered. What is key here is that they are used to provide anonymous ideas and anonymous feedback on those ideas.

[58:28](https://youtu.be/8F71LYxf-Jg?t=3508) They're different to brainstorming where some of you might be very, you know, the sort of standard way is you have a senior person sort of up on a board or writing ideas or at least directing the flow of conversation. In a think group, all your ideas are anonymous and then can be assessed based on their merit rather than the person who said them. What does this practically mean? This is something that we use a bit all the time, where if you use Google Docs, you can do this very simply. There's lots of other tools, other systems that can do this too. But the idea is that you have everybody in the group enter that document anonymously or the tool anonymously. And you have people write down the ideas, but you can't see who is writing those ideas. And so instead of giving credence the ideas just that came from senior management, the ideas are assessed based on their merit. And there's a certain freedom to having like the space to contribute ideas throughout the document.

[59:28](https://youtu.be/8F71LYxf-Jg?t=3568) We found it's like a very, it's very liberating. And it often means that you can critique ideas in a way that may not be possible if you sort of are worried about offending or angering somebody. We found practically that it's still useful to give a little bit of structure. So not just like having an empty blank page. Often we sort of have questions for people to answer and to sort of structure their thinking. Often you still need everybody contributing at once. So you still like arrange a meeting or sort of a virtual meeting. This is a virtual call where people are together. You can introduce the topic at the beginning, take some questions to get people up to speed, and then you actually just sort of spend sort of quiet time, often like 10, 15 minutes, just sort of brainstorming individually. We often ask people to just put all their ideas at once, similar to how you would do a normal brainstorming, and then let people go through and either critique, add on, or sort of plus one all of those ideas. So it's really simple concept.

[1:00:28](https://youtu.be/8F71LYxf-Jg?t=3628) It's kind of, you can use like really any technology. The biggest thing is just making the sort of contribution and critiquing anonymous. Yeah, so we can structure or provide an example. What's a good one? So sometimes you sort of might be thinking about an intervention, and you need to think about who the intervention is for, what a different, like how you're going to reach that population, what are sort of different formats for the idea. And so you could sort of think about like a design brief, and you could structure a think group with those sort of headings, and you sort of have people think about, okay, who would be all the people that we would want to reach and talk to? Have that people brainstorm that, and then they follow in the sort of section right after that around, okay, like what are the different like modes that we could reach them? Is it an SMS? We could reach them by email. We could reach them by going to our door, et cetera.

[1:01:29](https://youtu.be/8F71LYxf-Jg?t=3689) This is a very basic example, but just to give you a sort of flavor, there's a lot of freedom to it, and there's not a science. Often sometimes you end up with structures that didn't end up work. Nobody answered that one question, and that's okay too. Okay, moving quickly to sort of red teams. Has anybody done this before? If you can put your name in the chat, or just like sort of say, yes, that we've done this. This often has different names, but it really comes out of a sort of military context where you have a specific group of people whose specific job it is to expose weaknesses in your defenses or strategies, as often as another big thing in business where you may have heard it in like sort of cybersecurity, where they'll purposely invite hackers to try and expose flaws in their cybersecurity infrastructure. So it's like somebody who's like going in and trying to exploit and find any point of weakness in the system or the process.

[1:02:31](https://youtu.be/8F71LYxf-Jg?t=3751) In the context of policymaking, you can specifically direct the red team to poke holes in your policy or your intervention. So they should be thinking about ways in which the policy could be at odds with other strategic priorities or instances in which it may not be well received by stakeholders or may not have the compliance rate that's expected. So it's somebody who you're specifically empowering to say like, give me all the feedback, right? Try and brainstorm all the different ways in which I could be wrong. Being in a red team necessitates a mindset shift in and of itself. So the red team is empowered and charged with constantly looking for errors. It's literally their responsibility to do this. So you need to empower that red team to do that. It's their duty to formally be impolite, basically, otherwise they're not going to be doing their job. And this is a really powerful tool because it gives people license to be critical.

[1:03:33](https://youtu.be/8F71LYxf-Jg?t=3813) If you remember, one of the issues with brainstorming that we talked about is people don't want to upset the group dynamic, rock the consensus, offend people personally. They're worried about senior members of the team disagreeing with them. And so being empowered with this role makes you empowered to do that and we're willing to overcome that sort of self-censorship and just look at the problem and the issue with a slightly different mindset. Practically, some of you may not be surprised by this, we see red teams work better when those members are seen as part of the in-group. So those are the sort of people who aren't in the red team see and know that the red team is on their side. So that the criticisms are seen as being the team's best interest, they're more likely to be well-received. So this is not a thing where you're asking auditors to come in and people who are you sort of know that you're kind of at odds and you're already putting up for defenses.

[1:04:34](https://youtu.be/8F71LYxf-Jg?t=3874) It's sort of you want to empower people who are already like very well-liked, well-respected to be the people who do this. Practically, size can vary depending on the situation. There's no like one rule about this. You could also give people sort of different hats, different ways to sort of things that they're supposed to be looking out for. You really want to rely on clear instructions, you know, it goes without saying. And another thing about being in group, you probably want the seniority to match whoever to match yours or the most senior person who's going to be making decisions, right? You just want to make sure they have the right power and they're not going to be afraid to say something. Okay, last one here, running pre-mortems to fail in advance. Again, I don't know how much people are familiar with post-mortems, but that's basically when you try to understand what went wrong after an event has happened. A pre-mortem does that in advance. So in essence, before you've launched a program or you're just at the start of a project, you fast forward in the future and assume it's failed, sort of an imaginative task.

[1:05:47](https://youtu.be/8F71LYxf-Jg?t=3947) And you sort of now try and work out what could have prevented that from going wrong. The framing is very important that this has already gone wrong. So despite all of our plans that we have in the moment, the contingency plans, something went wrong and you want to brainstorm how did that happen? And then how do you work backwards from making sure that you sort of have plans in place to address that? So that helps basically by its very framing, overcome overconfidence by us, right? We're saying it already failed. Practical advice here is you want to get very granular on the failure. You want to be as specific as possible. I have colleagues who take this as a fiction writing task and develop the most detailed scripts of everything that possibly has gone wrong. They are fascinating and very fun. So I encourage all of you to use your creativity when you do this, be in that headspace.

[1:06:48](https://youtu.be/8F71LYxf-Jg?t=4008) Previous research has shown that imagining an event has already occurred increases the ability to correctly identify reasons for future outcomes by 30%. So it just like is a really easy way to sort of like eat away at that overconfidence. And then it gives your team insight into priorities and successes, help uncover hidden assumptions, potential pitfalls. We use this all the time. So I'm going to put up, or Erin's going to put up rather, a pre-mortem canvas. We weren't going to have time to do the activity today, but we will share this with you. And then we encourage you to run pre-mortems with your team on any new or upcoming projects or if you're starting the beginning of a very important phase around implementation, where this could be useful. The last slide here is sort of, again, on sort of practicalities. Again, similar to the think groups, you want to start, arrange a meeting to discuss the hypothetical policy failure at bit.

[1:07:49](https://youtu.be/8F71LYxf-Jg?t=4069) I will say that we combine the think group and the pre-mortem very often. And so you do the pre-mortem in think group style, where people are putting these things in anonymously. It's good to leave enough time to do this. So like one to three hours, depending on the complexity of the topic. But I will say that even a short 15-minute pre-mortem is better than no pre-mortem. So getting all of those ideas down, I think, always helps better planning. Inviting a diverse cast list is super key. So you want to make sure that you have representatives from sort of all the teams or groups that are going to be involved in the policy or program delivery. So remembering even small details, I'm sure all of you have examples of frontline staff or IT issues that have gone wrong, or having somebody from that group at the table very early would have helped solve a problem. They are key to making the difference between project success and failure. And then in the meeting, you want to provide background information, sort of state the sort of already known risks that you've already identified, and then really charge people with sort of thinking about sort of non-obvious things that could really wreck a project.

[1:08:59](https://youtu.be/8F71LYxf-Jg?t=4139) Okay. So a drive-by of all three potential solutions for you to come back to, over to you, Taryn.

**Erin Britton**

[1:09:10](https://youtu.be/8F71LYxf-Jg?t=4150) Great. Thanks so much, Emily. All right. So I'm going to quickly kind of wrap us up here, as we want to leave time for questions. We know there's a bunch of questions coming in on the chat. So what did we cover in this very quick session today? We talked about the problem of cognitive biases and how they can lead to worse outcomes in policymaking and decision-making. And then we talked a little bit about some of the solutions that Emily shared, red teams, think groups, and premortems, that hopefully you can take away and use in your teams now that you're aware of these cognitive biases to help you mitigate their effects in the policies that you're working on. So just to kind of wrap us up, we'd love to hear in the chat, what are some of your main takeaways from this session, and how might you apply these lessons in your work? And then, of course, also take the opportunity to throw any questions that you have in there for us. Give you a minute here.

[1:10:15](https://youtu.be/8F71LYxf-Jg?t=4215) A few votes for red teams, wonderful. That's one of the takeaways, that you can mitigate the effect of these biases. We aren't totally in their sway. Some fans of forecasting and that speculative or future visioning work. Problem of overconfidence, awesome, yeah, these are great takeaways. Wonderful. Calibration, yes, well, being well calibrated as a measure for decision-making. Yeah, I'm glad you found that interesting.

[1:11:15](https://youtu.be/8F71LYxf-Jg?t=4275) Wonderful, and it sounds like some of you are already implementing some of these strategies, and now you have strong science behind them in your teams, that's great. So some of our takeaways, what a well-calibrated government looks like. Everything that you're saying in the chat is spot on. A well-calibrated government is rarely surprised by its policy outcomes. That doesn't mean it doesn't take risks, it just knows the risks going in, and isn't surprised when something fails, and hopefully also has greater chances of success. It uses premortems and red teams to create effective mitigation strategies, those non-obvious project risks, and it has an accurate understanding of its own knowledge, and it's neither overconfident nor underconfident. So a great example of this, if you're a fan of 538, they are really well calibrated. So when they say something's going to happen 40% of the time, they actually check up on it, and they have a great forecasting calibration section that says it actually does happen 39% of the time.

[1:12:21](https://youtu.be/8F71LYxf-Jg?t=4341) So this, as Emily said, is our taster session on de-biasing decision-making in government. If you're really interested in this topic and you want to learn more, we'll share the link afterwards to our behavioral government report, which dives into a variety of biases that happen in the policymaking process. And then we do offer a longer version of this course, a half-day course for teams, that you can bring this session in an expanded format into your organization or to your teams, where we cover intro to cognitive biases, we practice applying some of these team-level de-biasing tools like Red Teams and premortems, and we do a whole module on forecasting and how to structurally de-bias and better calibrate your organization. So if that's of interest to you, definitely reach out to us, or you can get us at training at bi.team. So if I'm going to leave you with two words, that is to stay calibrated. Thanks so much for joining, and we'll hand it over to Hannah, who I think is going to pitch us some questions in the time that we have left.

**Bonnie McGilpin**

[1:13:22](https://youtu.be/8F71LYxf-Jg?t=4402) Thank you so much, Erin and Emily, that was fantastic. I'm just going to take us off recording.

* * *

Original page: https://innovate-us.org/debiasing-decision-making-in-government · Licence: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org