---
id: "workshop:evaluating-effectiveness-and-efficiency-through-benefit-cost-cost-effectiveness-and-risk-analyses"
type: "InnovateUS workshop"
title: "Evaluating Effectiveness and Efficiency through Benefit-Cost, Cost-Effectiveness, and Risk Analyses"
url: "https://innovate-us.org/evaluating-effectiveness-and-efficiency-through-benefit-cost-cost-effectiveness-and-risk-analyses"
date: "2024-06-27"
duration: "90 minutes"
language: "English"
instructors: ["Deborah Stine"]
recording: "https://www.youtube.com/watch?v=atrVucJRtrU"
transcript: "automatic, from the recording"
source: "InnovateUS (innovate-us.org)"
licence: "CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org"
---

# Evaluating Effectiveness and Efficiency through Benefit-Cost, Cost-Effectiveness, and Risk Analyses

InnovateUS workshop · 27 June 2024 · 90 minutes

Join us for a 90-minute virtual training with Dr. Deborah Stine, Founder of the Science & Technology Policy Academy, where participants will learn about three fundamental quantitative techniques that are essential for evaluating the effectiveness, efficiency, and equity of public policies: benefit-cost analysis (BCA), cost-effectiveness analysis (CEA), and risk analysis (RA)

## Learning goals

- Learn about three fundamental quantitative techniques that are essential for evaluating the effectiveness, efficiency, and equity of public policies: benefit-cost analysis (BCA), cost-effectiveness analysis (CEA), and risk analysis (RA)
- Learn the steps and methodologies involved in conducting BCA, CEA, and RA, including differentiating between high-quality and subpar analyses
- Learn how to identify sources of these analyses for use in your own work
- Develop skills to effectively communicate the results of BCA, CEA, and RA, linking these outcomes to policy decisions

## Speakers

- **Deborah Stine**, Senior Fellow, Innovate US

Recording: https://www.youtube.com/watch?v=atrVucJRtrU

## Transcript

Automatic transcript of the recording: names and terms can be misheard, and the times can be a few seconds off. Each time links to that moment of the video. The transcript alone: https://innovate-us.org/evaluating-effectiveness-and-efficiency-through-benefit-cost-cost-effectiveness-and-risk-analyses.txt (plain text) and https://innovate-us.org/evaluating-effectiveness-and-efficiency-through-benefit-cost-cost-effectiveness-and-risk-analyses.json (every segment with its start and end second).

**Unknown speaker**

[0:00](https://youtu.be/atrVucJRtrU?t=0) And I'll hand it over to you, Debra.

**Deborah Stine**

[0:03](https://youtu.be/atrVucJRtrU?t=3) Okay. So it's a, oops, here we go. Sorry, I'm waiting. Ah, there we go. Uh, you can see that right, Hannah?

**Unknown speaker**

[0:22](https://youtu.be/atrVucJRtrU?t=22) Yeah, we can still see the ones on the side. Um, it's not in full.

**Deborah Stine**

[0:29](https://youtu.be/atrVucJRtrU?t=29) Okay. So let me try this again. Oh, there we go. Yeah. Okay. It's there. You know, I just have to close the things happening on my screen. Okay, great. So hello everybody. Uh, my name is Debbie Stein. It's really great to be back here again. Um, you know, I should add to the introduction that I did work for state government for five years for the state of Texas, um, for the air pollution, um, agency. This is when I, as I said, I was a baby engineer and, um, and I've, I've come a long way since then. Uh, but, uh, you know, I, uh, I always, uh, understand or really understand the importance of, you know, sort of state and local government. So it's really fun for me to, to come and speak to this, uh, to this audience. So today we're going to talk about, uh, evaluating effectiveness and efficiency through benefit costs, cost effectiveness, and, uh, risk analysis.

[1:30](https://youtu.be/atrVucJRtrU?t=90) Um, all of these, I will say are relatively, um, complicated enough topics that back in PhD school, like I had a whole semester on some of these things. However, I have capsulated the key principles down for you in a one hour presentation. So, uh, feel free to, um, you know, put, uh, any questions and so forth in chat. I will stop briefly after each method, just to, you know, to make sure that you don't have any, um, like burning questions as, as Hannah mentioned about that particular, uh, method. Uh, the other thing that you'll see is I have sporadically put into chat, um, information on getting a free copy of a PDF, a fillable PDF version of my book. And, uh, you know, maybe Hannah, every once in a while, I could put that in again for anybody who's sort of late, uh, late arriving. Uh, so with that, let us, uh, get started.

[2:32](https://youtu.be/atrVucJRtrU?t=152) Okay. So this is one of my favorite cartoons because, uh, uh, it talks about the importance of, of being very clear in your analysis. So you had these two professors and, you know, you'd have all this data and information and things like that. And then all of a sudden you get to this conclusion, uh, but what happens in between, and, uh, it's not really a miracle. It's just a lot of hard work. And I'm going to basically talk about that in between, uh, area when we're talking about quantitative, uh, aspects of pulse analysis. So the goal of this workshop is to, uh, learn how BCA that's benefit cost analysis, CEA cost effectiveness analysis, CUA cost utility analysis, and RA risk analysis are used for evaluating the effectiveness, efficiency, and equity of current and proposed public policies, um, to, uh, identify sources of these analysis that you can use in your work and develop skills to effectively, uh, communicate the results of these analysis, um, linking outcomes to public decision-making.

[3:51](https://youtu.be/atrVucJRtrU?t=231) So this is a brief, uh, overview, um, of these, uh, these three core methods. So in benefit cost analysis, what you're determining is you're looking at different alternative interventions that have the same objective. So a good example of that is a bridge. So you can make a decision, okay, well, I can either build this bridge out of this material and it lasts 40 years. Uh, but it'll cost more, or I can bridge, uh, do this bridge and it's cheaper, but only lasts 30 years, which one should I do? And so cost benefit analysis helps you make those kinds of decisions. On the other hand, cost, so everything is in dollars, you know, the dollars in cost and dollars in benefits. When you come to cost effectiveness analysis, you instead are, you still have costs as dollars, but your sort of denominator, the, the, uh, like what you're looking at is usually some sort of desired outcome, for example, something like, uh, less cancer, less, um, uh, a decrease in colon cancer or something.

[5:10](https://youtu.be/atrVucJRtrU?t=310) Uh, less cases of diabetes, these things. So it's primarily used in the health arena and it's, it's used because, you know, it's very hard to put a dollar value on that. So when it comes to, to humans and human lives, uh, oftentimes people prefer cost effective analysis over cost benefit analysis, but then it comes to cost, um, and with cost utility analysis, uh, what you're looking at is, is the quality of what's occurring. So for example, you can live longer, uh, if maybe a certain health intervention is done, but you know, if you, uh, you know, have Alzheimer's or something and you have no idea what's going on, what is that quality of life to you? Maybe. Um, and that's going to vary by person in terms of what they think about what they want their quality of life to be.

[6:12](https://youtu.be/atrVucJRtrU?t=372) And so here you go and you get people's opinions and things like that. And that's called cost utility analysis. You're looking at, at, you know, not just living longer, but the quality of the life as you live longer. So, um, as I mentioned in terms of, uh, if we're going back to the next chart about how it's measured and calculated, uh, so cost and benefits, um, are in dollars, but the, the key thing to remember with benefits is it is that it is often avoided costs because that is what can get confusing for people, you know, so, um, because of this intervention, we're going to have less, um, kids going to, uh, the, um, the emergency room and that is going, and that's by, so by taking this intervention, we're going to have less kids going to the emergency room, so that would be less cost. Right. And those lower costs are the benefit.

[7:13](https://youtu.be/atrVucJRtrU?t=433) So it can get a little bit confusing because we are talking lots of different dollars, uh, that are around, but hopefully by the end of the show, you'll sort of understand this, but, um, sometimes that can be hard to do now. The other thing to keep in mind is that there's some benefits that are very hard to quantify. For example, uh, haze across the Grand Canyon. This was a big issue a number of years ago. And of course you could say, oh, we're going to have less tourism or something like that for Grand Canyon. Um, and there's haze across it, but that's just kind of like, not the way people feel about these sort of, you know, like this great, um, natural, uh, phenomenon. And so it's hard to quantify. And so there are rules or there's actually an executive order, uh, called one, two, two nine one, which sets, uh, across some of the principles of benefit cost analysis. And one aspect of it is you can, uh, include as part of your required analysis, these, uh, benefits, uh, and costs, which are not quantifiable, not easily quantifiable.

[8:21](https://youtu.be/atrVucJRtrU?t=501) Um, then, uh, when you're looking at, uh, costs in sort of in cost effectiveness analysis, you know, you're looking at what's going to sort of get the best bang for the buck. So, you know, we're looking at, okay, well, the, the measurement is averted visits to the, uh, emergency room. And then, as I sort of mentioned for cost utility analysis, you're looking at things like years of life, quality, just of life years, disability, just of life years, and generally health utility. That is, you know, like the quality of health, um, that you will have. Uh, so here's some examples. So, uh, the, uh, world health organization, uh, looked at the cost of improving care for depression and anxiety in 36 countries. Uh, the total cost of improving care was 147 billion. Benefits were about 500 billion. So therefore the benefits, um, uh, the benefits outweigh the cost.

[9:25](https://youtu.be/atrVucJRtrU?t=565) Um, and so therefore it makes sense to undertake that activity, at least based on that one fact. There was other factors and I'll talk about that in a second, but basically still in that aspect of it, you're going to save money. Uh, for cost effectiveness analysis, there was a, um, multi-site managed care, health managed care health plan in Wisconsin. Uh, they wanted to improve a screening for breast and cervical cancer for low-income women by sending a reminder letter, uh, from the primary physicians. They found that the women that received the intervention were four times as likely to be screened. So the cost effectiveness, um, of that was about $29 per screening test. Um, and then with cost, uh, utility analysis, there was a study that was in Texas that looked at, uh, community-based programs to improve healthy living among young adults, and it showed that there was somewhere between $1,300, $1,400, um, that was the cost per, uh, quality adjusted life years that were gained and the way cost utility analysis works is that there is a standard number that is determined in this case that was 50,000, and then that number is sort of well, um, um, well below that.

[10:46](https://youtu.be/atrVucJRtrU?t=646) So you would proceed with the action. So these analytical methods help, uh, policymakers by, uh, providing them with informed decision-making, not just kind of like shoot from the hip kinds of things, transparency, accountability, allocating resources, make comparisons. Um, they should also, oops, sorry. They should also talk about risk and uncertainty identification, um, looking at economic efficiency, what's going to get the best bang for the subtle buck. Uh, they can take, uh, into account long-term perspectives like over decades, they take account social equity and policy, um, improvements, but there are also challenges like there's some people who do not like these quantitative methods, uh, for some people just don't feel comfortable quantifying benefits and costs, uh, oftentimes we're lacking, um, the data that we'd like to have to make these decisions.

[11:46](https://youtu.be/atrVucJRtrU?t=706) Uh, time horizon discounted can be a big issue. Uh, if you've been around as long as I have, you know, that, you know, like the interest rates here seem so low, um, compared to like when I was like buying my first house where it was like nine, 10, 12%, things like that. That makes it challenging because they, you know, over decades, those things can change. Uh, there's a fair amount of subjectivity and bias, uncertainty, um, and, uh, and risk analysis and all these things. There's comparability assumptions, you know, social environmental considerations. There's a lot of concern about policy and political influence. And of course there's like ethical and cultural factors. So in general, all these methods I'm talking about are somewhat controversial. Uh, but you know, as somebody who really believes in evidence-based policy, I'd rather have some information than no information. Uh, but, uh, it is something to consider. Oh, now here's a, so now I'm going to talk a little bit about some of the introductory material in my book. So again, I put a PDF, um, a link in there, which again, maybe Hannah can just zap that in again for people to get it so you can, um, anybody goes to my workshops, you'd get a free version of this and it has like a lot more details about the stuff I'm going to tell you.

[13:03](https://youtu.be/atrVucJRtrU?t=783) So in policy analysis, um, you know, these are the sort of the seven steps. And what we're focusing on here is step five, which is analyzing the efficiency of the status quo and policy options. Or if you're doing program evaluation and you're looking at the past, you know, like in terms of, um, you know, has this program paid off or not? That's what you're looking at is you're looking at the efficiency, you know, of those, uh, of those options. Um, but you're also looking at efficiency in the context of what I call the four E's. Um, and so it is not just the, um, cost that people are looking at when you're a policymaker, but you're looking at effectiveness, which is the degree to which it's sort of goals achieved equity, which is the fairness of the policy and then ease of the political acceptability, which is not just, you know, like, it's not as much from a politician standpoint, but it's more the key stakeholders are, they're going to support the policy, uh, or they're going to oppose a policy.

[14:05](https://youtu.be/atrVucJRtrU?t=845) So it's important to understand that efficiency and all these methods, benefit costs and stuff with, I just one element, um, of a number in how policymakers make decisions. So here's an example of a program that's in Idaho. It's called the fit and fall proof program. Uh, so this is a 12 week, no cost system program that includes this balance exercise, strength training, and so forth, uh, for effectiveness. Uh, the studies that done on this program found it reduced the risk of falls by 40%. Um, in terms of efficiency, it saves a dollar 14 for every dollar spent. Um, the way they determine the benefit was the reductions in medical costs associated with falls. And then they also added in non-monetary benefits, uh, which is improved quality of life. Life and Reduce Dependency on Others. So this is an example, you know, very nice, easy to understand version of this. And now I'm going to get to the, you know, more complicated stuff.

[15:15](https://youtu.be/atrVucJRtrU?t=915) So benefit-cost analysis is fundamentally the net benefit which is determined by subtracting incremental cost from incremental benefits. And those increments are basically relative to what is already being spent. So, you know, the question is how much more can we get for that societal dollar? And that is what net benefits is about. So states use cost-benefit analysis in a number of areas. This is something that was done a number of years ago by the Pew-McArthur First Results Initiative. The biggest one you'll see here at least on this list is economic development, health care, environment, transportation, justice, education, regulation, and housing. Now this list would be probably quite different, you know, like when it comes to the federal government. It's primarily used for environmental aspects of it.

[16:21](https://youtu.be/atrVucJRtrU?t=981) And you can actually go to the Office of Management Budget and you can see like what regulatory decisions have the benefit, has the benefit-cost analysis been submitted when they're making regulatory decisions. But it is challenging. And the biggest challenge that states face in doing this is just basically resource limitations and data limitations. And that definitely, the resource limitations is does the agency have people on its staff, the state agency, that are economists that can do benefit-cost analysis? Some do, some don't. But that's really fundamentally the first issues is just like people. And then of course you need data, you know, sort of quantitative data to do these things. But that being said, if you go to like Google Scholar and if you type in benefit-cost analysis for just about any topic you can imagine, you will find benefit-cost analysis that have been conducted all over the world. And like I said, there's going to be uncertainty in it, but it is often a good starting point for when you're doing your analysis.

[17:43](https://youtu.be/atrVucJRtrU?t=1063) So here's another example of where benefit-cost analysis is a concern when you make decision why it should not be the sole factor. So in this case, what happens, and I've seen this happen in West Virginia, is in West Virginia, this is based in Pittsburgh, Pennsylvania. I do a lot of consulting work in West Virginia. And the housing sometimes is very, you know, particularly in these former coal towns, the value of it's very low. So it might be say $10,000 and you can get a house. But if you were going to respond, like when there's a flooding incident like this one, you know, maybe like say putting the house on stilts would cost $40,000. So from a benefit-cost perspective, it would not make sense to do the intervention because the cost is greater than the benefits. It's better to move people to someplace else, to Newham, rather than fix a town.

[18:52](https://youtu.be/atrVucJRtrU?t=1132) But then the problem is, is that, you know, you lose whole communities. And in some cases they have actually like relocated and moved towns. There's some interesting cases right now related to climate change, like say in Alaska and Louisiana, where they're basically trying to move entire towns to someplace because of sea level rise. And it's a very challenging thing because sometimes the communities are just not the same. So it's important to understand that it can be, again, a very sensitive issue and it's something that you want to do with care and consultation with community members. Okay, so these are the 10 steps. Oh, except I've got nine here. But there's 10 steps for benefit-cost analysis. This, sorry, this is a book that I recommend. So if you really want to to learn all the in-depth analysis and benefit-cost analysis makes sense. Oh, by the way, this one thing I should explain is that benefit-cost analysis is kind of like what people like these days because, you know, it's when you do the actual calculation, you're looking at the benefits, the cost less the benefits. But like when I learned it back in, actually, in engineering school, I learned it, we always call them cost-benefit analysis. So that's kind of like a term.

[20:25](https://youtu.be/atrVucJRtrU?t=1225) So this is basically the 10 steps that we're going to go through. And as you, so you can sort of see how this whole process works. We won't do this for all the different things, but this, what you do for best kind of, sorry, benefit-cost analysis will be pretty much the same with slight changes and some of these other issues. So the first thing is, is that there's different times you can do your benefit-cost analysis. You can do it at the beginning of a program, you know, like when you're deciding between like a bridge that lasts 30 years or 40 years. Maybe you want to do it some way, sometimes halfway through the program to see is this intervention working or is it not working? And then other times you want to do it at the end to say, okay, did it work or did it not work? So there's, you can do a benefit-cost analysis at different times in the policy analysis and program implementation process. But generally what you're doing is you're trying to look at the market or the government failure where something is, is not working. So from a, when you look at this from an economist perspective, you know, these are some examples of sort of the functions of government. But what happens is, is that markets are considered to fail when they don't take account of the externalities. So a simple example is, is that say you have, you know, like a industrial plant one, which is upstream from industrial plant two.

[22:01](https://youtu.be/atrVucJRtrU?t=1321) Industrial plant one emits pollution into the water. That water goes downstream to industrial plant two, which takes in the water. And before, because of the process, the manufacturing process in plant number two, they have to clean the water before they can use it in their process. So they have an additional cost for, for that, to clean up the water, but they don't get the benefit that plant number one gets, the one that's upstream. And so the benefit-cost analysis comes in to say, okay, well, what, what should be, how should we solve this problem, right? And from an economist perspective, it's the market that will solve it. But every once in a while, we need the government to intervene and benefit-cost analysis helps us figure out how much we should intervene, you know, relative to the impact that's going to have.

[23:10](https://youtu.be/atrVucJRtrU?t=1390) Okay. So here's an example. This is something, an actual thing that happened in Canada. So there is, there's this highway that's kind of, that's existing. And, but the problem is, is that there's, they're having a lot of accidents on the highway and there's two routes they can choose, route one or route two, and they're trying to decide between the two routes. So the reason why this is being done in terms of the purpose is that the freeways are a public good. But we have tolls that are politically challenging. That's again, where we're getting to this ease of public acceptability. They can't widen the existing road because of the canyon. And so the question is, should a new freeway be built? And so what you're doing is you're looking at the options to, in this case, the status quo is basically stay with the current road and the level of accidents that are there, or you could build a new freeway without tolls or one with tolls. And then this is basically how much it would cost. So something that's really important to benefit cost analysis is something called standing. So for example, in this case, we have the people who are the residents of British Columbia who would benefit the road and pay for it. But then we have other people, sort of non-residents who would use the highway but are not paying for it.

[24:52](https://youtu.be/atrVucJRtrU?t=1492) If you had the toll though, then they would help pay for the road as well as the residents. So the next thing is to look at the impact categories and figure out what they would, what their measures. So in this case, we're looking at the time saved due to reduced congestion for the existing users, which you'd measure in hours. The vehicle operating costs saved in dollars, the safety benefits, which would be the lives saved and the accidents reduced. You'd looked at the new users, so that they would have more ability to travel the road. Toll revenues, if in case of the toll road option, and then the horizon value, which is basically the project life. So basically thinking about how long these interventions would last. Then you would have the construction cost for building the freeway. So that's called capital cost. The operating maintenance costs, snow and removal, toll collection, toll booth construction.

[26:02](https://youtu.be/atrVucJRtrU?t=1562) And then you have what are called spillover effects. So things that you worry about, like what impact will this have on air pollution or the elk population, sea level rise, what impact will this have on air pollution or the elk population, scenic beauty, hotel and gas incomes. I still remember back in the days, maybe there's some of you out there that remember good old Route 66. And when Route 66 was closed, and I think it was like Route 40, whatever was built, then a lot of lives changed for a lot of people. And then you have the social cost, what's called the opportunity cost of the land. So if we decide to build a freeway there, then that means we can't say turn it into a national park because we have a freeway there instead. And so by taking this action, that means that we're doing something else. Okay. So this is something when you get the slide deck, you can listen to this, but it's basically talking about how you value a statistical life. And this is a confusing issue. The video here, which is from YouTube, is with Kip Discusi, who is a professor at Vanderbilt University. And he came up with a concept of how you do the value of a statistical life. This interview from this reporter who is, I think he's in Australia, New Zealand, somewhere south, was done during the pandemic. And it's a really good explanation of what happens. And the way the value of statistical life is, so it's basically about $10 million to prevent one expected death, was looking at the... For people who take jobs that are risky, how much more you would pay them to undertake that risky job? So you can imagine, I don't know,

[28:04](https://youtu.be/atrVucJRtrU?t=1684) somebody who is like cleaning windows on a big giant skyscraper, right? That person is at risk, more at risk than somebody who's doing windows that are on the ground. So how much more do you have to pay a person who has to go up high versus a person on the ground? That difference, that the dollar value, and this is from a study that he did a long time ago, is how we determine the value of statistical life. Now, this is under discussion. I haven't checked on it recently, but there has been discussion about how to change the value of statistical life. But this is what's been done for eons and eons, which is basically $10 million. Now, the good news, even though it's hard to put that kind of value in terms, is that it's so high that most of the times when there's anything related to health or death or things like that, then the benefits of intervention to reduce that pretty much always win. Rarely will they not win.

[29:14](https://youtu.be/atrVucJRtrU?t=1754) So I think that's important to understand. Another thing to know about the value of statistical life is that it's not based on age. So the value, like if it's a baby versus somebody who is like 80 years old, it's the same regardless of what age people are. And so this is what is used in federal evaluation. So then the other concept in benefit-cost analysis is willingness to pay. So the question is, how much are people willing to pay for the time they saved using the road in this particular situation? Or it might be how much Are people willing to pay extra to have a vehicle that is less polluting to the environment? What is that number? So, economists will conduct analysis on willingness to pay to have some idea of what the value is for a particular policy.

[30:24](https://youtu.be/atrVucJRtrU?t=1824) So, in this case, for our example, the leisure time saved per vehicle at that time, for this Canada example, was 25% of the gross wages times the average number of passengers, so about $6, $7 per vehicle hour. The business time saved per vehicle was about 12. The truck driver's time saved was about 14, and the value of a life saved was $500,000, which is, you know, like, quite low relative to what we are used to doing. But that value of Cisco Life varies from one country to another, and there's no requirement to use it in non-federal situations. But oftentimes, you do have to sort of figure out what the value is of a Cisco Life. Okay, the other thing I can do, and again, this is a useful video that explains sort of discount rates. So, the challenge is that you have a bridge, you're like, let's just say you have a freeway that lasts 30 years, another one lasts 40 years, and there's going to be, there's different costs initially for the capital cost, but there's also cost for operating maintenance.

[31:49](https://youtu.be/atrVucJRtrU?t=1909) So, the 30-year, say, roadway, you know, you'll only be paying operating and maintenance costs for 30 years. For the other one, you'll be paying 40 years. So, you have to figure out some way so that these two different interventions can be compared. And so that, so what you're doing is you're using a discount rate, basically an interest rate, to obtain what they call present values, like, so that they're on the same terms. And sometimes this can be a big issue, because discount rates can vary a great deal, like, you know, 3%, 6%, 9%, things like that. Oftentimes, you'll see those in analysis, like three different interest rates. But it's also something that might, depending on the cost of the intervention, can really be a major factor in determining if the benefits outweigh the cost, or the costs outweigh the benefits. So, because of this, you do something called sensitivity analysis, which we'll talk about later.

[32:53](https://youtu.be/atrVucJRtrU?t=1973) Okay, so, again, this is basically the second part of this. So, we take the net present value of a policy, is the present value of incremental net social benefit, and so we're bringing it all down to the same aspect. So, if the present value of the benefits is greater than the present value of the cost, then the analysis should continue. But if it's not, then from an economic perspective, it does not make sense to do the intervention. So, you're looking, again, at economic efficiency, what's going to get the best bang for the settlement. Okay, so here is a table that looks at the different costs. So, we have these things from a global perspective and a provincial perspective. So, one is, you know, for a small business, you know, the cost of the intervention is $1,000,000, and then the cost of the intervention is $1,000,000. So, here is a table that looks at the different costs.

[33:55](https://youtu.be/atrVucJRtrU?t=2035) So, we have these things from a global perspective and a provincial perspective. So, one is, you know, for, like, everybody who uses the road and others for those who actually live in this province, in this state, in Canada. And so, you'll see here the project benefits in terms of cost, the rise in value in terms of how long it lasts, safety benefits, you know, how that's going to vary, alternative routes, toll. In this case, you don't have the toll revenues, but you do here, and then the total benefits. And then you look at the cost. And so, the costs are construction, maintenance, toll collection, toll booth construction, and then you get those net set of benefits. So, you see here that there's cost for the toll, because you have to have people working it, and then the construction of the booths, but then there's also benefits when it comes to revenues.

[34:55](https://youtu.be/atrVucJRtrU?t=2095) And so, you'll see here the 2 options we're looking at with tolls and without tolls, without tolls and with tolls. And you are looking at, and you're comparing these 2 costs, the total benefits versus the project costs, which are the net societal benefits. Oops, sorry. Let me go back. Okay. So, you can see here in which cases the benefits are greater than the cost. And it's true for all the situations, except for the last one for D. And in that case, the cost exceeds the benefits. And so, these other options are still in play. Okay. So, this is the sensitivity analysis. Again, this is a video to do sensitivity analysis 101.

[35:59](https://youtu.be/atrVucJRtrU?t=2159) But basically, the concept behind sensitivity analysis is that the things happen, right? Wars happen, energy prices changes, we have pandemics, all these different things. And we cannot predict the future in terms of what's going to happen, particularly over decades and decades. So, what we do to take account of that is we do sensitivity analysis. So, for example, the price of natural gas has gone up and down, or let's just say gasoline has gone up and down a lot. You know, like when I was a kid, it was, I don't know, 23 cents or something. And now it's like three, four or five dollars. So, what you do is you focus on that one variable and you say, okay, well, what if it changed? What if it was half the cost, double the cost? Would that change our decision, or would it not change our decision? And so, when you're looking at an analysis, you always want to see, did they perform a sensitivity analysis or not? If they did not, then that is an indication that that is a poor quality benefit-cost analysis, because there should always be a sensitivity analysis. And because you just, you know, the world is an unknown place, particularly when it comes to cost and things that occur. Okay, so the final step is to make a recommendation.

[37:31](https://youtu.be/atrVucJRtrU?t=2251) You're looking for the policy option with the highest net present value that has the greatest economic efficiency. But again, economic efficiency is only one of these different factors that you consider. So, the other thing is that people have different perspectives on benefit-cost analysis and what should be included and not included. So, there's analysts, economists, they tend to include all the cost. There's people who, the authors are called the spenders. They tend to emphasize the benefits and try to monetize all those advantages, use lower interest rates, and then the guardians that emphasize the cost and try to limit spending. So, because of this, like you'll, if it, you know, if an analysis comes out, like here in Pennsylvania, there was a lot, there was a very controversial benefit-cost analysis about the benefits of hydraulic fracturing and shale gas. And part of the reason it was controversial is because there were each of these three types of individuals who like, who had issues with the analysis that was done by an academic, I think it was at Penn State.

[38:59](https://youtu.be/atrVucJRtrU?t=2339) So, the bottom line is you always want to be very careful in using any sort of, any sort of benefit-cost analysis and never trust somebody who is present, you know, other than the author who is presenting a benefit-cost analysis, because they will spin it to whatever suits what they're looking for. Okay, so, indeed, I went and I looked it up and you can go here and watch the video and look, you can see how old this analysis is, because see how old that car is? But the highway, the new highway was indeed built. And I just, I thought it was really interesting that to have this kind of like old car coming through. So, lots more details in this video. The good news is there's a lot that's online. So, here, that's kind of like an overview of how it's done formally by economists. Here's how somebody like myself, who's not an economist, but who's a policy analyst, how I do benefit-cost analysis. Okay, so, the first thing I'm going to do is I'm going to identify each option's three Okay, so, the first thing I'm going to do is I'm going to identify each option's three most important costs, three most important benefits, and I'm going to go to Google Scholar and see, is there any benefit-cost analysis that I can find? Most of the time, there will not be one for my specific situation, but I can sort of develop an estimate. If you can't, then you have to do some back-of-the-envelope kinds of things. And just always remember, more information, even if it's not perfect, is better than no information. So, in this case, the question we're asking is what actions the Food and Nutrition Service should take to respond to equity concerns about federal nutrition assistance programs, particularly for households with the children, headed by a single woman, that have the highest rate of food insecurity in the United

[41:00](https://youtu.be/atrVucJRtrU?t=2460) States. And so, we're looking at the status quo, which is the current program called WIC for women, infants, and children, and two different options. The first option would offer services over phone and video and allow remote benefit issuance. The second was to provide transportation vouchers. So, there's a lot more in the book that explains what this full situation is, but I'm not going to go into it here for lack of time. So, what did I do? For the status quo, the current situation, I went to Google Scholar, and I looked for benefit cost analysis on the WIC program. And I was lucky. I found a number of different things, including, interestingly, like analysis in other countries that was done on the United States WIC program. And then, I could sort of figure out what the total costs are. For option one, that is the one where the idea is, you know, so that women didn't have to go with their kids in person to a center.

[42:02](https://youtu.be/atrVucJRtrU?t=2522) They can instead apply online. I found one that discussed effectiveness and gave and had information on how much was saved by doing the program. For the second one, which is about having travel vouchers, like a travel voucher for like an Uber or something like that, I couldn't find anything at all. I found something that there was a program that there wasn't any information on cost. So, I went and found the average cost of a ride with a national ride share company, and I used that to develop a back-of-the-envelope estimate. And so, here, you'll see what the benefits are. So, for example, like whenever you see a number here, those are actual citations to things in the academic literature or to the USDA website where this program is run. And so, like one benefit was pre-NATO, WIC enrollment reduced first-year medical costs for youth infants by 1.9 million. It saved money for Medicaid, and then they paid for themselves within a year. The cost, I was able to find how much it costs per person for the Medicaid, for the WIC program if the number of participants increased.

[43:30](https://youtu.be/atrVucJRtrU?t=2610) Right now, 40% of those who are eligible do not participate, and that was the audience I was focusing on. So, if we have more people participate, then that will increase cost of the program. And then I looked at non-monetary considerations like improved cognitive benefit, more likely medical care, immunizations, and reduced fetal deaths and infant mortality. So, based on this, it just turned out that the benefits were the same as the cost, and this is again based on some academic literature stuff. I mean, I'm sorry, the benefits for both options were the same, but the costs were different. The digital option was less expensive by quite a bit than the transportation voucher option. And so, in this case, the cost of the transportation voucher outweighed the benefits, while the opposite was true for the digital thing where the benefits sort of cost. The benefits of the program outweighed the cost.

[44:37](https://youtu.be/atrVucJRtrU?t=2677) So, then you would kind of like put all the summary information together and come to the conclusion that the digital application benefits from the efficiency standpoint sort of made sense. Okay, so now we're looking at different things, like when you're communicating benefit-cost analysis. Some of the important principles here are Transparency and Clarity, Assumptions are Really the Big Issue, Data Sources, Making Sure you Prioritize Stakeholder Information Needs, and Contextualizing. So you want to keep in mind the goals, the socioeconomic and environmental goals of what you're doing, and you want to use clear and concise communication. So these principles are true for all of the methods that we're talking about, but I'm going to add in some additional elements. But these four things, you know, are true regardless of what analysis it is you're communicating. So now we're going to get into cost-effectiveness analysis. And so, as I sort of mentioned earlier, in benefit-cost analysis, cost is the primary consideration, and with cost-effectiveness analysis is primarily quality of life.

[45:50](https://youtu.be/atrVucJRtrU?t=2750) In benefit-cost analysis, you assign a dollar for a value for a statistical life, but in cost-effectiveness analysis, you do something called an ICER calculation, which is talking about economic cost. I'll talk about that more in the next slides. Then in benefit-cost analysis, it's only the social benefits, the net social benefits just have to be positive. The benefits have to weigh out by the cost to go to the next step. And it analyzes the cost of the interventions relative to the status quo within each other. But in cost-effectiveness analysis, you're basically comparing two options relative to each other, two different interventions. So it's most commonly used for issues related to health care, defense, transportation safety, almost anything that involves like, you know, human lives.

[46:50](https://youtu.be/atrVucJRtrU?t=2810) And you know, you'll sort of see here, just like a comparison of the two, looking at how they measure outcomes, the context of the decision, the comparability, the perspective, the goals, and the resource allocation. So cost-effectiveness analysis, you define the scope, measure, identify and measure the cost and the outcomes. And then this, then you calculate this thing, which is called incremental cost-effectiveness ratio. Again, you perform the sensitivity analysis and then you interpret it relative to the willingness to pay threshold that's determined by policymakers, and then you report the findings. So, what you're trying to do is you're trying to figure out what policy options can save the most human lives at a fixed cost. So you're comparing two or more interventions. So cost is on the top in the numerator and health effects are in the denominator.

[47:57](https://youtu.be/atrVucJRtrU?t=2877) And it's important to understand that cost-effectiveness is not the same as saving costs. It just basically means that it's more effective in achieving a set of goals than the other costs. So here's an example of two programs. So in Program A, the intervention was the mobile text messaging to encourage people to take some sort of medication for diabetes. And the other one was having like a real person do it, a diabetes care coordinator. The cost of the text messaging was $40,000 and the cost of the diabetes care coordinator was $150,000. And then you have these quality adjusted life years. So what this is basically saying is that when it comes to the human factor, that having a diabetes care coordinator was better than having mobile text messaging.

[49:09](https://youtu.be/atrVucJRtrU?t=2949) It wasn't like one versus another, but there's like a difference of about 10 between the two. So what you do is you take the cost on the top, 150 minus 40, and then you put in this quality adjusted life year factor on the bottom, 35 minus 25, and that leads to $11,000. So that means it's cost effective, right? But then there's another step. And the other step is that there is usually an arbitrary threshold. And based on this threshold, you decide whether or not to go forward with the action or not. Okay, so again, there's no perfect answer for this. It's going to depend on the policymaker. But in the United States, it's considered, it's generally about $50,000. In the UK, it's about $25,000 to $35,000. And either in this scenario, it means that you would like go forward, you know, with option, sorry, with Program B here versus Program A.

[50:19](https://youtu.be/atrVucJRtrU?t=3019) Okay, so cost utility analysis is sort of a variance of cost effectiveness analysis. And you basically look at the difference between the not, you're not looking at the, sorry, the cost is the same in terms of the numerator, but you're looking at the quality adjusted life years for the denominator. So you'll see here that, yeah, in this example here, you're going to live five more years. The utility, that is the quality of life, is say 0.8, where the zero is death and one is perfect health. That's determined by the patient community. You multiply those two things together, and that gives you your quality adjusted life years. And here's an example of how something like this might be determined. You'll see that the variables will differ. Like, this is like assisted daily living, I always forget what that is, exercise, mental clarity, emotional well-being.

[51:25](https://youtu.be/atrVucJRtrU?t=3085) Jane's evaluation is different than Joe's and because of this, their quality of adjusted life years differs. And so you put this sort of in your calculation, and that determines your quality adjusted life years, and then that's going to vary dependent on how long the person lives in terms of their additional years. So some of the disadvantages, it can be challenging to measure preferences, that they can vary over time. Personal life context can vary, as can the impact of the health intervention. Of course, there's always uncertainty, you can always just get hit by a bus. So the key is you just want to make these things more transparent. And then you want to always also figure out, can we afford it? So here we have two drugs. So one has an incremental cost of registration of $28,000 compared to drug B, so it's cost effective.

[52:28](https://youtu.be/atrVucJRtrU?t=3148) And then the drug B costs $70,000, but then there's like 10,000 people eligible for it. So you have like $980 million. But what if your budget's only $500 million? Well, that you still have to figure that part out. So just because it makes sense, doesn't make sense that you can do it. Here's an example in Oregon, where what they did is they wanted to make the greatest health care benefits for the resources available. They rank 709 medical procedures. According to the benefits for the public to serve for Medicare, state legislator determined the threshold. Then the annual threshold was determined based on health care services, cost estimate and budget. They took input from physician panels and a public survey. It was controversial, so it got rejected by the federal government because it was about Medicare, but then it did get approved.

[53:30](https://youtu.be/atrVucJRtrU?t=3210) Okay, and then here's another example. This is from FEMA. So here's an example where the federal government decided that California earthquake standards should be imposed on all U.S. communities near faults regardless of risk. The earthquake risk in Memphis was 10 to 30% of that in California. So if they had done this kind of analysis, they would have found that there were other better ways to spend subtle money, like highway upgrades, flu shot, heart defibrillators would have saved more money than these earthquake requirements. In terms of communicating cost effectiveness versus benefit cost, we have things related to the health outcomes that are in these natural units. The target is typically health policy makers, so they have some familiarity. And, you know, you're trying to emphasize maximizing health outcomes within a given budget, and that often requires these kind of more detailed explanations.

[54:34](https://youtu.be/atrVucJRtrU?t=3274) So is it okay, Hannah, if I keep on going through risk analysis?

**Unknown speaker**

[54:40](https://youtu.be/atrVucJRtrU?t=3280) Yes, yes, absolutely.

**Deborah Stine**

[54:42](https://youtu.be/atrVucJRtrU?t=3282) Great, thanks. Okay, so the last category here is risk analysis. Okay, so risk analysis has sort of like three parts to it. So you have risk assessment. So that's basically how hazardous something is. And then you have risk management. So a risk, somebody who is qualified in risk assessment says, oh, well, I don't know if I'm going to do this. But then it's up to a risk manager, typically a policymaker, who says, hey, we should or should not do something about that. And then a risk communicator then communicates that analysis to the public. So here's an example of what you do when it comes to health risk assessment. Okay, so the first is you identify a hazard. So let's say you have a hazard. So an example might be, say, asbestos. And then you have what's called the dose response, which is basically if I take a dose of an insect, it's going to be a positive dose.

[55:50](https://youtu.be/atrVucJRtrU?t=3350) And then if I take a dose of an insect, it's going to be a negative dose. So the reason why you do that is because it's not an accurate dose. And then you have the dose response, which is basically if I breathe in so much asbestos, this bad thing is going to happen to me. But on the other hand, if I'm never exposed to asbestos, I don't have to worry about it, right? So you have to go down here and figure out the exposure that person has. And when you put all this together, you get what's called risk characterization. And then this risk characterization, along with different ways to control the risk to reduce, say, the amount of asbestos to an acceptable level, is fed into the risk manager who sort of makes that decision. Here's another example of that same sort of concept. So this is, again, FEMA looking at risk assessment.

[56:53](https://youtu.be/atrVucJRtrU?t=3413) So you have all these hazards, bad things that can happen. But then you're also looking at vulnerability, which is the different assets and so forth that are at risk. And then those two combined together determine the impact. So if you take fire that's in the house or something that's going to affect people, that will result in casualties. So risk assessment is risk analysis and so forth. It's done in different ways depending on the context that you're using. So here's yet more bad things that can happen. Avalanches and droughts, there's these natural things. Technological things like dam failure. Hey, you know, we just had that right in Iowa, I believe. And then we have things that are human caused. So these are all risks and things that can happen and where you might be conducting a risk analysis.

[58:00](https://youtu.be/atrVucJRtrU?t=3480) Sorry, my mask has decided it's tired of moving. Here we go. So here is how FEMA does risk analysis. It's a little bit different. You'll see that they do look at the cost and the benefits. But then they look at here the asset value. They do that assessment. They do vulnerability assessment, threat hazard assessment. All those combined to go to risk assessment. Then you identify mitigation options. And then it goes to the decision. This is something where this is like an example of how they work with stakeholders. You know, so if you're trying to, for example, like say people are worried about a dam in their neighborhood. And they part of what you're supposed to do is sort of like engage with the community. So you can understand what the hazards and threats are and what it is that they are worried about.

[59:06](https://youtu.be/atrVucJRtrU?t=3546) And so this is how this is again managed by Department of Homeland Security by identifying threats and hazards, giving threats and hazards context, and then establishing the capability to respond to those interventions. So you'll see here at the top, community identifies the threat or hazard that presents the greatest challenge to them. And then sometimes you'll see that it's sort of a single threat. And you can sort of figure out like where it is. So it's going to be different depending on like where the community is. Obviously, someplace like New York City is going to be, you know, different than the middle of Iowa. So in terms of risk communication principles. These are sort of the fundamental things to think about, which is, first, to use plain language and avoid technical jargon, to really provide complete and transparent information about the risk and the uncertainties and the limitations of the analysis.

[1:00:20](https://youtu.be/atrVucJRtrU?t=3620) Oftentimes, you're communicating these to the public, and so it really is something that you need to make very clear, and sometimes things can happen where there are assumptions the risk analyst has that are incorrect. You want to state what's known and unknown. You don't want to downplay the potential risk. You want to place the risk in context by comparing it to other risks, and then engage with the audience, encourage questions, provide opportunities for dialogue, and then responsible to concerns and feedback, and explain the reasoning behind the analysis. And that is it. This is my webpage, and I will stick the link into chat. I do have a newsletter that comes out for other free webinars and things like that that I offer, and also I have on LinkedIn some science, technology, policy job opportunities.

[1:01:28](https://youtu.be/atrVucJRtrU?t=3688) I now have just started an opinion piece that I'm trying to write each week. I still got to get it done for this week. So that's it, Hannah.

**Unknown speaker**

[1:01:36](https://youtu.be/atrVucJRtrU?t=3696) Thank you so much, Debra. That was great. I'm just going to take us off recording.

* * *

Original page: https://innovate-us.org/evaluating-effectiveness-and-efficiency-through-benefit-cost-cost-effectiveness-and-risk-analyses · Licence: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org