---
id: "workshop:ai-snake-oil-what-artificial-intelligence-can-do-what-it-can-t-and-how-to-tell-the-difference"
type: "InnovateUS workshop"
title: "What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference"
url: "https://innovate-us.org/ai-snake-oil-what-artificial-intelligence-can-do-what-it-can-t-and-how-to-tell-the-difference"
date: "2024-11-06"
duration: "60 minutes"
language: "English"
instructors: ["Arvind Narayanan","Beth Simone Noveck"]
recording: "https://www.youtube.com/watch?v=ZxlaFCT4kpY"
transcript: "automatic, from the recording"
source: "InnovateUS (innovate-us.org)"
licence: "CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org"
---

# What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference

InnovateUS workshop · 6 November 2024 · 60 minutes

- Learn about the differences between predictive artificial intelligence (AI), generative AI, and other types of AI

- Understand how to recognize AI hype and AI snake oil

- Learn how to effectively incorporate AI into your workflow

- Discuss which risks of AI are pressing and which are overblown

- Learn about what the future of AI holds

## Learning goals

- Learn about the differences between predictive artificial intelligence (AI), generative AI, and other types of AI

- Understand how to recognize AI hype and AI snake oil

- Learn how to effectively incorporate AI into your workflow

- Discuss which risks of AI are pressing and which are overblown

- Learn about what the future of AI holds

## Speakers

- **Arvind Narayanan**, Professor at Princeton University & Director of the Center for Information Technology Policy
- **Beth Simone Noveck**, Founder of InnovateUS and Director, Burnes Center for Change and the GovLab

Recording: https://www.youtube.com/watch?v=ZxlaFCT4kpY

## Transcript

Automatic transcript of the recording: names and terms can be misheard, and the times can be a few seconds off. Each time links to that moment of the video. The transcript alone: https://innovate-us.org/ai-snake-oil-what-artificial-intelligence-can-do-what-it-can-t-and-how-to-tell-the-difference.txt (plain text) and https://innovate-us.org/ai-snake-oil-what-artificial-intelligence-can-do-what-it-can-t-and-how-to-tell-the-difference.json (every segment with its start and end second).

**Beth Simone Noveck**

[0:00](https://youtu.be/ZxlaFCT4kpY?t=0) Innovate US, this wonderful program that we're happy to bring to you with free learning around AI, tech, innovation, and engagement skills focused on the public sector. So on this post-election day in the United States, I'm glad to see everybody joining us. And I'm especially glad to welcome Arvind Narayan, who is, of course, needs really no introduction, but is a very distinguished professor of computer science at Princeton University and director of the Center for Information Technology Policy. You're all here because he is the co-author of this wonderful new book called AI Snake Isle that we will hear him talk about in just a moment. I'm going to put a link to the book in the chat so that you can all follow along and hopefully get your copy. And I'm also pleased to say that for attendees in this group, there's also a discount code if you want to order the book.

[1:01](https://youtu.be/ZxlaFCT4kpY?t=61) Here at Innovate US, though, we're very focused on how we can work better and differently in government to serve our residents and to serve the public. So we've asked Arvind to make this a very skills-based practical talk for what each of us can do better and differently in our own jobs as we confront opportunities to use AI for the greater good. So with that, what we're going to do is I'm going to turn it over to Arvind and to kind of introduce the topic, tell us about the book, give us some advice, but we're going to welcome your questions, which I'm here to moderate. So by all means, use the chat for discussion. But if you have a question for him that you're going to want to ask, put it in the chat and just label it with the words question in capital letters. And I'll try to moderate some of that as we go forward. In addition, I'm just going to stick a link in the chat to Arvind's fantastic blog by the same name of AI Snake Oil, which you might want to join the other 40,000 people who have signed up to follow it.

[2:07](https://youtu.be/ZxlaFCT4kpY?t=127) But right now we're going to take our own personal in-person dose of that newsletter and hear from you, Arvind, about the book and welcome. And again, just for folks here, please do rename yourselves and identify yourselves by renaming yourselves with the clicking on your where so Arvind knows who's here. Okay, with that, let me turn it over to you. Thank you.

**Arvind Narayanan**

[2:32](https://youtu.be/ZxlaFCT4kpY?t=152) Thank you so much, Beth. I hope you can see my slides. Hi, everyone. It's great to see so many of you here on a day when obviously there's a lot going on. So as a co-author of the book, AI Snake Oil, let me start right away by saying I am not an AI skeptic. That is not what this talk is about. There are so many opportunities to use generative AI in government and really everywhere else. But in order to be able to do that, to take advantage of those opportunities, we have to look past the hype. I think there is so much hype out there. That's why we wrote the book. It often confuses us. And so in the next 30 to 40 minutes, I hope to be able to share some tips with you. So the way I want to do this is maybe for the first 10 minutes, I want to talk about some foundational concepts. I'm sure some of this is going to be familiar, but hopefully there's something useful there. But then I want to spend the bulk of my time on thinking about, okay, what are the different use cases and how do you select a tool? How do you match that to different use cases?

[3:37](https://youtu.be/ZxlaFCT4kpY?t=217) And towards the end, I want to talk about the more direct snake oil part of it. What are some of the overhyped claims that we hear and what should we be careful about when we hear those claims? Okay, so let's get started. Now, in the process of writing the book, what we noticed is perhaps the most confusing thing about AI that makes some of the discussions, makes people talk past each other, is that AI is kind of an umbrella term for a set of loosely related technologies. Generative AI is what we're going to talk about today. And this is a screenshot of ChatGPT. But that is nothing similar to when, for instance, a bank might say they're using AI in order to judge who is a good credit risk and who is not. There is something similar between these. They're both examples of learning from data. That's why they're called AI. But in terms of what kind of data that is, what can go wrong, what kinds of guardrails are needed, these are very, very different. So this credit application is an example of predictive AI. And by predictive AI, we mean making decisions about people by predicting something about their future, their behavior, what they might do in the future, will they pay back a loan or not, or some outcome, something that might happen to them. Are they going to succeed in this job or not? So AI is used a lot in hiring, for instance. It's used in the criminal justice system in making a whole lot of consequential decisions about people.

[5:06](https://youtu.be/ZxlaFCT4kpY?t=306) This kind of AI is often in the news for bias and discrimination, but we argue in the book that there is something more fundamentally problematic about it, which is that predicting the future is hard no matter what data you have, what AI you have. And we ask whether it's even morally permissible to be making these sometimes life and death decisions about people based on predictions that are only slightly more accurate than the flip of a coin. We have many, many horror stories we talk about in the book. For instance, in medical insurance, an insurer might use AI to predict that this patient is going to need 17 days to recover in the hospital after a surgery. I don't know where they got such precise predictions from. But then after 17 days, even though the patients can no longer walk, their insurance payments have stopped, and they're on the hook. So lots of examples of misfires of predictive AI. And that's what we're most critical about in the book, not what we're going to talk about today. And generative AI, I don't need to explain what it is. But I do want to mention, you know, there are many other types of AI. There are recommendation algorithms on social media, robotics, for instance.

[6:18](https://youtu.be/ZxlaFCT4kpY?t=378) We talk a little bit about these types of AI as well in the book. I do want to mention one kind of funny definition of what AI is, because this relates to my optimistic vision for the future of AI. And that definition is, AI is whatever hasn't been done yet. AI is whatever hasn't been done yet. What people mean by that is that when a particular type of technology doesn't work well, when it has a lot of flaws, maybe dubious social implications, that's when we're more likely to call it AI. But once it starts working well, it kind of fades into the background. We take it for granted. We don't think about it. We don't necessarily call it AI anymore. So these are all examples of things that would once have been called AI. Even spellcheck was a cutting edge technology, I remember, a few decades ago, but we wouldn't call these AI anymore. And my hope is that much of today's AI will similarly fade into the background.

[7:21](https://youtu.be/ZxlaFCT4kpY?t=441) So, for example, self-driving cars often make the news today because of occasional crashes and even deaths. But that's largely a solvable engineering problem. I do think we've come such a long way in solving that problem. Of course, there are something like a million rides per month, I think, already that Waymo is doing with its self-driving taxis. And also, we have to figure out social consequences like job development. But I do think one day we will have figured that out. And we will take self-driving cars for granted as part of our physical environment. The word car will probably mean self-driving car. We're going to need a new word, maybe manual car, to refer to what we call cars today. And so this is, I think, what we should hope for. But I think in the process of getting there, there are a lot of risks. There is a lot of hype. There are a lot of harms that we should be concerned with. And so while I'm going to be talking a lot about generative AI today, I'm going to be talking about its uses. I will also, to the extent possible, try to mention some of these risks and caveats.

[8:27](https://youtu.be/ZxlaFCT4kpY?t=507) Okay, just a couple more minutes on some of these foundational discussions, and then I'll jump into how to select a tool for a particular application. So most AI today, I'm sure you know this, is based on machine learning. It wasn't always like this. It might not always be like this in the future, but it's what it is today. And one important consequence of this is that by default, the data for training machine learning systems comes from data that's created directly or indirectly by people. And so it is going to contain the biases and stereotypes in our society. So unless we take explicit steps to correct those biases, we should expect our machine learning based decision making systems to be biased as well. So that's one point to keep in mind. So one other point specific to generative AI that I want to give is scaling. Again, I'm sure you've heard of this, but I would just want to emphasize this for a second. Generative AI is trained on content from the web. And let's look at some numbers. So today's most powerful models are trained using something like, in terms of the number of tokens or the number of words that go into training. It's something like the number of words that go into training.

[9:42](https://youtu.be/ZxlaFCT4kpY?t=582) Training, it's something like Perhaps if people can mute themselves. Sorry. Okay, great. So trillions of tokens and somewhere between 100 billion and a trillion parameters or connections between neurons. And 10 to the 25. So that's 10 trillion trillion operations, math operations during training. And this is what in the last decade has led to so much progress in generative AI. A big question is how much longer scaling will continue. I will come back to that briefly towards the end of the talk. And this is, of course, what is responsible for generative AI's tremendous energy consumption as well. Okay, so let's dive into the practical part of what I want to discuss today. And the way I'm going to do this. I'm going to walk you through a bunch of use cases and then the use cases themselves are not so important. It's just to Give an illustration of the wide variety of different things we can do with generative AI. But then I want to make some observations in terms of how do you go about making decisions.

[10:51](https://youtu.be/ZxlaFCT4kpY?t=651) So yeah, these are just for illustration. People often ask me, which model should I use. I think that's the wrong question. What we see, at least as of today is a remarkable degree of convergence between the top models. Released by different companies and research labs. There's not that much difference between them from a practical perspective. So what are the actual differences. I think that will become clear as we talk about these use cases. I will also try to mention what the risks and caveats are. And one high level thing I want to highlight right away is that I'm going to divide these into automation use cases and co creation use cases. When I'm talking about automation. So it's a task you're doing in your everyday workflow and you want to use AI to automate that task so that you can save time. Co creation is quite different. You want to use AI to enhance your creativity to perhaps do new tasks that you can't do now or to do tasks better than you can do without AI assistance. And I want to advocate for thinking about those kinds of tasks differently. I mean, there's some blurry middle, but I've found it personally helpful to think of them as different.

[11:59](https://youtu.be/ZxlaFCT4kpY?t=719) So here are a few automation use cases. By the way, these are all things that I use on a pretty much on a weekly basis and none of this here is kind of aspirational like a lot of the hype that you see this is something I could do one day. These are things that, you know, that, that have that aren't perfect, but work well enough that you can take advantage of them. So semantic search. Semantic search is the idea of going beyond simple keyword matching. Of course, traditional web search moved beyond simple keyword matching a long time ago. But yet, it is far from being able to understand the nuances of your query, but recent AI based search methods using embeddings can, to some extent, allow you to do that. So this is a typical example of a query that a traditional web search would struggle with. If you want case studies of AI implementations in government agencies, for instance, a web search is still too focused on keywords. So if a document doesn't have the term government agency instead of talking about the IRS or any particular agency, it might not score very highly.

[13:04](https://youtu.be/ZxlaFCT4kpY?t=784) So I use a tool and my team members use a tool called Elicit to search the academic literature and it has proven to be a much faster way of doing searches for the research literature to find the papers that we want to find. It is still far from perfect. And we have to think about, you know, how do we do this? And if you want to search your own documents, for instance, on your computer, you would obviously want to use a different tool. So my point here is not to give you tool recommendations, but just to help you think about some of these use cases and Here's another one that I use a lot. Data cleaning. I made up this example, what kind of data you want to clean, obviously depends on your own workflow, but broadly speaking, this is a tool that I use a lot for data cleaning. So I've got a lot of data here. I've got a lot of data. I've got a lot of data. I've got a lot of data. I've got a lot of data. I've got a lot of data. I've got a lot of data. But in this specific made up example, these addresses have a lot of errors. Some zip codes are missing. There are lots of spelling errors. Traditionally, if you wanted to fix this, you would have to have someone who can program and they would have to create a program to fix it.

[14:13](https://youtu.be/ZxlaFCT4kpY?t=853) But in this specific made up example, these addresses have a lot of errors. Some zip codes are missing. There are lots of spelling errors. Traditionally, if you wanted to fix this, you would have to have someone who can program and they would have to create a bunch of regular expressions or whatever it is. It would take at least several hours of work, but today you can copy paste this into any chat bot and it will use its quote unquote common sense in order to be able to fix it. And you can invoke that in a programmatic workflow where you can have tens of thousands of addresses that you pass into it. Maybe a slightly more sophisticated variant of this is if you're working in a spreadsheet and you want to write a formula to process your data. This is one of the things I struggle with the most because I'm not such a writer. If I'm a regular spreadsheet user, it takes me, you know, sometimes 10 minutes to get the formula right, but with chatbots, I can just specify what I want in natural language and it will come up with the formula. The onus is on the user, of course, to verify the correctness and so forth, but nonetheless, this can be a big time-saving technique.

[15:21](https://youtu.be/ZxlaFCT4kpY?t=921) So one difference I want you to notice here, compared to the previous examples, is that this is integrated into an existing product, a spreadsheet. So a lot of the time, the value of AI comes from this integration. Another example of this might be in Photoshop, you have generative fill. Imagine how different the user experience would be if you didn't have this feature in your photo editing software. And instead you had to do a bunch of edits in Photoshop for the simple things, and then take your picture, go to a generative AI tool, and then fill in the rest of the image, come back to Photoshop and edit it and so forth. Would just not work. And so what we're seeing today is, in a lot of cases, people are figuring out some of these early use cases by experimenting with chatbots. But from a productivity perspective, when you're thinking about these automation use cases, I think it's going to be much smoother if it's integrated into the software that you're using. And so that is something you kind of want to be on the lookout for. If there are things that you're using a separate AI application for, to what extent can those be integrated into your existing software?

[16:30](https://youtu.be/ZxlaFCT4kpY?t=990) Excuse me. Another one I find myself using a lot is note-taking and summarization for meetings. Perhaps, you know, this very talk, I believe there is actually a note-taking bot that is live right now. Yeah, and I use it for a bunch of things as well. So this is another example where integration into existing software like Zoom actually helps it work much smoother. I do want to point out some of the risks here. You can see in this example, there was a leak of confidential data because the integration between AI and existing software often doesn't work that well. And the access controls that are programmed into Zoom are not necessarily respected by the AI bots that are integrating with them. Okay, and once again, if people could mute themselves, that would be great. Let me give you a few co-creation use cases now. So to me, these are different. These are not about automating tasks on a large scale.

[17:31](https://youtu.be/ZxlaFCT4kpY?t=1051) So let me give you a few examples. One thing that I often find myself using AI for, as someone who frankly has a, I'm embarrassed to admit this, I'm only admitting it because it really helps me make my point about an AI use case. I have very poor memory. So for instance, when I was organizing a conference, I wanted to think about who should I invite to this? Who are the experts on this topic? I found myself really benefiting from being able to ask the chatbot that question. And I want to clarify something here. The people that it's telling me are people that I know. Some of them are my friends. I know their work really well. I mean, it could be Beth, for instance, right? But because my memory is poor, I often have trouble recalling the names of people whose work I know really well. So this is different. Using it as a memory aid is different from asking about who should I invite to a conference or what topics should I cover in a lecture? When you yourself don't know the answer to that question and then deferring to the output of a chatbot, right?

[18:33](https://youtu.be/ZxlaFCT4kpY?t=1113) I think that is a very bad use case. That's not the way we should be using these tools. But when using it as a memory aid, you're not deferring to the machine's response. It's just for you to jog your own memory. And yeah, I find myself, you know, when I'm teaching a topic, I try to make sure that I'm not missing important topics, which again, I'm an expert on. I've probably written papers about, but have forgotten when I'm preparing this lecture. OK, I also find myself using generative AI a lot during writing. In general, I don't find bots, chatbots to be good at writing at an advanced level, but there are a lot of specific things I use them for. Some simple examples, formality, grammar, etc. Slightly more complex examples, like restructuring a document. But the ones in this last item, the more complex examples, these are the ones I'm more excited about.

[19:38](https://youtu.be/ZxlaFCT4kpY?t=1178) So this one in particular, what would a person who is an expert on this topic or has this particular perspective on this topic, what would they say about my paper if they were to read it? So these types of things have been genuinely very useful to me. And yeah, of course, I could try to reach out to experts on those topics and get their feedback, which I do. I'm not using AI as a replacement for human expertise. But to be able to do that in five minutes instead of waiting a week for a response really changes the feedback loop that you have when you're writing something and you want to make it more persuasive and you want to anticipate potential objections. I also use it a lot for data exploration. So something like, you know, I have a spreadsheet. This is something you can, if you haven't tried doing this, very much recommend it. You can just upload a spreadsheet and you don't have to give it any instructions. Just, you know, help me visualize this data set and it will do something pretty reasonable.

[20:42](https://youtu.be/ZxlaFCT4kpY?t=1242) So this is a chart of house price changes in various countries. And it went further by helping me visualize the data in a different way. So you can see here some kind of nice normal distribution emerging out of this. Most countries have a perhaps a healthy rate of house price increase, but there are some bubbles you can see on the right and there are some bubbles bursting as well. So, again, the reason I think of this as an example of co-creation and not automation is that I'm not going in knowing exactly what I want to do. This is actually helping my creativity. Now, of course, I have to verify the output by examining the code or whatever, or by double checking the answers before I use it for anything important. But making this creative loop faster, right? Thinking of new things to explore in the data. I find that to be quite helpful. Extremely helpful. You can even create simple apps with a prompt without writing any code. I find myself using this quite a lot.

[21:44](https://youtu.be/ZxlaFCT4kpY?t=1304) Again, I'm going to give a made up example here because the kinds of apps that are useful to me are probably different from the ones that are useful to you. But here's a really simple thing you can do. This is a feature in Cloud called Artifacts. And so if you ask it to create an app that shows the live time in various places, it can do that no problem in various other kinds of somewhat more complex apps as well. So I think AI is not yet good at software engineering. It's not yet good at handling a million line code base, but it's very good at creating these kinds of simple apps. So to summarize what I've said so far, when you're thinking about using AI, I think you should approach it differently depending on whether you want to use it for automation or for enhancing your creativity. And in particular, when using it for these co-creation tasks, I often find that even identifying what are those areas where generative AI could be useful is half the battle. As opposed to automation, where it's an existing task, it's something you do every day, you just want to automate it versus what new things can I now start doing, given that I have the benefit of generative AI.

[22:55](https://youtu.be/ZxlaFCT4kpY?t=1375) OK, so like I said before, the difference between models is not that important. What I do think is very important is a few things. So am I going to a general purpose tool like Cloud or ChatGPT and doing something, or is it integrated into existing software that I'm using for my workflow? And there are various specific features that you want for certain kinds of tasks, not always. So long context, the ability to be able to put in a whole book, for instance, and be like, extract all the parts that talk about this specific topic. So that's called long context. Code execution is another important feature, being able to search the web instead of just being able to answer from its own memory, if you will. And reasoning, so OpenAI's recent model release, O1, is capable of some amount of what we might call reasoning, although the way it happens internally might not be the same way that people reason agentic capabilities. So is this a bot that's going to be able to go online and automate things for me, like a simple shopping tasks, perhaps?

[24:01](https://youtu.be/ZxlaFCT4kpY?t=1441) So this is an area with a lot of hype and companies want to create agents that are going to be able to handle a lot of daily tasks for us automatically. Maybe that might happen one day, but we have a few papers on this topic, and I think the current state of the art is quite underwhelming. Another thing to, of course, consider is open versus closed source and so forth. Okay, so that was on how to select a tool, and one thought I want to give you is, as you're thinking about upskilling your own AI capabilities, you might think, oh, you know, ChatGPT happened a couple of years ago, so there's a lot of things that we need to learn about generative AI, and then we're done, we're, you know, we're going to be productive users of these tools. I don't think that's going to be the way it's going to play out. In our book, for instance, we talk about kind of the 80-year history of generative AI, how this has been a very slow, gradual increase in capabilities, and from a computer science perspective, ChatGPT was not radically novel.

[25:02](https://youtu.be/ZxlaFCT4kpY?t=1502) OpenAI didn't even realize that this was going to be the precise moment when chatbot capabilities exceeded that threshold, where it becomes useful to a large set of people, and therefore it kind of took off and went viral. That's why it's named ChatGPT, instead of a name that they would have surely picked if they wanted it to be a mass market product, that was not their intention, they released it as a so-called research preview, right? So, yeah, again, from the AI developers' perspective, the improvements have been really gradual, it's just that many people on the outside weren't paying attention before ChatGPT, and those improvements are continuing, they're not really going to stop, and because of that, they're not going to stop. And because of that, I think it's going to require, you know, every year or so, we have to continue acquiring new skills for how to use AI. Now, there are a lot of fears about job replacement, and what we've seen in our research, from a computer science perspective, also agrees with what economists are saying, which is that AI automates tasks, not jobs.

[26:13](https://youtu.be/ZxlaFCT4kpY?t=1573) And, you know, jobs are bundles of tasks, and because of that, while we're seeing in many cases that AI can automate specific tasks, we're not seeing a lot of cases where AI is going to be able to automate the entire bundle of tasks that goes into a particular job. And I'll have a case study of that when it comes to AI and law, I'll come back to that in a few minutes towards the end. So in terms of upskilling, another thing I'll say is one really good thing about generative AI is you don't have to trust the experts. I mean, it's nice that you're coming to this talk series, but I would say that, honestly, what is much more important than listening to talks about generative AI is just playing with it yourself so that you understand its capabilities and limitations. There are lots of people out there saying, oh, you know, this is going to change everything, you're going to get left behind if you don't start using gen AI for everything.

[27:16](https://youtu.be/ZxlaFCT4kpY?t=1636) At the other extreme, there are people arguing theoretically that, oh, you shouldn't use generative AI for search or coding or whatever it is. I think both of those are oversimplified. I myself, you know, the week or two after ChatGPT came out, I was making such arguments on Twitter. But once I started spending some time using these AI tools, I came to the conclusion that while there are risks with just about every use case, I'm not comfortable making these blanket statements. I thought it would be a terrible idea to use AI for learning because it can make mistakes, it can tell you the wrong thing. You don't want to learn the wrong thing about whatever topic you're trying to learn. But in fact, I've ended up using AI pretty heavily when I'm learning new topics, and I'm happy to get into that in the Q&A if there's interest, why it turns out it's so useful despite the fact that it can make errors. And so, you know, the way in which it's going to be useful for you is different from everyone else.

[28:17](https://youtu.be/ZxlaFCT4kpY?t=1697) So I would encourage at least 10 hours of experimentation to really get a good idea of what it can do for you and what the risks are in those specific use cases. So I want to briefly acknowledge that it's not all about individual responsibility. There are lots of things that need to be handled at the organization level, obviously, procurement of AI tools and any policies that are negotiated as part of that, such as confidentiality, setting internal policies for risk mitigation that will apply to everybody in the organization and fostering a culture of experimentation. For instance, we're seeing reports that many people in companies are using AI to enhance their workflow, but are kind of hesitant to share it with their organization because they worry that if management found out that they've managed to improve their productivity, they're just going to be rewarded with more work. Right. So that's not a healthy relationship between, you know, between bosses and workers when it comes to automation.

[29:26](https://youtu.be/ZxlaFCT4kpY?t=1766) And I think we should have a more healthy relationship where people are actually rewarded for experimenting and innovating with how to use AI more effectively. All right, so let me spend the last 10 minutes on the kind of dubious claims that one might often hear and then happy to go into the Q&A. So we talk about this a lot in the book where there are many products being sold that we think are just impossible, just conceptually impossible. I started on this topic, AI snake oil, more than five years ago because I kept seeing these AI. Hiring tools. These are vendors who are going to HR departments and saying, look, you're getting hundreds, maybe a thousand applications for each open position. There's no way you can wade through all those resumes. So instead have candidates use our AI and upload a 30 second video where they're talking not even about their job qualifications, but about their hobbies or whatever it is. And the AI tool is going to use their body language and facial expressions to pick up their personality traits and how good they're going to be at a particular job, and to this day, there is no evidence that anything like this is even remotely possible.

[30:37](https://youtu.be/ZxlaFCT4kpY?t=1837) And we strongly suspect that these kinds of products are just elaborate random number generators, basically. Now, we also talk about types of AI that are certainly conceptually possible, but so far no one has managed to pull off, so those are things we should be careful about. Often the evaluation of the AI product is completely missing or faulty or misleading in some way. Many vendors will say we have a 97% accuracy or whatever, but you really need to drill down into what dataset was it evaluated on, what were the false positives versus false negatives to be able to understand a claim like that. And I think that's due diligence we should all exercise and not necessarily believe the oversimplified claims that we're hearing. One thing about AI, as opposed to a lot of other kinds of software, is that what works for one organization simply might not work for another, or even one individual versus another. I know I made this point earlier, but I want to emphasize it again. Compare it to, let's say, Zoom.

[31:37](https://youtu.be/ZxlaFCT4kpY?t=1897) So let's say it's a few years ago, right at the start of the pandemic, or even before that, your organization is not using video conferencing and you want to consider whether to use video conferencing or not, you can use Zoom. Whether to use video conferencing or not, you could talk to other organizations who are using it successfully and ask them how it's working out for them or what to watch out for, but that doesn't work with AI. I think the benefits and downsides are so individualized that there is just no alternative for piloting it in one's own organization and seeing how well it performs, there are lots of stories. And for instance, in medical AI, where a company evaluates the performance of a medical AI system to be so much, but then when you go deploy it in one hospital versus another, or one patient population versus another, it performs much more poorly, and obviously this can come at a really high cost when you're talking about automating medical decisions. Okay. So those are flaws with AI tools themselves.

[32:37](https://youtu.be/ZxlaFCT4kpY?t=1957) There are also many cases of flawed deployment. Recently, the Federal Trade Commission reached a settlement with Rite Aid. And if you read that complaint, you know, there are lots of things that went wrong here, but I don't think any of them are the fault of the AI vendor. They're all the fault of how Rite Aid deployed this technology. A lot of people will say facial recognition is too error prone and too biased. I think this used to be true 10 years ago. I think today the technology has improved dramatically. Certainly if you deploy it in careless ways, such as in the ways in which Rite Aid did, then it's going to be error prone and discriminatory. But I think that's a deployment issue. It's not an issue with the product itself. And then particularly in public sector deployments, we really have to be careful about, do we have due process if we're using AI to serve citizens or make some decisions about citizens in ways that could potentially help or harm them?

[33:37](https://youtu.be/ZxlaFCT4kpY?t=2017) There was a scandal in the Netherlands where there was an algorithmic system for spotting welfare fraud, and I think it falsely accused something like 30,000 families of fraud and even led to some suicides and stuff, and this was such a big scandal that the prime minister and the entire cabinet resigned. There are many other stories like this, including some in the U.S. And the British post office scandal, that's not AI, it's just an automation failure, has been so deeply problematic that there is even a TV show about it. And one pattern we've noticed with AI vendors is that when they're pitching the product, they talk about how it's going to be able to automate everything. But then the fine print says there always needs to be human oversight. And in a lot of cases, if a human is going to be in the loop and kind of has to verify every AI decision, then it's not even clear if automation is actually saving costs, so those are things to be mindful of.

[34:41](https://youtu.be/ZxlaFCT4kpY?t=2081) OK, I said I'll talk about AI and law in particular and the claims of replacing lawyers, which are, you know, you might think are ridiculous claims, and I agree, but these claims have often been so persuasive to a lot of people. So I do want to spend a minute on this. Yes, Janchi Pitti has passed the bar exam in some sense of the word, and there is even this company, have folks heard about this, Do Not Pay, claiming that they will pay a million dollars to any lawyer who uses their lawyer to argue a case in front of the Supreme Court, like, first of all, electronic devices are not allowed in the Supreme Court. A person knows this, this is just a publicity stunt, and they got into trouble with the FTC, as they should, for the unauthorized practice of law. But more importantly, the reason this went viral, the reason so many people, you know, thought this was at least plausible is because I think they have mistaken ideas of what AI can actually do in professional settings, whereas when you actually look at examples of lawyers trying to use AI for meaningful things, we have so many disaster stories.

[35:47](https://youtu.be/ZxlaFCT4kpY?t=2147) By the way, Zoom is telling me my Internet connection is unstable. I apologize. I'm on a good university network. If you have any serious quality issues, please interrupt me. Let me know. But I hope it's OK so far.

**Beth Simone Noveck**

[35:58](https://youtu.be/ZxlaFCT4kpY?t=2158) We hear you loud and clear.

**Arvind Narayanan**

[36:00](https://youtu.be/ZxlaFCT4kpY?t=2160) OK, wonderful. Thank you, Beth. So we have one or two papers and essays about this. So as I said before, lawyers' job isn't to answer bar exam questions. So there's a big gap between how these products are evaluated and what we want to use them for. There are evaluations that are led by legal professionals, and those show a more nuanced picture. I think it's more about augmentation rather than replacement of what lawyers do. OK, so there is, let me talk about maybe two more pitfalls, take a minute to talk about each, and then I have one concluding thought to share. So you often hear the claim that models are going to keep getting better indefinitely. I do think AI progress is going to continue, but it's very much up for question whether you're going to see just the huge qualitative improvements we've seen in the last few years when you compare, let's say, GPT-4 to GPT-3 to GPT-2 or something like that. So it's true that there are so-called scaling laws that tell us that models get better as we make them bigger and train them longer on more data, but those laws actually refer to how good the model is at predicting the next word.

[37:13](https://youtu.be/ZxlaFCT4kpY?t=2233) Those laws don't refer to how well they can do tasks for us. That's still very much an open question. My hunch is that scaling is very much going to slow down or the returns from scaling are going to slow down. And further, scaling is a business decision. Just because companies can make models that are 10 times bigger than today's models doesn't mean it makes business sense for them to do so because these larger models are also harder to monetize. And we've seen this before where a trend was increasing exponentially and then just suddenly flatlined. My favorite example is CPU clock speeds that you see on the left here. So the y-axis is a logarithmic scale. So this straight line is actually Moore's law. It's an exponential curve. So doubling every couple of years or so. That happened until the mid-2000s. And then suddenly CPU vendors realized that making the clock speed faster is actually not helping the product quality. They just decided to stop competing on that dimension. Right, so it could very well happen that companies at some point stop deciding to build bigger and bigger models.

[38:19](https://youtu.be/ZxlaFCT4kpY?t=2299) OK, one last thing to be careful of is what we call the capability-reliability gap. And this happens especially with AI agents, which are a topic with a lot of hype. And as you can see in this example, these agents are capable of doing interesting, complex things like automatically ordering food for you and selecting a restaurant and so forth. But they make mistakes some fraction of the time. Now, if traditional Siri, you know, play the wrong song 10 percent of the time, that's annoying. But if this AI agent is going to order food to the wrong address 10 percent of the time, that product is dead on arrival. Right, I blacked it out because it doesn't matter which product it is. This is a pretty general phenomenon. And you can see in this example here. Yeah, how things went wrong in this particular case, but this is not an isolated example. So I think when you're evaluating one of these agents in particular, it's important to ask not only what is it capable of, but how reliably can it do that task? OK, here's the last closing thought I want to give you.

[39:21](https://youtu.be/ZxlaFCT4kpY?t=2361) This is kind of more philosophical, not at a practical level of which product should I use or something like that. I think every government agency has to undertake a careful balancing act in terms of how quickly to embrace AI. I think both ends of the spectrum are very risky. If you have hasty adoption, you get into the situation like the New York City's AI chat bot. This was an investigation by the markup, I believe, which found that the bot was misfiring and telling businesses to break the law. We don't want that. But at the same time, if we're so risk averse that we do a whole bunch of impact assessments before we try to do anything with AI, we end up in a runaway bureaucracy situation. And a couple of notable scholars have been raising the alarm about this. You can look up this op-ed. There's also other references I can give you. And I think getting this balance right is really, really important. And related to that, there's this really nice book by Jeff Ding, who I believe is at Georgetown. He talks about great power competition.

[40:27](https://youtu.be/ZxlaFCT4kpY?t=2427) And his point is that while we often think about technological advantage and great power competition in terms of who is able to innovate more, he says what actually matters is who is best able to use existing innovations and diffuse them into the economy, right, both in the private sector and in the public sector. And what generative AI has shown that while innovation is happening very fast, we're not able to adopt those technologies so quickly. There has been a paper making the rounds recently claiming that generative AI is being adopted rapidly, but I very much dispute that. Happy to get into that in the Q&A. I do think diffusion is happening much more slowly than innovation. So I think the big philosophical question for all of us, in particular for the government, is are we going to be able to try and close that gap or will that gap keep increasing further? So that's all I had to say. I hope you found these three parts of the presentation interesting. Thank you so much. And I look forward to your questions.

**Beth Simone Noveck**

[41:23](https://youtu.be/ZxlaFCT4kpY?t=2483) Fantastic. Thank you so very much. Again, I'm sure we are not going to have enough time given the number of questions we already have in the chat. So I'm putting a link to the full book and the discount code in the chat. But we've had a number of questions come in and so a number of them are sort of very practical to start with. So I'm hoping you can really try to respond to the questions we have from Naman about things like reducing hallucinations. So he asks about how he reduced hallucinations while trying to extract a system map out of a policy case study, for example. So I think that question comes up a lot from people, as I'm sure you've heard. And there's a number of other questions that are coming in. So let me ask you these at the same time. Marty asks about the example of entering first and last names into things. So the question about privacy. So as people are trying to navigate their way between those two ends of the spectrum, what advice do you have about the privacy and sensitive information questions?

[42:28](https://youtu.be/ZxlaFCT4kpY?t=2548) And what advice do you also have on trying to limit hallucinations? And if you have a question, given the short time we have, I'd ask you to just write it in the Q&A, excuse me, in the chat.

**Arvind Narayanan**

[42:40](https://youtu.be/ZxlaFCT4kpY?t=2560) Great, thank you, Beth. Yeah, privacy as well as hallucinations are two of the biggest risks. Let me address privacy first. I think the good news is that as far as I know, every AI chatbot vendor at least has an option to not use the information that is entered for training purposes. I think it's important to take advantage of those options. But also, if you're negotiating a contract with a company, I think that should be at the top of the list of things that you look for in that contract. And finally, I think with open source models, you have the option of running it yourself, and that can be another important privacy enhancing measure as well. So on hallucinations, there are a few things to keep in mind. One, I think as these models have scaled up, the rate of hallucination has gone down. It's unlikely to go to zero, but things are improving. In terms of what we can do, what's in our control, I think more kind of agentic workflows where a bot answers a question and then looks back at its own answer and tries to correct itself.

[43:49](https://youtu.be/ZxlaFCT4kpY?t=2629) Those things can further drive down the rate of hallucination further. I think in a lot of cases, we're going to need innovation where we try to figure out what is the most time efficient way in which we can have human oversight over the output of AI instead of looking for purely automated ways of trying to drive the hallucination rate to zero. I don't think that's going to happen. I think the question is going to be, how can we spot hallucinations with the minimum of human effort? And there are many ideas along those lines. So, for instance, you can have AI generate a confidence score. Are you highly confident of this answer? Are you somewhat confident? And then a person comes in and looks at the ones where it's not so confident. So, for instance, that's one technique that can help. There are no perfect solutions.

**Beth Simone Noveck**

[44:37](https://youtu.be/ZxlaFCT4kpY?t=2677) So, Mark is asking a question in the chat about disclosure of the use of AI with the prevalence now of AI. So, a year ago, a number of policies came out, including the one we wrote in New Jersey, the one the city of Boston wrote that said you always have to disclose it. But now, with it being on your phone, with it kind of being everywhere, do you still think that sort of disclosure model?

**Arvind Narayanan**

[45:00](https://youtu.be/ZxlaFCT4kpY?t=2700) Disclosure is usually thought of as a way to have accountability for a particular output, a particular document, let's say, that an agency might have prepared. And disclosing AI is maybe a way to let citizens know there might be mistakes here or to alert them or something like that. I think that's okay. I think it's more important for the creators to actually take responsibility for what's in there, to have manual checks and so forth. To me, disclosure serves a secondary purpose, which is often not considered, which to me is actually more important, which is to let people know, in general, how much of the information they encounter was AI generated or enhanced or something like that. Because people don't have, I think, a good idea of how quickly this is happening. So when people see images on social media, I think they're still too trusting, whereas we always need to be having the thoughts, could this image have been AI generated, and it's going to happen with videos as well.

[46:21](https://youtu.be/ZxlaFCT4kpY?t=2781) And so seeing lots of those disclosures on an everyday basis, I think, will flip something, flip a switch in people's minds where they always remember to be skeptical about information and to try to consider whether the source of the information is trustworthy, as opposed to

**Beth Simone Noveck**

[46:36](https://youtu.be/ZxlaFCT4kpY?t=2796) assuming that if something looks realistic, that it was created by a person. I want to let everybody know that Arvind has to go teach in 10 minutes, so I want to encourage you to put any additional questions into the chat. In parallel, Agata is going to put in a quick survey about this, and we welcome your feedback on this workshop, as well as future workshops you'd like to see. But let me go to Mark's question, which is in some ways the flip side of this. Actually, I'm sorry, Steve's question, which is the other side of when people don't disclose the use, are there AI detectors that are actually reliable? Do you believe in them? Should we be using them? And I've heard all kinds of stories, including people doing interviews with candidates who seem to pause very pregnantly when they respond, because you can tell they're asking their little helpful chat GPT or Claude to write their responses even in real time. So curious what the thought is on the state of the art in AI detectors.

**Arvind Narayanan**

[47:38](https://youtu.be/ZxlaFCT4kpY?t=2858) Yeah, thank you. AI detection could refer to a whole bunch of things. One is there is a piece of text. Can you use AI to figure out if that piece of text was AI generated? That doesn't work, at least not as of yet. Maybe it'll start working in the future, but right now there are so many abuses of the kinds of products that claim to do that, especially in a university setting. So many students are getting falsely accused of having used AI. And it also has a higher rate of flagging non-native English speakers, for instance. So there's lots of problems with this. Another thing it could mean is watermarked text or images. So there are kind of human invisible signals that are present in a piece of text or media so that the same company that put in those signals can create a detector. And those are much more robust. They're not infallible. But the downside is those are not available to everyday users. So that's for specific purposes, but it's not something we can generally rely on.

[48:42](https://youtu.be/ZxlaFCT4kpY?t=2922) A third one is even if you can't tell if a specific piece of text was AI generated, if you are looking at a social media account, you could look at a pattern of behavior over a long period of time and try to detect if this is a bot or a real person. And that is much more feasible. And that's much more feasible, A, because you have more text and images and interactions and so forth. And B, you don't just have to go based on the text itself. You can also look at other patterns, like are they posting at every hour of day? Do they seem to never sleep? So I think that kind of bot detection, I still have hopes that it can be done in a lot of cases. And finally, there's always a cutting edge in terms of trying to look for forensic patterns, if you will, in images. And these are the kinds of investigations that many journalistic organizations have now. The New York Times, for instance, have a visual investigations team. Does this image that claims to be of this public place at this point in time match with other images that were taken at the same place at that point in time? And I should mention one more.

[49:49](https://youtu.be/ZxlaFCT4kpY?t=2989) There are tools like C2PA where cryptographic provenance authenticators are inserted into the image or the piece of content itself. So there are a bunch of ways of doing this.

**Beth Simone Noveck**

[50:00](https://youtu.be/ZxlaFCT4kpY?t=3000) Some of them work better than others. So you've been very helpfully, extremely practical in this conversation, but I want to invite you to spend a few minutes as we wrap up to kind of bring, come back to some of the big ideas in the book, including the notion that AI snake oil, that these false promises bring out the worst flaws in our institutions. You already gave us that jobs and hiring example. I'm wondering what you see as being the dangers of the intersection between the worst of government and the worst of AI, and conversely,

**Arvind Narayanan**

[50:34](https://youtu.be/ZxlaFCT4kpY?t=3034) the best of government and the best of AI. Yeah, definitely. I mean, the impetus for using, whether it's AI snake oil or any other dubious technical tool is when budgets are getting cut, right? So you're trying to do the same thing with a much more limited budget, and it's very tempting to automate your way out of the problem. And it's kind of similar in the hiring example there. It's less of a budget issue, but then you have too many applications for each open position. And we observe this over and over again, and we say in the book, broken AI is appealing to broken institutions. We also talk about social media, for instance, where there is a crisis of trust, a crisis of legitimacy, where people are asking these social media companies who have set themselves up to be the arbiter of the public square, how are you going to remove bad stuff, if you will, and make social media a welcoming place while also protecting freedom of expression? That's very hard. There are no right answers. So it's very tempting for social media companies to say, oh, we're going to objectively draw that line using AI. But to us, that makes no sense. It's not going to work. It's only going to deepen the crisis of legitimacy.

[51:45](https://youtu.be/ZxlaFCT4kpY?t=3105) And so when we see these broken institutions, our answer should be to try to fix the institution, no matter how hard it is, instead of trying to put a Band-Aid on it. In terms of the best of AI intersecting with the best of government, the bad part is when budgets are being cut, but there are good and thoughtful ways to see what new public services can be provided that couldn't be provided before. So it's, again, a little bit of that automation versus co-creation distinction. Instead of just seeing how can we provide existing government services at lower cost, which is also important, but can lead to some bad incentives, we could try to think, what new services could we be providing that are very hard to provide today? So simplifying very complex documents, for instance, with the help of AI. It's hard to give concrete examples of that, because by definition, it's a new thing that we don't have a name for, we don't have a category for. But those are the kinds of innovations that I'm most excited about. And I know Beth, one of our colleagues, Peter Henderson, is working with your office to try to find new innovative use cases. So I'm excited for all of those.

**Beth Simone Noveck**

[53:02](https://youtu.be/ZxlaFCT4kpY?t=3182) Absolutely. Let me thank you as we wrap up here by reminding people of the link to the book and where they can get a copy. And we'll see if there's any last thing you would like to share about what people will find in the book beyond what we've had time to cover in the 60 minutes. It might be nice to get your quick take on what else is in there that they will find useful, and that they might want to read for.

**Arvind Narayanan**

[53:33](https://youtu.be/ZxlaFCT4kpY?t=3213) Sure, certainly. We talk about AI risks a lot, and in particular, potential catastrophic risks in the future that we should be thinking about. And our point of view on this is, yes, we should be thinking about them. But we don't think this requires dramatic policy interventions, for instance. And we have a chapter in the book explaining our view on this. And we also have an upcoming paper called AI as Normal Technology. And what we mean by normal technology is, now that it's not transformative, even technologies like electricity are normal in our conception. But we think for the foreseeable future, AI is going to remain a tool that we can and should control and shape in ways that societies want AI to behave, as opposed to some parallel human-like separate species that's going to create its own kind of organizational structures, which are the kinds of visions that a lot of tech companies are selling. And we think that's both unrealistic and also problematic to try to pursue.

[54:36](https://youtu.be/ZxlaFCT4kpY?t=3276) So that's one thing. Maybe I'll mention one last thing. We talk about AI in kids. So for someone who is a toddler today, for instance, AI is going to be such a huge part of their lives. And this is true of my kids. They're five and two. In a way, that's qualitatively different, I think, from the role of AI in our lives. And the influence of AI in our kids' lives could be enormously positive, or it could be addictive and negative, or anywhere in between. We have a lot of agency. I think we're not thinking about them as a society, as policymakers, as parents. And I think for social media, we were too late to recognize both the positive and negatives when it comes to kids. And I think for AI, we need to start thinking about them now, as opposed to after the issues are already serious.

**Beth Simone Noveck**

[55:25](https://youtu.be/ZxlaFCT4kpY?t=3325) So last question with our three minutes. And this comes from the chat here. I'm going to give credit where credit is due in a second. It's from Anuradha, who asked the question about how do we avoid, instead of moving fast and breaking things, how do we move fast and fix things in the world on this very fraught day after the election with a lot of complex emotions? Any

**Arvind Narayanan**

[55:51](https://youtu.be/ZxlaFCT4kpY?t=3351) closing words for us that are hopeful? Sure. I'll go back to the thing I said about finding the right balance, right? So I think speed is important. But yes, we can overdo it. We can hastily adopt technology. But on the other hand, there are also, if I may get, for my closing thought, maybe a little bit controversial, but I am channeling Nicholas Bagley, who's written a lot about this. He's written a paper called The Procedure Fetish, for instance. And he's a critic of environmental laws in particular, where he says that the way it's implemented is too much upfront assessment of what could go wrong. And I think that is also true in AIs. Often, there is too much of a focus on figuring out upfront what could go wrong. And in my experience, a lot of the time, we actually have to pilot the technology. We have to start using it before we actually have good information on what the risks are. And therefore, while we should be careful respecting rights and so forth, we should not operationalize that through inaction by waiting for certainty about the safety of AI technologies. But we should instead operationalize that by being able to learn rapidly from experience as we scale up.

**Beth Simone Noveck**

[57:10](https://youtu.be/ZxlaFCT4kpY?t=3430) Arvind, thank you very much. We will not make you late. And thank you to all of you for spending an hour with us. Please come back and share these opportunities with your colleagues. We are here, again, to move fast and fix things by learning how to work better and more effectively using new technology for the benefit of the residents we serve. So thank you. Congratulations on the book. And thanks to everybody for coming. Agata, thank you for your help in setting this up. And we'll

**Arvind Narayanan**

[57:36](https://youtu.be/ZxlaFCT4kpY?t=3456) see everybody at the next one. Thank you, Beth. Thank you, Agata. Thank you, everyone. See you later. Thank you. Take care. Bye-bye.

* * *

Original page: https://innovate-us.org/ai-snake-oil-what-artificial-intelligence-can-do-what-it-can-t-and-how-to-tell-the-difference · Licence: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org
