---
id: "workshop:defining-problems-with-data-june-30"
type: "InnovateUS workshop"
title: "Defining Problems with Data"
url: "https://innovate-us.org/defining-problems-with-data-june-30"
date: "2022-06-30"
language: "English"
instructors: ["Eric Reese"]
recording: "https://www.youtube.com/watch?v=C5QyBfCZPp8"
transcript: "automatic, from the recording"
source: "InnovateUS (innovate-us.org)"
licence: "CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org"
---

# Defining Problems with Data

InnovateUS workshop · 30 June 2022

Join us to hear from Eric Reese, former director at the GovEx academy, about how data can be used to help define and solve problems. You will walk away with an understanding of how data can be used to help us identify and define our problems, as well as strategies for approaching and analyzing data.

## Learning goals

- Describe how data can be used to help define and solve problems
- Understand how data can be used to help us identify and define our problems, as well as strategies for approaching and analyzing data

## Speakers

- **Eric Reese**, Former Director, GovEx Academy

Recording: https://www.youtube.com/watch?v=C5QyBfCZPp8

## Transcript

Automatic transcript of the recording: names and terms can be misheard, and the times can be a few seconds off. Each time links to that moment of the video. The transcript alone: https://innovate-us.org/defining-problems-with-data-june-30.txt (plain text) and https://innovate-us.org/defining-problems-with-data-june-30.json (every segment with its start and end second).

**Audience**

[0:01](https://youtu.be/C5QyBfCZPp8?t=1) I'm Graham Simpson with the New Jersey Office of Innovation, and I've been working with Cassidy and other folks on putting together this Innovate US Innovation Skills Workshop, open to folks across a number of states that have been collaborating on innovation skills trainings. Eric, if you want to click to the next slide. This is part of this Summer Skills Workshop. Each of these workshops, we're hosting them every other week for 90 minutes, generally a 60 minute presentation and then 30 minutes of Q&A. As you all know, we sent out some advanced readings that Eric assigned to get everybody on the same page. That's true for all of these and we'll follow up with all of you after with some information on what we hope you got out of today's session. I just wanted to highlight that if you complete seven of the 10 that we're hosting this summer, you'll be eligible for a LinkedIn badge and the chance to apply to work with a coach to support you in advancing your project or idea. And on the next slide I just kind of want to highlight where we are in this series. So this is, we're kind of in part, we're in the second session of step one defining the problem.

[1:07](https://youtu.be/C5QyBfCZPp8?t=67) We have joined for our last session on human centered design and designing problems with people. This is really another side of the same coin, how do we think about public problems and using data. And so some of the other upcoming sessions are up there. And at the end, I'll sort of highlight very quickly, how you can sign up for those and what some of the upcoming sessions are but we're at a good number of participants now in 202 so Eric I'm going to turn it over to you. I'll let you introduce yourself but Eric's a great presenter I've seen I've seen Eric present on these topics before and really excited for what he has to share with us today.

**Eric Reese**

[1:42](https://youtu.be/C5QyBfCZPp8?t=102) Awesome. Well, thank you, Graham and thank you all for for being here. It's a really impressive series of workshops I'm really excited to be a part of this one today. For those of you that are participating and others I'm sure you're going to get some really great stuff and I hope today will live up to those, the highest standards that you are going to get for the rest of the time so I'm very excited about going through the topic today of defining problems with data, and in particular on data tools for government that you all can use, and to have some discussion with you all about ways to think about using data in your daily work. So, just a little bit to get us started so what we're going to cover today is a couple of different components. So one in addition to what Graham just gave you, we're going to take just a little bit of time to go over some logistics some goals, in particular some ways that I hope that we all can participate today beyond just me talking. We are going to do a little bit of reflection on the pre survey data so since I asked people to fill out a pre survey I like to go over it with folks as we as we go. And then the main bit of what we're going to go through today will be focused on patterns, using data frameworks to identify and manage data and then a couple of cases, just to show you how it could work in practice and how we might start into use in the real world. Before we end with some final thoughts and then we will definitely have time for q amp a at the end today. So that's the main structure for today.

[3:14](https://youtu.be/C5QyBfCZPp8?t=194) And what I want to start with is just a very brief background on myself so this is the bio that you all got sent just so that you have it here like you don't you can find it all in one place. But I have been working with governments, in particular state and local governments around the world for the past 10 years 15 years, mainly through some work that I did at Johns Hopkins University with a variety of city governments in the US as well as state governments in the US and outside. I've worked with the OECD I've worked with the open contracting partnership and a variety of NGOs and foundations all around the world. And I really love talking about data, I love talking about how folks can learn and use more data. And in particular, I love facilitating workshops like this this is one of my favorite parts of my job so I'm really excited to be here with you all today, to answer questions about this kind of things. As you all have questions about my background and experience, fire away in the chat or the q amp a, and we can get to some of those for sure.

[4:16](https://youtu.be/C5QyBfCZPp8?t=256) Probably more directly relevant for all of you today is what we're going to cover and so these are my three goals for us today. So I want to make sure that we cover how to describe typical patterns of data use in government, some ways to think about data use. I'm going to cover some identifying data sources that can be used in your daily work, as well as some issues related to data management and communication. And these are all things that have come up in my work over the years, and I mainly intend them as kind of food for thought for all of you to think about how they might apply in your work, how they might apply to direct things that you're doing right now in your day to day job, and how they might spur you to think a little bit differently about data, and how it helps you define problems. So how we're going to get there is a couple of things so first on the left side on ground rules. I hope you all can follow along actively and to do that, there is a link for the deck, which Cassidy or Graham can can put into the chat so you all have access to it.

[5:20](https://youtu.be/C5QyBfCZPp8?t=320) There's no wrong answers today so open questions like it's all good I encourage you all to to think broadly and openly. I also encourage you all to bring your own challenges and expertise, we all have particular things that we've worked on in our experience to be able to help others so I encourage you all to do that, as well as to listen and be inclusive and encourage others and finally, just to relax and have a little bit of fun as much as you can have fun with data I'm going to make it as fun as I can, and participate as much as you're able. So on the right side on logistics there's a couple of things as I mentioned the deck to follow along with the slides, but the other thing that we're going to use today is called Slido, and that's where we're going to have all of our q&a. We're going to upvote questions from others in there and we're going to have a couple of polls, as well just to kind of feel out the crowd a little bit as we go through today. If you have logistical problems you can send Graham or Cassidy a message in the chat sorry for leaving you out there Cassidy. So we're going to have that and we'll try to get to it ASAP. And the last thing, although you know we're all old pros at this by now to just a reminder to mute yourself when you're not speaking to for the benefit of others.

[6:31](https://youtu.be/C5QyBfCZPp8?t=391) So with all that said, with the logistics going on what I want to do first is a quick level set in terms of how you all feel about your own knowledge of data. So what I'm going to stop sharing for just a second, so that I can get to. So I can get to the Slido and launch our poll here. So Graham or Cassidy if you all can fire that into the chat. Hopefully folks will be able to get to the Slido link and I'll see folks interacting and I'll share my screen that's the Slido here in a second. So what we're seeing here on the, on the Slido this is the first poll question how do you all feel about your own level of data knowledge I encourage you all to answer so zero I know nothing. One I know a little, two a lot, three I'm a data master, and however you all feel like you want to classify yourself, go for it. This is what it should look like for for all of you.

[7:36](https://youtu.be/C5QyBfCZPp8?t=456) So I'm going to answer, I feel like I know a lot but not everything. And so this is the the results that we're seeing so far so thank you all for for participating those of you that have no judgment from me for those of you that might be at zero, there was there certainly room for that and I think we're going to have great tips for those of you that do feel like you don't know much, but also for those of you that are the data masters or there might be at the two or three level. So I encourage you all to participate actively and add your thoughts both in the q amp a and in the chat with others. As we go through today. So what I'm seeing here is like people feel, and this is generally the way it goes and the way I try to set it up is to force you to not pick something in the middle so one or two is usually the way it goes. So I've identified a zero or three and I appreciate you all bringing that and most folks identifying themselves as knowing a little about data with about a third, thinking that they would consider themselves to know a lot so that so that's great.

[8:38](https://youtu.be/C5QyBfCZPp8?t=518) And I really appreciate you all sharing that with me it's helpful for me just to get a kind of sense of where people are so that when I'm going through examples. The other thing that I'll highlight before I really dive in is here on the q amp a, you can feel free to type your questions in here, it's all good and and as I said you can upload other questions like anytime throughout our session today and Graham and Cassidy and I will be monitoring that so we can get questions answered. So, with that, thank you all for for your poll answers I will stop the poll here in a second. And I'll go back to the slides but I just wanted to kick that off and also let you all have a chance to take a look at the slide Oh, before we really got going today. So let me go back here and share my screen again to get back to the deck, and we'll be good to go. All right. So, as we go today what I want to do to really kick us off is come back to your all's experience, and in particular, you know what we started with on the pre survey, so I know some of you all responded and maybe not all of you got a chance to and that's okay. But there are really only two questions on the pre survey. So one was when was the last time you use data. And the other was when what data did you use and what you use it for.

[9:57](https://youtu.be/C5QyBfCZPp8?t=597) So I wanted to just to provide some context for me, and looking through those answers can provide us a little bit of insight so when I looked through the answers to those I tried to think about a couple of things. How can I group and categorize the examples. What examples and uses are similar, and can we draw any similar conclusions about those use cases. So these three questions I want you all to think about as we go through today as well because it's similar to what I'm going to be talking about in terms of data use in terms of categorizing use cases and examples that are similar. So first, when was the last time you use data. The majority of folks almost 75% of people said this week or within the last two weeks, which is really good that's encouraging I'm glad you all are diving into data regularly 7% within the last month, and then 19% can't remember. That's really cool. We often I think most of you probably have used data but we are so awash in data we're using data all the time we don't even realize it. So, I am glad though that many of you, you know, three quarters, almost have said you're regularly using data this week or within the last two weeks.

[11:10](https://youtu.be/C5QyBfCZPp8?t=670) So what I did for the second question was, what data did you use and what you use it for. I basically picked out from the free text, all the verbs that people were using in those answers and did a little bit of a word cloud about frequency. And so the most common ones were reports and monitor plan evaluate analyze, we're kind of close behind but reporting and monitoring were a huge huge component of what folks said they were using the data for. And then a couple other things like setting goals, potentially prioritizing. And I'll add that to the list. After today. So thank you all for those of you that did answer the pre survey and again it's helpful for me in particular to think about the uses of reporting and monitoring and how that might play into how we use it to define problems as well.

[12:20](https://youtu.be/C5QyBfCZPp8?t=740) So as we go forward today, we're really going to kick off here with patterns and problems. And so I always have to start off all my presentations with a question and for us it's, how can data be used to help solve problems, and in particular starting off with identifying problems. One of the things we often do that I want to kind of put out there at the beginning, is that we shouldn't be starting with our data, we should be starting with the problems that we have, and identifying issues that we need problems that we need to solve, and then adding data and seeing how data can help us define it more clearly. How can data help us identify potential solutions, but in particular I want us to note, and kind of hit the point home really hard, solving problems and identifying problems comes first. And this is actually leads into the kind of three prongs that I'm going to talk about today, which are using data, managing data and then communicating with data.

[13:23](https://youtu.be/C5QyBfCZPp8?t=803) One thing that I'll point out for all of you that these things you'll see the arrows are kind of double sided. They all interact with each other, there's no linear problem solving pattern that's going to give you the answer for using data with your problem, we might be defining a problem and then looking at the sources that we have that might lead us to refine our problem, we might see a map that might lead us to look at the data a little bit more clearly, all these things are constantly interacting with each other, and there's no linear process so keep that in mind as we go through today as well, that we're going to have to be doing some back and forth as you do this and that's all okay. So, to start today, we're going to, we're going to look at using data. First, in particular the patterns problems and analysis that you all might use to help define those problems and help solve the problems eventually. So, as I said, at the very top kind of problem definition comes first, and then data. So this is a screenshot from the New Jersey innovation skills accelerator which I was a part of as well.

[14:31](https://youtu.be/C5QyBfCZPp8?t=871) And in this kind of module which which the folks from New Jersey obviously might be familiar with but others as well. There's a pathway that Beth outlines that I really really like for problem definition so it starts with I know it's really small on the screen. From issue to problem, then articulating root causes, identifying those most affected and stage for building an evidence base. What I'm going to talk about today is mainly in stages one and two, and particular kind of, I would say like stage one and two. Page 1B, 1A, 1B, something like that, to really help us use data as a part of going from issues to a problem definition. And as we get to our articulating those root causes, so what we're going to do today is think about the kind of patterns that can help us to go from issue to problem and clearly define things maybe a little bit more easily than we had in the past. So one of the pre-readings that I had was about the London Broad Street cholera outbreak in 1854, not the most topical, super relevant, you know, from a timing perspective article, but I found that this issue of what happened from John Snow, you know, unrelated to Game of Thrones, John Snow in 1854, found about studying root causes of a cholera outbreak.

[15:56](https://youtu.be/C5QyBfCZPp8?t=956) So his basic analysis was that he was trying to find out where cholera was happening and why and determining that the cause was contaminated water and not air particles. And so this is really interesting to me for a couple of reasons. One is that you see this map here. The map was a key to his work. And then like right here in the middle, this was a water pump that he found. And you see these kind of cluster of cases that he made right near there. And the reason I find this really interesting is that he did a couple of things in terms of using the data to define the problem. First, he started with the problem. The problem was people are getting sick with cholera. Second was why? Why are they getting sick? Then he started using his data to answer that question. It seemed like the cases were clustering around water pumps in particular parts of the city, and he used this map to help communicate, but both to determine the problem, but also to communicate to others that water might be the source of the problem. It also is interesting to me because it required him to kind of go against the countervailing theory of the day that the prevalent theory of the day was that we got sick because of things in the air and not anything to do with germs.

[17:10](https://youtu.be/C5QyBfCZPp8?t=1030) Germ theory wasn't a thing at this point in time, but he was able to define a problem, use some data to help him define it further, and design potential solutions, which in this case was super simple. There was a water pump. They took the handle off the water pump. People couldn't use it anymore, so they stopped getting cholera, at least in that area. So this use case, despite its age, I think continues to be relevant because of its simplicity and the lessons that we can learn from how we use data to define problems. It's also really interesting to me because he was combining different data sets. He was using a map. In this case, maps are really great visualizations to help us detect patterns. And so I encourage you all to the extent that you can to think about how you can visualize your data to help detect problems as well and define your problems a little bit more clearly. The idea, though, of this kind of mapping and what I wanted you all to get out of this example is that it's kind of a kickoff to the fact that there are patterns in our problems and not just using maps or visualizations, but we can categorize them.

[18:14](https://youtu.be/C5QyBfCZPp8?t=1094) So this is one that I really like that came from the Harvard Kennedy School that just gives you a couple of taxonomy, if you will, of potential types of problems that we see in government. So prioritizing, for example, we're trying to enable more efficient use of resources. We might want to plan for a range of possible outcomes and do some scenario analysis and on down the line. But the idea is that kind of topic agnostic. These are the kinds of things that government and policymakers are trying to solve. They're looking at particular types. There might be others, but these are some some general main ones that we try to do. Another example that I really like comes from the city of New Orleans, in which they were looking at a very similar type of list, but they added a couple of other tools and they added some some other things to help people use it a little bit more easily, which I really like. So the ones that they added at the bottom there were kind of finding the needle in a haystack, optimizing resource allocation, early warning tools, and then what they call better, quicker decisions by narrowing focus and identifying subsets for their definitions.

[19:25](https://youtu.be/C5QyBfCZPp8?t=1165) So these problem types were common things that they saw across different departments, across different agencies, across different levels of government that they were working with. And the other thing that they did, if you click through on this link when you have the deck, you can see that they have these different problem types, but they also have examples for each one underneath. So I encourage you all to click through to the to the data driven analytics site so you can see this kind of examples and see what they kind of look like in practice, at least for the city of New Orleans, to inspire you to think about a little bit how you can use these different problem types and what types you might be facing in your daily work. The reason that I really like these problem types and this taxonomy is that for particular problem types, that leads us to particular uses of data, particular types of data, particular types of analysis. So once we know our problem type and what type it is, that can actually help us figure out what to do a lot more quickly than if we were just trying to pick out of thin air.

[20:27](https://youtu.be/C5QyBfCZPp8?t=1227) So I encourage you all to think about problem types in your work and to look at these examples and see whether they fit or not the kind of problems that you're facing. And mainly, I mentioned all these things to think about, in particular in government, we think of using data to solve problems like it's some really, really complicated algorithm. We hear about algorithms, data science, analytics, like, and we feel like we have to come up with the solution from scratch every time. But the reality is that most of the things that we're doing are pretty basic in terms of their analytic capacity. And actually, if we define our problems well and think about the types of problems in the data that we have, we don't often need to do those kind of really complicated things. Yet data science can be a really useful tool. Analytics can be a really useful tool. But we don't have to use them all the time. If we think about defining our problems well, we can get to action a lot more quickly in these kind of ways. So I encourage you all to think about that and kind of demystify the types of problems that you're facing on a daily basis by using these categories and by thinking about the type of data you have.

[21:34](https://youtu.be/C5QyBfCZPp8?t=1294) And determining your problem type and then adding data also allows you to think strategically about the data you're using and what type of data that your problem typically involves. So, for example, if we're prioritizing or we're optimizing resource allocation, for example, that is almost always going to include some component of financial data. We like to do cost-benefit analysis when we're doing resource prioritization. And so if we know, for example, that we're doing that, we know we're going to need financial data. We're probably going to need some statistical data as well and maybe some of these other types. But these are just some examples of if you know the type of problem first that can lead you in particular directions for the data that you want to use. And this is not an exhaustive list, but it is a way to think about categorizing your data as it relates to a problem type. Another way to think about categorizing your data is a little bit more back of the house.

[22:36](https://youtu.be/C5QyBfCZPp8?t=1356) So it might be administrative data. We have tons of administrative data in government that's often collected during service delivery. So we have people's names. We have their addresses. We have their ages. We have when they show up for programs. We have what kind of services they're receiving. We have all these kind of things that we just kind of gather that are about our programs. How many staff members do we have? How much does it cost to run a program? All those kind of things we have in government just kind of exists as administrative data. We also have actively collective data, which is what most people usually think about. Survey data, tax forms, or really any kind of form. Government forms obviously are a huge source of data. The census is actively collected data. So we also have those things that we're going at and seeking information that we can use. We also have passively collected data like traffic monitoring, CCTV or weather data from NOAA.

[23:39](https://youtu.be/C5QyBfCZPp8?t=1419) Like all those things are just happening in the world, and we're observing them and collecting that data somewhere. State governments do some of that. The federal government does a lot of that. Finally, structured or unstructured data. I only mention this because sometimes we might be using unstructured data, which just means it's usually not in a table. It might be a picture. That's the most classic example. I know when I lived in D.C., they had traffic cameras, red light cameras that took a picture of your license plate. This is unstructured data, but it's still useful for governments in particular to add structured data to later. The reason I'm going through these different types is to encourage you all to think about if you have a particular problem and a particular type, which of these types of data might be useful? Maybe administrative data, in particular if it's a budget or financial related thing. Those costs are going to be really useful. If it's a program, are we actively collecting data about participants? Is there census data that we can use to help us as well?

[24:41](https://youtu.be/C5QyBfCZPp8?t=1481) Are we passively collecting data? This often is the case for a lot of infrastructure related projects like roads or storm water or environmental projects. Are we passively collecting data? Or is anyone else collecting that data that we might be able to use? The reason I put these out there is just to get you all to think about a little bit more broadly the data you have, the data you have access to, and who might be using that data or have access to that data as well. So we're going to go to a poll really briefly to kind of break things up here. And mainly to hit the point of this section is that, does your team spend time defining problems? So let me get back into my side of this, of the Slido. I'm curious that the poll should be live now for all of you there. And I'll share that back in a second. I'm curious to hear from all of you. Does your team actually do this? Do you spend time before doing data analysis? Before really diving into things like defining problems? And I'll give you all a second to respond.

[25:41](https://youtu.be/C5QyBfCZPp8?t=1541) But I'm mainly just curious to hear what you all are doing. So I see most of you, this is what I expected for this one. Most of you say sometimes, often. But we have a good chunk for often and always, which I really appreciate. And for any of you, I want to just pause for a second on this and open the floor. For any of you that said often or always, are you willing to share what that means for you? Like how often, on always, and maybe a little bit about how you do that. I'm really curious to hear from the source for any of you that put that. So feel free to raise your hand if you want to speak. And if not, I'll just pause for a couple of seconds.

[26:53](https://youtu.be/C5QyBfCZPp8?t=1613) All right. Hearing none, OK, Roseanne, thank you so much. Roseanne, go ahead and go off mute. And fire away.

**Audience**

[27:03](https://youtu.be/C5QyBfCZPp8?t=1623) Hi. So I work for the Department of Transportation in New Jersey. And I work in traffic engineering. So we get a lot of requests to revise signals at signalized intersections. And they'll say, oh, you know, there's a lot of crashes. Well, before we go, you know, we go looking into the crashes. But before we do that, we actually define what the problem is. So we'll say, OK, well, you know, I'm sorry. I'm a little nervous. There's like 100 people. But we do, before we look at the data, we do define what the problem is. So if someone might come and say there's problems at this intersection, well, what are the problems? Are there crashes? Are people, you know, not looking where they're going? Are there, is it congested? So we try to define what the actual problem is before we go into the data. If it's just congestion, we'll go and get traffic counts. If there's crashes, we'll look into the crashes and we'll do crash analysis.

[28:06](https://youtu.be/C5QyBfCZPp8?t=1686) So we kind of have to define the problem before we decide what data we're going to use. That makes sense.

**Eric Reese**

[28:13](https://youtu.be/C5QyBfCZPp8?t=1693) Awesome. Well, thank you for sharing, Roseanne. And I think you did very well. No need to be nervous. We are among friends here. Thank you. But thank you for sharing. I think that's really helpful. And because for a couple of reasons. One, I really liked your point about we have to define clearly. And that means knowing what words mean. When we say it's a crash, well, what kind of crash was it? Where was it? What kind of intersection was it? These are all things that help us define our problems a little bit more clearly. And if we don't do that, we're going to set ourselves up for failure later, right? We're not going to have very clear analysis. We're not going to have solutions that people can actually act on. So I really appreciate you sharing that. And also because you are the representative of the silent 46% who says often or always. So I really appreciate that. And I also really appreciate all of you doing that work and thinking about it. And for those of you that don't do it regularly, I encourage you all to think about how you might be able to do that and build it into your process.

[29:14](https://youtu.be/C5QyBfCZPp8?t=1754) Because I think it'll help your data analysis work and in particular other things that you do going forward. So thank you again, Roseanne, for sharing. And I'm going to go back to my screen here. But for that part, the main idea for all of you is just to kind of think about how you're defining problems and how you're using data to really help you refine things as you go forward. And what kind of patterns you can use to help make that process faster, easier, and how you can rely on other people's expertise and past work to help you define and analyze your data. So that's part one. Part two, which... It's going to cover the rest of the, my part of talking mainly today. It's going to be about frameworks to identify and manage data in particular. So this is going to go into a little bit about communication, but, but mainly talking about issues you all want to consider, and again, food for thought, as you manage data.

[30:18](https://youtu.be/C5QyBfCZPp8?t=1818) And once you define your problems and you go to adding that data, what are things you need to think about? So in this section, we're going to talk about some sources, quality, privacy, and bias. Those are the main considerations I want you all to keep in mind as you all think about adding data to, and solving your problems. So first I want you all to think about where you usually get your data from. I mentioned a couple of different sources about administrative data, maybe forms, maybe the census. And this is also the screenshot here is another set of tools that my friend Carter here put together for others to think about, which are starting conversations, interdisciplinary projects, conducting data inventory, using open data or FOIA, surveying people. And I am going to focus particularly on number three, which is conducting a data inventory, using data governance. So oftentimes when I ask people, where do you get your data from?

[31:20](https://youtu.be/C5QyBfCZPp8?t=1880) They either don't know the answer, or they would say some sort of database or a spreadsheet, and that's where they're getting it from. But they don't necessarily have a sense of what all data they might have access to, or what all data they might use to help define their problem or even do analysis. And so I am a big proponent of doing data inventories to get as much metadata as you can and get as many data sources as you can, and literally just to list them out. And so a data inventory goes through these three steps. And I really like this resource and example, which comes from the city of San Francisco, in which they put together a very simple process, which is identify your data sources, then identify your data sets, then complete a catalog. And it's literally like a list on which they use a spreadsheet to have a list. So a data source might be a technology, it might be a system, a shared drive, an online application, and this might be a source of data for you that creates data sets or a database.

[32:27](https://youtu.be/C5QyBfCZPp8?t=1947) And the reason those are different is that a single database table or a worksheet might be a data set. And one spreadsheet might contain several tabs or several tables, and they might be their own data set, or you might be able to combine them to create your own. But identifying what those different data sets are is really important. And then also thinking about who can help you complete the catalog. And so this is particularly where I really like data inventories, because you need to go and talk to people and see where data is, what people have access to and where it lives and who's responsible for it. So data inventories can be really useful. They don't necessarily need to be like super exhaustive, but just going through the exercise of trying to identify data sources and data sets and listing them out can be really useful for agencies and departments as they think about using data to define problems and add in different data that might be useful as they go forward. This is just an example of what a data inventory might look like.

[33:31](https://youtu.be/C5QyBfCZPp8?t=2011) You might have the owner, nature, you might have the location, what it was about, any listed websites, what is the source and all these things that might or might not be useful for you, but just that could be used to make a list of as much metadata as you can. And when I say metadata, for those of you that aren't familiar, I'm just talking about data about data. You know, we have a data table. What do we actually know about it? And to go back to Roseanne's example, how are we defining the terms within our data set? That might be something that you want to look at as well. Oftentimes there are very technical definitions, but do you know them? Can you list them? Are they written down somewhere so people can react to them? So I encourage you all to think about data inventories and how they might be useful for you and how they might help you brainstorm different ways to use data to define your problems. As you go through data inventories and the first question that we always get when when people talk about data or using data to answer questions or define problems is about quality.

[34:36](https://youtu.be/C5QyBfCZPp8?t=2076) Is the data high quality? Can we trust the data? And when I think about data quality, I break it down into these five elements that are the most common. So it's accuracy, completeness, uniqueness, timeliness and consistency. Some other folks add different elements, but these are the five that I really like to help us. If those things are there, then we can be relative confidence that our data is of high quality. And I'm going to go through each of these really quickly for all of you just so you can see an example. So accuracy, according to the Data Management International Association, means that the data set you're reviewing should reflect the objects or events in the real world. So I took this example and this is a real example from an open data set that I found online. I will not name the source, but it's a real data set that has some issues. So this is a was an animal shelter database that's listing cats and dogs. So on there, there's a cat that's ninety nine years old and a dog four hundred twenty five months. I'm pretty confident that neither of those things are real because cats can't be ninety nine years old.

[35:39](https://youtu.be/C5QyBfCZPp8?t=2139) Right. So this data set might not be accurate. We need to go and check the accuracy of some of our data. The next thing that I want to talk about is completeness. This one's pretty, pretty straightforward to think about is all the data that should be in the data set there. Is it all present? Is everything there that should be there? So in this example, all these were tags from a three one one service. They have no created date. They were created, obviously. They were created as records, but we don't know when. And so this is a this is a problem for the completeness of our data set, which which makes it a little bit quality. Uniqueness is another thing that I'll use. So this is if data set should only be one one unique thing, they should all be unique. So a single view has a record or field that does not exist more than once. So there are some things that can exist more than once. So, for example, status can repeat, but we shouldn't have the same request ID for seven different requests. Right.

[36:39](https://youtu.be/C5QyBfCZPp8?t=2199) Like there should only be one request ID. So this data set is not necessarily great in terms of the uniqueness angle. Timeliness is another one that I think will be really common for you. Is this data relevant? Is it timely? So this data set from twenty eighteen. If you know, maybe it's not the data set to use. Here we are four years later. Maybe it's not the right data set to use and consistency. So storing records or data sets in the same kind of way. So you see all these date time formats are all different. That makes it a lot harder to use and a lot lower data quality in terms of this data set. So those are just some examples that I really like. And for all of you to kind of think about in terms of data management, is your data of high quality? Can you go through and answer these kind of questions and scan or or clean up your data sets to make them higher quality? That'll help us have more confidence and use data more easily. Another key issue that I'm going to talk about briefly in particular relates to if you're going to share your data with anyone is privacy.

[37:48](https://youtu.be/C5QyBfCZPp8?t=2268) What data are we collecting? Can it identify specific people? What are we sharing with others? Who are we sharing with? Do we have certain permissions about what data can be shared and what can't? How do we share it? It's a particular format for sharing better. And what do we do after we share it? How long do we keep data? Many places have archival rules. Are we following them? Those kind of things can really help us ensure that our data is of high quality when we're sharing it and that we can be confident that we're following policies and protocols well when we share data and not doing it when we're not supposed to as well. And finally, bias. So this also can help us make use our data better and think about thinking about bias and data to be more confident. So is our data bias, our underlying collection and policies, creating biased data? Are we selecting is our selection bias? Is it unrepresentative? Is there a confirmation bias that data has been selected to serve preconceived notions and are there confounding factors?

[38:48](https://youtu.be/C5QyBfCZPp8?t=2328) Data analysis shows a relationship, but a missing factor is actually driving the relationship. This is the most common example of that is correlation versus causation. And the classic every every course that's ever taught that in the world that I've ever taken anyway has talked about ice cream versus crime in the summer. Is ice both those things go up at the same time? Is ice cream causing a rise in crime? Of course not. But we need to think about confounding factors when we're looking at the data we're using and the data we're managing. So those kind of issues related to data management are really critical because they if we do have relatively good confidence about quality, privacy and bias, we trust our data more. It allows us to draw more conclusions and to use it more actively and communicate about it more actively to others. So I encourage you all to think about those things as related to your own data practices, your own data at work. Now, are you confident that your data is of high quality, that you're doing high quality privacy practices as well as checking for bias in your data to to really have a good, robust sense of trust in the data that you're using?

[40:00](https://youtu.be/C5QyBfCZPp8?t=2400) That will also help you potentially answer questions from other people that might ask you about the data analysis or the data you've used to feel really good about what you've done. And finally, I want to talk pretty briefly in this section about communication. So as I said, using, managing, communicate, they all interact with each other in a nonlinear way. But communicating with data is really, really important. And just throwing out a few things really quickly to talk about data communication and visualization, like it could be its own workshop. It is often its own, you know, year long course in graduate school programs and things like that. We're going to just cover the very, very tippy top of the surface today. But I hope to give you a couple of issues to think about when communicating with data around purpose, audience and tools, and mainly to think about as you're doing that problem definition and analysis and solving problems, you know, what story are we wanting to tell? How does data help or not help us tell that story?

[41:04](https://youtu.be/C5QyBfCZPp8?t=2464) And does our problem have typical ways of being communicated? And what I mean by that in terms of data communication is oftentimes if we're seeing, for example, something we're seeing a change over time, a line graph is almost always the best way to show that there are other ways. So if you think about common examples of what you're trying to show, there are many examples online of kind of visualization guides that can help you choose charts, for example, or choose different types of things to show on maps related to your problem. So thinking about kind of like earlier about the patterns is the kind of thing you're kind of problem you're addressing. Does it have a typical way that others have used to communicate it? So I encourage you all to think about those things. Second, in terms of audience, and this is also really critical, I'm sure you all have thought about this before, but just to reiterate, when you're using data to define problems, who are you communicating that those needs to? What are their needs? How do they understand or process data?

[42:06](https://youtu.be/C5QyBfCZPp8?t=2526) And in particular, for those of you that are doing reports or budgets to your bosses or or potentially to politically appointed folks or politicians, what are their needs? What are we what can we communicate to them and how will they understand? Data are really, really important things to help us kind of narrow in and hone in on what we want to say. And finally, tools. So how is the message delivered? What tools can help you and can you tell the story in multiple ways? In the pre-survey, many of you mentioned some tools like Excel, Tableau, some other database systems. Are those kind of things helping you communicate as well? Obviously, we can make visualizations in Excel. Maps are great tools because people really understand them well. But is the map a right tool for us right now? Like just these questions to kind of help you think about what you're communicating and who you're communicating with to tell a good story and to also help you potentially refine your problems as well. If we think about who we're talking to, does that mean we should think about how the problem is defined or what data we're using as well?

[43:09](https://youtu.be/C5QyBfCZPp8?t=2589) So hopefully I've reiterated that those things interact often enough to hammer home. And I really want to kind of end this with. At least this this active component right now with encouraging you all again to think about how these three elements do interact, how they work in your daily work. How are you defining problems using data and then communicating that back to others? And how is that informing your practice on a regular basis? And so I'm going to pause also really briefly here as well for our last poll of the day. And so I'm going to go back to ask you all to go back to the Slido because I'm going to turn off the last poll and go to the last one, which is, do you all have a data inventory? So let me share that back really quickly. Boom. So do you all have a data inventory? No. Yes. Maybe I'm going to put maybe because I feel like most people will probably say and go from there.

[44:10](https://youtu.be/C5QyBfCZPp8?t=2650) So. All right, we're getting some some folks saying yes, about third, third, third almost. Now, yes, maybe very cool. So a lot of you actually do have data inventories, that's great. I'm really happy to to hear that and see the practice. And for those of you that said maybe. I would probably you would know if you had one most of the time, but maybe someone else on your team knows and that's that's good to have in mind. And what I want to do before we get to the last kind of 15 minutes of the very brief case discussion is ask anyone again, open the floor to anyone who'd want to share. In particular, I'm interested this time for any of you that have a data inventory. Are you willing to share? Again, this is however many of you, almost more than a third of you. So I know that there's at least, you know, 40 or 50 people who said that they have a data inventory. Are you willing to share how you use it, if you use it at all? I'm very curious to kind of hear thoughts from the group on that. So again, I'll mute myself to force myself to pause. If you would like to share, please do raise your hand.

[45:30](https://youtu.be/C5QyBfCZPp8?t=2730) We have a couple of folks sharing in the chat as well. Awesome. Thank you. Thank you, Graham. When I'm sharing my screen, I can't see the chat. So I appreciate you pointing that out. So Martin, in particular, thank you for sharing that. Our data inventory is used to analyze roads and if they need repairs. That's a really great, interesting comment. And I'm going to save that one for a second, because one of my case studies, we're going to talk about some roads here in a second. And what type of information we might include when we're thinking about roads and repairs and whether they need repairs. How do we prioritize them, all those kind of things. So again, thank you all for sharing, for those of you that shared in the poll and in the chat. This is going to be really interesting information, I think, also for follow up for the team. So thank you for sharing.

[46:34](https://youtu.be/C5QyBfCZPp8?t=2794) So we have about 10 minutes left. And what I'm going to do is kind of go through a couple of examples for you all really quickly, and then open it up for Q&A for the rest of the 30 minutes because we have another 30 minutes blocked after that. I'll try to end on time in the next 10 minutes. So, let's go back to the share for the homestretch here. The last thing to think about is a couple of cases. And so, I'm going to go through two examples that obviously, you know, they're going to be a little bit contrived. I made these up right so like there, but I think that they're real questions and real things that folks might be going through. But I'm going to walk through the examples together in an attempt to answer the following questions for each case and encourage you to do the same on your own as we go through. So I'm going to go through what type of problem do I think it is? What data assets could we use? Where can I find them? Who or where would you find those assets? What data quality, privacy or bias issues might exist? And who might we interact with using this data from the communication side?

[47:42](https://youtu.be/C5QyBfCZPp8?t=2862) So, the first example, and thank you Martin for teeing this up beautifully. So state highways need a lot of maintenance to keep road quality high, time and resources limit the amount of work that can be done. So what type of problem is it? I encourage you all to think about this on your own. This one, I would view as probably actually what I need to do here is go back to share like this. This is what I'm going to do. Sorry if it's not big enough for all of you, but it's going to be easier for me because I did some pre thinking about this. So I see this as prioritizing work or optimizing resources. Generally, what data assets might we use? So I would say we might reuse a road quality study, maybe staffing patterns to know like where folks are to help us do that maintenance. Seasonal data, I would imagine we don't do a whole lot of road maintenance work in the winter, if this is a winter state. Maybe how much materials are costing though right now materials are really high. With traffic patterns like these are just types of data assets that we might need to think about when we're planning road maintenance. Now I look forward to Martin's comments in the chat or to correct or add into some of those things.

[48:52](https://youtu.be/C5QyBfCZPp8?t=2932) So where would we find some of those assets potentially? We might have administrative data that the Department of Transportation keeps. NOAA has a lot of weather data from a federal level. Finance might have data. Suppliers might have some of the data about costs. So those are some things that we might need to have. For data quality, privacy or bias, there's not as much on this one that I can think of, but others might have examples. I was thinking that some of our data might be biased because climate change patterns might mean that our road, our seasonal data might not be as reliable. Might mean we need different planning for different types of resources. That's all I could really think of. And who might we interact with? We might interact with the budget department if we need to ask for more budget potentially. Might be interacting with local governments, the federal government doing highway planning, all those kind of things. And this again is not exhaustive. But it's just an example to help you all think about using these categorizations and helping them kind of plan your problem definition, maybe refining that and doing some analysis.

[49:54](https://youtu.be/C5QyBfCZPp8?t=2994) The second case is one that is a little bit different. And so excessive force violations by police officers have huge negative repercussions in our community, police departments and for public safety. So this is one that I had come across last year. People had asked me to go through this one. So I'm going to go through it with you all today. And in particular, for those of you that might be doing more work where you're actually doing work on service delivery with people. This might be a little bit more relevant. And so what type of problem is it? I saw it potentially as, you know, needle in a haystack or early warning problem. Maybe we're trying to detect early excessive force violations so that they don't get worse. Maybe there are very few and actually we're trying to detect where they're happening. So those are the two types that I could see, although there might be many other types, depending on what the priorities are. What data assets could we use? We might have use of force data. We might not. We might need to get that from someone else.

[50:54](https://youtu.be/C5QyBfCZPp8?t=3054) Community demographics might play a big factor in this. And I know this has been a huge issue across many places, in particular where places are folks are dealing with very active Black Lives Matter movements or very active social movements that are related to community demographics and policing. We might also have crime data that we might want to throw in there that might help us think about what's going on with excessive force. Where might we find those? We might find them from police departments or the state police agency, if there is one. Census, Department of Justice might also have some extra or additional resources. You all might be able to think of others. What data quality, privacy or bias issues might exist? In this case, there are many more. We have very personal information. We have very sensitive subjects. We might be dealing with victims of crimes. And that means we might need to treat our data very differently than, for example, the costs of road maintenance supplies. They're not quite as sensitive and there's a lot less risk in terms of that data.

[51:56](https://youtu.be/C5QyBfCZPp8?t=3116) And so this is something that would be really important for this kind of case. And who might we interact with? And this one, I honestly can't think of someone we might not interact with. Police, community, policymakers, politicians, health might be an example. The list goes on and on for this one. And you all, again, might be able to think of others. But those are just a few of the folks that I thought of initially when going through this case. And so I hope that those examples give you like a quick sense of how we might go through and use some of these problem categorizations, maybe our data inventory and this kind of quality, privacy and bias issues just to get us brainstorming and thinking about how they interact with our problems in the real world. And so to kind of wrap things up today, what I want to do is conclude with two main thoughts. So this is the other pre-reading that I gave you. This was an interview with a man called Ken Stief who unfortunately passed away last year.

[53:01](https://youtu.be/C5QyBfCZPp8?t=3181) But he was a professor at the University of Pennsylvania focusing on analytics from public policy and in particular for planning. But there's kind of a couple of main thoughts that he had in that article that I really liked. So first is that, you know, governments are collecting the same types of administrative data, often working on the same types of use cases. And that means we should be able to share and potentially replicate solutions, taking into account local factors, but that we should be able to share more often what we're doing in terms of data and analytics. But he also says, you know, if we're doing resource allocation, for example, if that has been historically driven by inequality, why should we assume that a fancy new algorithm will be any different, which is exactly right on the head. And I really like the sense of this interview because he is both encouraging us to share and use data. And he's a big proponent of that himself, obviously, but also to take into account other factors and to be human about what we're doing.

[54:01](https://youtu.be/C5QyBfCZPp8?t=3241) These are complex, complex problems, complicated issues, and we need to approach them with humility and use as many of the tools that we have, including data, to help address them going forward. So I encourage you all to go back and read that interview. And also you can find in that link, he has a whole online free book that's public policy analytics that you can find that's all available for free. So I encourage you all to take a look at that. And to kind of recap, you know, these are the three things that I hope that we talk about in this hour. I think we covered, you know, patterns like resource optimization, finding a needle in a haystack, talked about data sources like data inventories or open data, administrative data. And we identified those issues like bias, audience, data quality. So we covered the things that I wanted to cover. And I hope for all of you, there is at least one or two things you could take away that will be very practical. And I want to leave you all with this question that I always ask at the end of all of my sessions, which is, you know, what can you do to put something you learn into practice tomorrow, next week, next month?

[55:07](https://youtu.be/C5QyBfCZPp8?t=3307) So maybe tomorrow you can say, like, I want to go back and redefine a problem that I've been working on to be a little bit more clear. Maybe by next week I can say I want to check some data sets for quality and go through and see, you know, whether the quality is really good. Maybe by next month we can have started having conversations about a data inventory if we don't have one. All those kind of things that I hope that you all can think about what you can start doing and really keep momentum going, because a lot of these things, you know, seem like they're not, they could take a long time, but really just got to get the ball rolling and get started and hopefully have some tips that can help you do that very soon. So what I'll do very, very briefly, and you guys have all the slides already. You have my email and LinkedIn if you want to get in touch with me, again, in the deck, and Graham and Cassidy can share those. And I want to leave the rest of the time today to do Q&A. Graham and Cassidy, maybe there are questions in the chat or on the Slido that we can go through, but I also want to open the floor in a second just to kind of see how it's going after Graham gives us his words of wrap up, of course, as well.

* * *

Original page: https://innovate-us.org/defining-problems-with-data-june-30 · Licence: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org
