---
id: "workshop:data-series-the-science-of-data-questions-and-exploring-data-sources"
type: "InnovateUS workshop"
title: "Data Series: The Science of Data Questions and Exploring Data Sources"
url: "https://innovate-us.org/data-series-the-science-of-data-questions-and-exploring-data-sources"
date: "2023-11-01"
duration: "60 minutes"
language: "English"
instructors: ["Stefaan Verhulst"]
recording: "https://www.youtube.com/watch?v=HodbbLyV4Rg"
transcript: "automatic, from the recording"
source: "InnovateUS (innovate-us.org)"
licence: "CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org"
---

# Data Series: The Science of Data Questions and Exploring Data Sources

InnovateUS workshop · 1 November 2023 · 60 minutes

This 60-minute virtual training will focus on the science of data questions, understanding minimum viable data, and exploring data sources. Participants will learn the importance of asking the right questions in a data-driven world and how to design questions in a participatory way. The workshop will explain what minimum viable data is, why it is crucial, and how to determine it, with the aim of balancing data comprehensiveness with utility and efficiency. Participants will walk away with an understanding of how to identify potential data sources, evaluate data credibility and relevance, and gain tools and techniques for data sourcing and verification.

## Learning goals

- Learn the importance of asking the right questions in a data-driven world
- Learn how to design questions in a participatory way
- Gain an understanding of what minimum viable data is, why it is crucial, and how to determine it
- Learn how to identify potential data sources, evaluate data credibility and relevance, and gain tools and techniques for data sourcing and verification.

## Speakers

- **Stefaan Verhulst**, Co-Founder of The GovLab, Co-Founder and Principal Scientific Advisor of The DataTank

Recording: https://www.youtube.com/watch?v=HodbbLyV4Rg

## Transcript

Automatic transcript of the recording: names and terms can be misheard, and the times can be a few seconds off. Each time links to that moment of the video.

**Stefaan Verhulst**

[0:00](https://youtu.be/HodbbLyV4Rg?t=0) As those that joined last week, you already have got a brief introduction about what the purpose of this three series is, but last week I was hoping to basically set the stage about how to think, what's the kind of mindset that one can embrace when thinking about using data for solving public problems or for addressing critical public needs or designing public services as well. And today I will go and build a little bit about what we discussed last week by focusing on what I feel is a crucial element of any kind of data initiative, which is really about how do we actually go about formulating questions and how do we then go about understanding what data is needed to answer the questions. And specifically, what I will try to argue for is that we really need to start adopting an approach of a minimum viable data kind of approach in order to rapidly make progress and also to do so in a responsible way. And so let me share my screen. I hope this will work. Hannah, can you see my screen? Yeah, there we go. So anyway, we've done that. And Hannah already introduced myself and those who are familiar to the Innovate US series probably are familiar to the GovLab. So we are an action research center focused on how do we change the way we make decisions in society, leveraging new technologies. And obviously one asset that technology has generated is data. So for doing so, we really need to start opening up data because it's great to advance data-driven

[2:05](https://youtu.be/HodbbLyV4Rg?t=125) decision making. But obviously, if you have no data, or if the data resides in other places that you cannot access, then obviously that might not be a good way forward. And so at the Open Data Policy Lab, and at the GovLab, we specifically focus on how data can be used to innovate how we solve public problems. And obviously, this is a bigger question that requires some kind of deconstruction on, well, what do we mean by data? And last week, I went through a set of observations that we quite often use as a starting point, starting with the fact that we, of course, have massively more data than we've ever had. We also have massively new types of data. So when we talk about data, it's not anymore just about surveys, or administrative data. It's also about data that the private sector collects that could be used if unlocked for decision making. Or it is also data that can be crowdsourced, for instance, in ways that provides for more lived experience than typically quantitative data might provide. But also, we emphasize that it's not a thing.

[3:20](https://youtu.be/HodbbLyV4Rg?t=200) It's not just an asset. It's actually a process where multiple decisions are made, starting from the planning. And today, we will focus mainly on the planning, actually, on anyway, what is it that you want to achieve with the data? And more importantly, what's the question you seek to answer? And how do you go about formulating that question? But obviously, once you have a question, you need to think about what data we need. That quite often leads to either collecting data, or trying to access data. And subsequently, of course, we need to make sense of the data through analysis, and then subsequently using. I'm not going to go through the whole aspect again of what we discussed last week. But suffice to say is that in order to make sense of the data, we really need to understand what the question is, or what is the kind of problem that we seek to solve. And unfortunately, while this might sound pretty obvious to most of you, what we see in the data space is quite often this phenomenon of looking under the lamppost when you've lost your keys. Everyone knows that it's a pretty well established kind of phenomenon, where if you lose something, then you start looking where the light shines, instead of looking where the key might potentially be. And a similar kind of phenomenon happens quite often with the data, or in the data space, by which a lot of the data initiatives starts from the data, not from the question, and then tries to generate insights from the data without understanding whether actually the answer that the data provides is actually the answer to the question that matters, and to the problem that needs to be solved. And so what I hope to do is to really change a little bit the mindset that if you initiate a data project, don't start with the data that you have access to, or with the data

[5:23](https://youtu.be/HodbbLyV4Rg?t=323) that you already know could be made accessible, but start with the question, and start more importantly with the Oh, I can, is everyone else seen?

**Unknown speaker**

[5:45](https://youtu.be/HodbbLyV4Rg?t=345) Stefan, sorry, you've cut out twice. Sorry, yeah, as I said, I'm in Mongolia, hopefully the

**Stefaan Verhulst**

[5:55](https://youtu.be/HodbbLyV4Rg?t=355) internet will keep going. But where did you, where did you lose me, Hannah?

**Unknown speaker**

[6:02](https://youtu.be/HodbbLyV4Rg?t=362) We just lost the last like five seconds, ten seconds.

**Stefaan Verhulst**

[6:05](https://youtu.be/HodbbLyV4Rg?t=365) All right, okay. So ultimately, if we want to use data, I think it's super important to start with the problem, or with the decision that you seek to inform, not with the data that you might have access to. And so what I will suggest today is to develop a flow of development that starts with the problem, subsequently leads to a mapping of actually the topic and the interdependencies of the problem, so that you'd know that your problem depends quite often on other topics as well. And then from a data perspective, it might be important to also address the other topics first, before you start answering questions that might be directly related to your problem. But obviously, you need to really turn that into questions, because data, the only thing data can do is actually answering questions, right, is providing insight vis-a-vis particular kind of questions. And that requires far more sophistication than typically is assumed. And then once you have a question, then it's really about what is the data that I need, and also what is the expertise and the capacity that I need. And what we will look into is that this data, or the expertise, can be residing both within the inside the corporation, or inside the organization, inside the public sector, or it can be external. And I think it's really important to do this kind of mapping, in order to then understand what data is needed to answer what question that then can support solving what problem or can support the decision statement as well. And so the first premise for today, as to how do we go about leveraging data is to really start with the problem. And it turns out that problem definition, and I know that this course Innovate US has already spent quite a lot of time on actually problem definition. So I'm not going to replicate

[8:08](https://youtu.be/HodbbLyV4Rg?t=488) and repeat what was said. And there are, I'm sure Hannah can point you to a whole range of videos that goes into detail there. But it's suffice to say is that we need to start from the problem, not from the solution in search of a problem. Nor do we quite often, and that's also the problem with data quite often, is that we focus on the symptoms, not the problem. And so that requires some understanding of what is the problem? What are the interdependencies? And how do we go about answering the why question? Why is the problem taking place? Why is it that I am the one that are tasked to solve the problem? And what does that mean in terms of the information that I need to obtain in order to start answering the problem? And so, as I said, I'm not going to go into detail about problem definition. But at GovLab, we've developed a problem definition tool, because we have come to the realization that actually problem definition, while it sounds obvious, is not as obvious and not as well established as one would hope. And is especially important if we start using data in order to start answering questions that can inform the problem and how we go about solving the problem. And so typically, we suggest to define the problem through a few stages. One is really about a kind of articulation. What is it that you seek to solve?

[9:34](https://youtu.be/HodbbLyV4Rg?t=574) What is the problem that informs your query or your quest for more information? Then we typically go into some kind of a clarification cycle, which is really sometimes answering the five times why, which is typically a methodology that we use to actually get better sense about what is the problem and why is the problem manifesting? And are we actually focusing on the root causes as opposed to the symptoms? And then we do a mapping of all the interdependencies and all the topics that are associated with the problem, so that we can understand what is the issue and the topic that we want to prioritize. And then when we have a kind of prioritization of the topic and the issue that we seek to become smarter about, then we can start formulating the question in a way that is answerable by data. So I will mainly focus today not on the problem identification or the clarification cycle. And by the way, we can use data for doing that as well. But really trying to focus on the mapping that we believe is sometimes crucial to really get a better sense of what are the issues that you seek to inform and then also the question cycle. And so the implications of that is that it's really important to think about the environment in which the problem resides and to also solicit input from lived experience, not just big data. And also as the context evolves to also be able to evolve your problem definition and subsequently data quest as well. So that's sufficient to get started to say is that every data initiative should not just be a data initiative for the sake of being a data initiative, but should really try to understand how will my data, my insight contribute to clarifying the problem or to clarifying the potential solution

[11:38](https://youtu.be/HodbbLyV4Rg?t=698) vis-a-vis the problem as well. Now, the question then is, of course, how do you then start formulating the question? And how do you go about actually understanding what are the topics that need? My apologies for these breaks.

**Unknown speaker**

[12:02](https://youtu.be/HodbbLyV4Rg?t=722) Anna, did I break up again? Yes, yes, we lost like about three seconds.

**Stefaan Verhulst**

[12:08](https://youtu.be/HodbbLyV4Rg?t=728) And otherwise, you're very clear. And so apologies to our participants, but you come back pretty quickly. All right. Sorry about that. Yeah. I mean, it's as good as I could do here, unfortunately. And perhaps I could switch off my camera if that helps. But then it might be hard to actually the audience to

**Unknown speaker**

[12:31](https://youtu.be/HodbbLyV4Rg?t=751) just see the slides. And I know, Hannah, what would you suggest?

**Stefaan Verhulst**

[12:34](https://youtu.be/HodbbLyV4Rg?t=754) Yeah, I think we can continue like this. Great. So the question then is, of course, how do you formulate questions? And I, a few years ago, started this initiative, the 100 questions, because I became kind of frustrated and also troubled by the fact that we did a lot of work with people worldwide around data. And the question was always, how do I get access to data? And how do I get more data? And when I then started formulating the question, so, okay, that's great. But what's the question you seek to answer with the data that you are supposedly seeking to open up or have access to? Most of the time, it was a blank stare, or it was a really muddled kind of question formulation as well. Or in other cases, it was a question that was formulated by one individual without understanding whether actually this question was the one that mattered for the problem and mattered for the community that one seeked to solve, serve as well. And so towards that end, I created the 100 questions initiative, which was really an effort to identify the 100 questions that matter, but that we have not answered yet, but we could answer if we find a way to have access to the data. And so towards that end, we developed a more participatory approach to actually formulating questions. And for those that are joining today, this could also be a methodology that you start using in order to really understand what is actually the priority question that I should seek to answer with the data that I try to have access to or with the data that I might already have access to as well. And it's a multi-step process, but more importantly, it's also a participatory process. Because my concern quite often is that questions are being formulated without quite often taking into account, A, the ones that have a particular

[14:38](https://youtu.be/HodbbLyV4Rg?t=878) kind of expertise about the domain that you're working in, or B, quite often also don't take into account the kind of priority questions that the community cares about. And so in the past, I've been advocating that we not only need to advance data equity, we also need to advance question equity. We need to make sure that the ones that can benefit from the insight are also part of the process of formulating the question for which you seek insight as well. Now, it's a multi-step process, and it typically starts, as I mentioned in my first few slides, it starts with a topic mapping, where we actually really try to understand, well, for instance, if urban transportation is the topic, well, what are all the other topics that are associated with urban transportation, so that we actually know what to focus on.

**Unknown speaker**

[15:34](https://youtu.be/HodbbLyV4Rg?t=934) Stefan, we lost you again, maybe we should try without camera.

**Stefaan Verhulst**

[15:49](https://youtu.be/HodbbLyV4Rg?t=949) Yes, yes, let's do that, let's do that. All right, there, I'm still here, and at least you have the slides, and I hope you, this will not happen all the time. So, the second step involves creating a cohort of bilinguals, and by bilinguals, we mean people that have a domain expertise, but also a data expertise, so that we can really tap into the kind of expertise that can formulate the questions that matter. And then we basically ask them to formulate a question vis-a-vis the topics that we have prioritised, that they believe could be transformative, and we do that using a taxonomy, and I will go into the detail in a second what I mean by a taxonomy of data, of data questions. And then subsequently, we cluster those questions, because it turns out that in this community of bilinguals, typically comprises about 100 individuals.

[16:53](https://youtu.be/HodbbLyV4Rg?t=1013) It turns out that, of course, several of them have similar kind of questions, and so by clustering, we then see a pattern emerging of kind of priority questions that there is consensus about, that these are the questions that matter to advance understanding about the problem, or advance understanding how we can solve the problem. And that leads to then a prioritisation, but also to a public voting, where we put the top 10 questions that have been identified by the bilingual up for voting by the public, so that it's also some kind of a public participation in actually formulating the question as well. So we've done this now across a whole range of domains, including migration, gender, but we also have done this at the subnational level, for instance, in New York, we've started on what are the 100 questions for New York, for instance. We also have done it at the national level. We worked with, for instance, Australia to think about what are the 100 questions for Australia that matter, for which you need a data strategy to then start answering.

[18:02](https://youtu.be/HodbbLyV4Rg?t=1082) And it changes the mindset so that you don't start with the data, but you really start with the question. Now, as I mentioned, the first topic that we do or the first step involves a topic mapping. And so by topic mapping, we really, what I call means building a gestalt of a problem. So if we think about a problem that we understand, what are all the issues associated so that if you would see it, you would say, yes, this is actually the domain. Now, once you have a topic mapping, and a topic mapping quite often is used to actually not get lost in the forest, right, and that you only see the trees and not the forest anymore, or vice versa. So you actually spell out all the issues associated vis-a-vis a particular kind of problem domain. And once you have a topic mapping, and I will show you an example of a topic mapping in a second, you can also do an actor mapping. Because then you will also start understanding, well, who are actually the actors vis-a-vis that topic that we need to be aware of or potentially involve for formulating questions.

[19:09](https://youtu.be/HodbbLyV4Rg?t=1149) But you can also do an understanding of what is already known vis-a-vis certain topics. So that, for instance, you can actually identify what I call orphan issues, issues that no insight is yet been developed. Because a topic mapping also allows you to understand what is the current state of knowledge vis-a-vis certain kinds of topics. And it allows you to understand what is it that we actually don't know yet that we need data for, or that we need additional kind of data initiatives as well. So this is, for instance, an example of a topic mapping that we've developed with several actors around urban mobility and transportation. Obviously, each of those topic mapping quite often leads to kind of a 50-page kind of further analysis of all those topics. And by the way, this can be done very quickly, right? Typically, we'd spend like a week and a half on actually doing this in a participatory way. This is not something that you have to spend your life on designing a topic mapping.

[20:13](https://youtu.be/HodbbLyV4Rg?t=1213) And given the fact that quite often you already work in a particular domain, you quite often can get started by just spelling it out for yourself. But here, for instance, when we were looking at urban mobility and transportation, what were some kind of subtopics and categories of topics that came up was, for instance, urban mobility and well-being and safety, right? Well, are we going to focus on actually safety of transportation? Or are we going to focus on gender safety or social justice, who has access to transportation? Or are we going to focus on innovation of transportation? Or are we going to focus on the environmental impact of transportation? Or are we going to focus on the policies and regulations that currently exist and that need to be changed? And so by spelling this out, you see that there are – anyway, when we just talk about urban mobility, well, there are multiple topics associated with urban mobility. And it's important to clarify what is the topic you are actually focusing on, because that will inform what's the question that you need to seek as well.

[21:21](https://youtu.be/HodbbLyV4Rg?t=1281) So these are other topics, for instance, with P-STEC. I'm not going to go into detail here. But I think it's important to just spell this out and do this kind of clustering. I already mentioned bilinguals. And here it's really about – and that also dovetails a lot with the work that GovLab is doing in some kind of smarter crowdsourcing. Because the question here is, of course, who are super questioners in your domain, which is inspired by Phil Tetlock, who some of you might know, who is a professor at the University of Pennsylvania, where he has developed this concept of super forecasters, where he has built cohorts of super forecasters that can basically predict what might happen on a particular kind of topic. And it turns out that these are particular kind of individuals, quite often not the experts that one would think, but the ones that have a particular kind of interest or passion around a particular kind of topic. And they are much better in a position to predict what might happen than, for instance, professional futurists.

[22:25](https://youtu.be/HodbbLyV4Rg?t=1345) And what I've tried to do is to really try to understand, well, can we have a similar kind of category of individuals as it relates to formulating questions? And so that's where we identified this concept of bilinguals, people that have this domain expertise and data expertise to actually identify what those questions might be as well. So again, I'm not going to go into the lessons learned. Suffice to say is that formulating questions is not easy. And it quite often relies and it depends on collective intelligence in order to really formulate the question that matters. Now, what I really want to focus on today, which is also part of the 100 questions, is to really start focusing on when we talk about questions for data, what type of questions are we actually talking about? Because what we've learned from the 100 questions is that if we ask bilinguals or we are formulating questions, there's quite often confusion about what data can answer or what are the different types of questions that can be formulated.

[23:45](https://youtu.be/HodbbLyV4Rg?t=1425) And from our point of view, we started developing kind of a taxonomy of questions. And from our point of view, there are four types of questions that you can use to formulate questions vis-a-vis the topics that have been prioritized as it relates to the problem you seek to solve. As you can see, you have a problem that relates to understanding what is the topic that is associated with that problem. How do we understand the full aspects of that topic and how do we then prioritize the topic and then what is the question? The four questions that we have identified, and again, this is not rocket science, are on the one hand, descriptive questions, meaning questions that seek to answer what is the current situation, questions that quite often relate to baseline of a particular kind of problem, a particular kind of topic. Then we have diagnostic questions, which are questions that really try to explain why is it that we are in the current situation, quite often also questions that try to understand cause and effect.

[24:55](https://youtu.be/HodbbLyV4Rg?t=1495) Then we have predictive questions, and that's pretty straightforward. These are questions about what will happen with regard to the current situation. And then we have prescriptive questions as well. Very quickly, descriptive questions, as I said, these are questions that really try to focus on what has happened, what is the current situation, who is involved, when and where. And so quite often, these are the questions that try to understand a baseline. For instance, if we are trying to understand urban transportation and the baseline could see, well, what's the current situation of, for instance, urban transportation in terms of use of public transportation? That's a typical descriptive question. So it describes the situation. And it is a very important starting point, because if you want to start using data for later on, understand whether the situation is changing or whether anything has changed as a result of an intervention, you need to have a baseline.

[25:56](https://youtu.be/HodbbLyV4Rg?t=1556) You need to understand what is the current situation before you can actually improve it. And more importantly, before you can start measuring whether there was an improvement. And so descriptive questions quite often are the starting point questions to get going. And they typically are quantitative, but they can also be qualitative. And they situate in the context. Now, there are some challenges, because you could have selective framing and naming of the question, because the question will determine what is the data that you will need. And if the question is either too narrow, then you might lose out certain kinds of descriptors. There can also be perception bias, right? So there can be bias in how you actually see the current situation based upon the data that you have. Or there might be lack of qualitative knowledge and insight to actually make sense of the current descriptor or the current situation.

[26:58](https://youtu.be/HodbbLyV4Rg?t=1618) So you always need to, if you have a descriptive question, making sure that you actually also test whether those, whether actually the description resonates with the lived experience of individuals as well. Now, the second type of questions are those diagnostic questions, which is really trying to understand why are we in this situation? And so those are the questions that seek to, for instance, is that why is it that we have so much traffic jam? So it's not about taking stock of this is the current status of traffic jam, but it's really about trying to understand the root causes or at least the correlations as well. This typically requires more than one data set, right? Because if you want to do some kind of analysis between X versus Y, right? X explains Y. Then you actually need data about X and data about Y, for instance, right? And so you need data aggregation as well. Now, the biggest challenge with diagnostic questions, questions that seek to understand the root causes, is, of course, this fundamental challenge between correlation and causation.

[28:08](https://youtu.be/HodbbLyV4Rg?t=1688) It's not because you see a correlation, right? For instance, you see, all right, this variable might explain this impact, that this is actually a causation. It's actually the cause. It might just be a simple correlation. And so correlation determines that there is a relation. But the question is, of course, does that relation mean that one causes the other or does it just exist independently from each other, right? And so correlation and causation is basically a challenge of dependency, right? If I see a phenomenon, is that actually dependent on that phenomenon or is it just an independent existence as well? And so you always have to be careful about this kind of causation versus correlation as it relates to diagnostic questions. Then we have predictive questions, which is really about what will happen next, what will happen in the future. But also predictive questions can, for instance, be based upon inferences.

[29:10](https://youtu.be/HodbbLyV4Rg?t=1750) Now I know, right, based upon my diagnostic questions, now I know that traffic jams emerges from, for instance, everyone starting at 8 o'clock in the morning and they all go and use the same route. Well, then you can actually start predicting, well, if everyone starts at 8 o'clock again tomorrow, then surprise, surprise, there will be another traffic jam. So you have some kind of inferences that you then can start using for predicting the future based upon a diagnostic understanding of the situation. But of course, predictive questions can also be more futuristic questions as well. Now, again, here is a whole, I'm not going to go into detail here, but there's a whole. concern with regard to biased inferences, right? And that is, for instance, a very concern in the area of criminal justice, right, where there is quite often a, an inference of geolocation and, for instance, threat, right, based upon diagnostic, diagnostic data. But the question is, was that actually a biased kind of inference, right? And which then can lead to prediction. That's why predictive policing, for instance, is quite often based upon assumptions, right, inferences, right, that might be biased. And as a result, predictive policing will automatically be biased as well.

[30:43](https://youtu.be/HodbbLyV4Rg?t=1843) And so that's why you have to be very careful about what, what are the inferences, what are the assumptions that informs your predictions, because if they are biased, then that might lead to actually, for instance, discrimination, or ethical concerns as well. And then the last bucket of questions are kind of the prescriptive questions, which are more the impact questions, right? And the other ones that quite often are about, all right, what do I need to do knowing what we have, but also what happens or what has happened now that that I have interfered in the kind of problem as well. And so quite often, prescriptive questions are also impact questions, which are, by the way, quite often, in the public policy community, quite often, the questions that are being used the most is to actually evaluate or monitor what is happening following a kind of intervention. Now, here is the, here's the challenge, is that there is a certain hierarchy of questions, right?

[31:46](https://youtu.be/HodbbLyV4Rg?t=1906) Meaning that in order to actually answer, for instance, the impact questions, you quite often have to have already have answered a situation analysis, right? So because as already mentioned, it's very hard to answer, have we made an impact, if you have no baseline of what is the current situation, right? It's very hard to predict, if you have no understanding on what explains, for instance, evolution and action, right? And so that's why there's a certain kind of hierarchy. And it's important for you, who anyway, wants to start using data, to really start thinking about, well, do we actually know the current situation? Or is that where we should start our data initiative, right? Should we start with actually first having a baseline before we start predicting or before we start measuring impact? Or should we first try to understand why is it that we are in the current situation, so that we can predict, right, if the cause if we know what the causes, and we know how the cause is evolving, we actually can predict what the potential impact might be, as it relates to the problem as well.

[32:58](https://youtu.be/HodbbLyV4Rg?t=1978) If we know, for instance, that there is a lot of climate change happening in, in the developing world, we might actually anyway, and we know that there is a correlation between climate crisis and migration, then we can predict that there will be a lot more migration as well, as a result of actually understanding this cause and effect as well. And so this is important to a understand that there are different types of questions. And also, more importantly, that there is some kind of a hierarchy of questions that you may have to start thinking about where in this hierarchy, can we actually move with regard to our data analysis as well? Again, the question is here quite often, alright, so what are the how do we cluster questions? I'm not going to go into detail here. But it is important for you to also try to come up with criteria to understand, well, what are the priority questions for us, right? What are the criteria to determine? What are we going to focus on first? What is the question that we need to answer first, before we start answering all the questions, because typically, there are multiple questions that you seek to answer. And you need to start thinking about how do we go about this. And this can be anyway, one criteria can be the impact that you seek to, if we would have answered to that question, the impact would be tremendous, because we would actually know how to allocate resources. But in some cases, it might be novelty, it might be well, we're going to prioritise that question, because we actually don't know, we have some sense of understanding of the other questions, but we have not yet answered that question at all. Or it could be feasibility. And in some cases, we are developing kind of a matrix where you have feasibility, novelty, impact, and then, of course, it's kind of high impact, high feasible, that's kind of the priority question that you might want to prioritise as well. Alright, once you have the question, and that quite often, anyway, we call that formulating the demand for data, or formulating the

[35:02](https://youtu.be/HodbbLyV4Rg?t=2102) problem for which you seek to leverage data, then the question is, of course, how do we then understand what data is needed? And here, I've been advocating for really focusing on the minimum viable data to make progress, because the mistake that quite often is being is quite often happens is that give me all the data I can get, right? And then I will seek to what extent I can actually use that data to answer my question. That quite often leads to a problem from a governance perspective, because it doesn't dovetail with this notion of proportionality, which is a fundamental data governance principle, right? You only use the data, the minimum data that is needed to answer the for the purpose or to answer your question. But it also leads quite often to noise, because you then get all kinds of data sets that might not be relevant. And so here, the question is, what's the minimum viable data point in order to become more targeted? And towards that end, you need to start thinking about a what's the type of question I seek to answer, right? If it's a descriptive question, then quite often one data source as it relates to the problem, like for instance, amount of traffic jam at that point in your city, well, that might be sufficient. But if it's around anyway, what's the root causes, then you have to start thinking about well, do we actually need a how much traffic jam, but also other aspects that might explain traffic jam. So I can show that there is a certain causation, or at minimum, a correlation between the phenomena, right? So the type of question will determine the type of data you need. Same for impact assessment, you might need data about what did you do, versus what was the impact vis-a-vis the baseline as well. The second type is really when we go, and these are mainly if it's about

[37:05](https://youtu.be/HodbbLyV4Rg?t=2225) questions, for instance, about socio demographic phenomenon, then it's really about the level of disaggregation. Do I need data about women and men? Or do I just need data about all the people? Do I need data about age groups? Do I need data about geographical location, and so on, right? And so that's the other element that will determine your data needs as a kind of the level of disaggregation, and also the sample size that is needed in order to be representative. But then other elements, like for instance, do I need data about the time when something happened? Right? Do I need a timescale? Do I need a timestamp? Do I need geolocation? Do I need data about where that phenomenon happens? What's the level of abstraction that I need? And to what extent do I need data that is interoperable as well? And so here, we typically suggest that in order to do so, you do this kind of data audit, right? Start with the question, and then what is the kind of data that I need? But you can also do some kind of a data ecosystem mapping in order to understand then, all right, if this is the kind of data that I need, what's the minimum viable data that is available? And this is an example of a mapping tool that we have built for UNICEF, in order to have them become smarter about data for and about children. But it's also important, for instance, to use some kind of taxonomy of data. Do we need personal data? Do we need data that is less personal that can be observed without personal data? Because that will determine your basic data governance regime. And then also trying to understand the types of data they need. Is it administrative data? Is it open government data, where this kind of data might reside? Do I need

[39:08](https://youtu.be/HodbbLyV4Rg?t=2348) to think about scientific data, meaning where I can go and visit a repository of open science data? Is it data that I can get from civil society? Crowdsourced data? Is it private sector data that I need, that can actually answer my question might not be publicly available. But at least if you have a question that is formulated, and you have a clear rationale behind the question, i.e. that question will solve that problem. It's a much easier ask to a data holder to actually share some of the data as well. Or is it indeed business to government as well? Again, I don't have the time today to go through all examples of possible data sources that can start answering your questions. But suffice to say is that there's a lot of open government data already available. There are a lot of open science research portals, which quite often gets ignored, however, by the public sector. But I would advocate is that there's a lot of data that scientists and researchers collect, that could be available to the public sector in a far more easier way, if you have actually a better understanding of the question you seek to solve, because you can also see anyway, what researcher has already collected data in my region, that I actually could start reusing, because it has been made open in a portal. But there are also other portals that provide access to civil society data, or data that NGOs have collected. You have crowdsourced data as well, that quite often has been made available, private sector data that you have public interfaces for. And then you also, for instance, have data intermediaries, repositories of repositories, where you can actually start looking for data as well. The other thing you could do, and this is something that we quite often do at GovLab as well, is to actually map in

[41:10](https://youtu.be/HodbbLyV4Rg?t=2470) your city, for instance, right? Who actually collects data? And and for what purpose? And, and that goes beyond public sector actors, right? Could also, by the way, could also be other agencies, right? Because it's not just other public, other private, other non-public sector actors that collect data. But in the private sector, you could start thinking about, well, who collects data, for instance, commercial and financial transactions, right? And they can say, well, actually, all the grocery stores are collecting data. And they, anyway, they have e-receipts. That's a data point that I could use, if I'm interested in, for instance, consumer pricing of my city, or I'm interested in, for instance, understanding what are the food patents that might explain child obesity, for instance, and that are kind of concrete examples on how retail data has been used to actually clarify certain kinds of phenomenon as well. And so think about the type of data, what business actors are involved? What is it structured, unstructured? Is it look, does it, is there a location? Is there a timestamp? And anyway, how can it be used? And what are the possible use cases as well?

[42:29](https://youtu.be/HodbbLyV4Rg?t=2549) And this kind of mapping of the data ecosystem allows you, especially when you categorise it like that, allows you to then understand, well, this is the kind of data that are actually the minimum viable data that need to have access to it as well. There are other classifications as well. For instance, the commercial sector has data like Yelp has a lot of data that could be made accessible, and that quite often is accessible for the public sector to understand decision support, for instance. But you have actors like Nielsen, that also, anyway, that aggregates data, and that could be used for answering some questions as well. So again, I don't have enough time here to really deconstruct this. But it suffices to say is that once you have a question, then you can start thinking about who actually has data that could help me answer that question, if you know, what is the type of data that you need, and the type of quality, the type of veracity, right? And velocity is a data that I need, that is real time data, or the data that I need that was collected last year. And so all of this can, you can then start thinking about, well, who actually, in my state, in my city, in my country, or in my sector, actually collect data, that if I share my question, I actually might make progress as well. And then it's not about just the data, you also, if you also have to start thinking about, if this is the question, right, what's the kind of expertise that I need in order to answer the question, right? And, and this could be both lived experience, right? So it's not just about, actually, I need to have someone who lives in that community. So I can understand what actually this question makes sense, or more importantly, how to answer that question in a way that resonates with their lived

[44:29](https://youtu.be/HodbbLyV4Rg?t=2669) experience. But it could also be, of course, technical expertise, quite often, also expertise that will be necessary to allow you to actually have access to data. But from, and also, to what extent, internally, you have that expertise presence as well. But for instance, from a data analytical perspective, at GovLab, we did kind of a mapping of internal expertise in NHS, the UK Health Department, Because what they wanted to do was to say, well, we need external expertise. We need to either hire new data scientists or we need to hire consultants. And so what we tried to do with them was to actually say, well, before you do that, try to understand what is the expertise that already resides within your organization. And this mapping of internal expertise that you then can leverage based upon the kind of question you seek to answer is super important to know, because otherwise you're always going to assume that there is no expertise. And basically we categorize the expertise according to those kinds of capacities, structured analysis, which is kind of typical statistical analysis, or unstructured data analysis. For instance, if your question is more about sentiment analysis, right, then quite often the data is more unstructured. It's really about trying to understand perceptions or opinions that are not structured. Well, that requires a particular kind of expertise as well. And then anyway, there are other taxonomies that can be used on kinds of skills that are required as well. And I'm happy to share, well, I will share this slide, but that you can perhaps use that to also try to understand, well, what is the expertise that, anyway, the kind of data skills that are already in my organization that I can tap upon as well. And you can also start building networks. Vivo, for instance, is kind of a network

[46:31](https://youtu.be/HodbbLyV4Rg?t=2791) that tries to identify expertise, but then also then connects it with the demand side as well. So this is in a nutshell my suggestion to start developing what I call decision science, which is really about becoming smarter about how do you formulate the decision, the questions that can inform a problem, a decision, but can also inform what is the data you actually need before you start with the data. Because as I mentioned, too often people start with the data, start with, anyway, what can, what are the answers I can get from the data, without really understanding whether actually it is the question that matters vis-a-vis the problem as well. So start with the problem, not the data. Think about also how do you engage others in actually formulating the questions. Think about your taxonomy of questions, knowing that there are different types of questions, and thinking about, well, what in the hierarchy do I fit? And then once you have a question, start thinking about what is the type of data I need, and how can I get it, who has it, and then more importantly, who can actually make sense of it, or who do I need to engage in order to make insight as well. So that's for today. I'm going to stop sharing here, because what I hope to do next week is that once you have a question, and once you have an understanding of the data you need, then quite often the biggest challenge is to actually have access to that data. And so next week I will then focus on, now that you have a question, now that you know what data you need, how do you establish a data collaborative to have access to the data that then can start informing the question that you need, and how, what are the different models, how do you govern it,

[48:31](https://youtu.be/HodbbLyV4Rg?t=2911) but also what are kind of the technical requirements for doing so. And again, in 40 minutes I will hope to basically provide you the kind of the basic elements to start thinking about it, and then hopefully share some resources that can go into more detail as well. So let me stop sharing here, and hope that I was actually not going in and out all the time. But hope in kind of the 50 minutes here that I didn't overwhelm people, but that at least people start understanding here that questions matter, and that we have to become more sophisticated actually about how we formulate questions, because quite often we get answers and then we actually realise that this is not the answer that we were looking for, because we didn't actually formulate

**Unknown speaker**

[49:28](https://youtu.be/HodbbLyV4Rg?t=2968) the question in the first place that mattered. Thank you so much Stefan, and actually once you turned your camera off it was totally clear, so hopefully this will work and you can keep it on for the Q&A. We've got a number of great questions that people have put in the chat. I'm just going to quickly put a link to our feedback survey that's just one minute, and we'd love if you could fill it out after this session, it's really just a few short questions, but we'd like to know, to hear from you how this all went. All right, so I am going to just ask one question with a recording, because the person had to leave, but I will be able to send

**Audience**

[50:07](https://youtu.be/HodbbLyV4Rg?t=3007) her the recording so she can listen to your answer. Yalil asked, at what point do you recommend bringing in legal counsel and or regulatory staff when formulating questions? Do you want to bring them in once the conceptual conversations with the technical and data teams have taken place,

**Stefaan Verhulst**

[50:23](https://youtu.be/HodbbLyV4Rg?t=3023) or earlier in the process? Yeah, well it's a good question, and again as you saw in the expertise needed to move from a question to accessing the data, legal was a critical expertise that I would suggest you make sure you take into account, but I think it might be too early if you engage with legal already before you actually know what is it that you need, right, and that's something that we've seen quite often, is that people are spending a lot of time on what is the regulatory legal conditions for having access to data. For instance, they are spending a lot of time on developing data sharing agreements with data holders that might have data that is relevant without actually understanding is this actually the data we really need, and so I've seen in many places where they have data sharing agreements, memorandums of understanding that the legal office has developed, and then they come to the realization that they actually don't need the data because the question is different and the data that they have actually doesn't answer that question, and so get really clear on the purpose, really clear on the question, really clear on what is the minimum viable data, and then legal can help you with really understanding what are the implications. For instance, if the question involves personally identifiable information, then they will also be able to give you an insight whether you need additional kind of ethical review before you start accessing data or collecting data, or whether there are other legal implications as well. So I would advocate for formulating your purpose and question, and then legal will be in a much better position

**Unknown speaker**

[52:22](https://youtu.be/HodbbLyV4Rg?t=3142) to help you as well. Thank you, Stefan. I'm going to just stop the recording now.

* * *

Original page: https://innovate-us.org/data-series-the-science-of-data-questions-and-exploring-data-sources · Licence: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org