---
id: "workshop:the-identity-challenge-tackling-user-disambiguation-and-data-integration-across-programs"
type: "InnovateUS workshop"
title: "The Identity Challenge: Tackling User Disambiguation and Data Integration Across Programs"
url: "https://innovate-us.org/the-identity-challenge-tackling-user-disambiguation-and-data-integration-across-programs"
date: "2026-04-13"
series: "AI and Human Services: Innovating to Serve People Better"
instructors: ["Ariel Kennan","Chris Correa"]
moderators: ["Rob Asaro Angelo"]
recording: "https://www.youtube.com/watch?v=jeVnawryyTM"
transcript: "automatic, from the recording"
source: "InnovateUS (innovate-us.org)"
licence: "CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org"
---

# The Identity Challenge: Tackling User Disambiguation and Data Integration Across Programs

InnovateUS workshop · 13 April 2026

Accurately identifying and serving individuals across multiple programs remains one of the most persistent challenges in human services. This session explores how agencies are using, and struggling with, AI, data matching, and entity resolution to connect information while protecting privacy and dignity. Participants will examine the technical, ethical, and operational dimensions of user disambiguation, including data quality, error rates, governance, and cross-program coordination. The session highlights tradeoffs agencies face when integrating data and the consequences those choices have for clients and frontline staff.

## Learning goals

Learning Goals: By the end of this workshop, participants will be able to:

- Understand the core challenges of identity resolution in human services systems
- Identify risks associated with data integration and automated matching
- Evaluate how AI and analytics are used to support—or complicate—cross-program coordination
- Recognize governance and design choices that protect clients from exclusion or harm

## Speakers

- **Ariel Kennan**, President, Axiom Foundation
- **Chris Correa**, Director of Solutions Engineering, Criminal Justice, Code for America

## Moderators

- **Rob Asaro Angelo**, Senior Fellow, Burnes Center for Social Change & Professor, Northeastern University

Part of the series [AI and Human Services: Innovating to Serve People Better](https://innovate-us.org/workshop-series/ai-and-human-services).

Recording: https://www.youtube.com/watch?v=jeVnawryyTM

## Transcript

Automatic transcript of the recording: names and terms can be misheard, and the times can be a few seconds off. Each time links to that moment of the video. The transcript alone: https://innovate-us.org/the-identity-challenge-tackling-user-disambiguation-and-data-integration-across-programs.txt (plain text) and https://innovate-us.org/the-identity-challenge-tackling-user-disambiguation-and-data-integration-across-programs.json (every segment with its start and end second).

**Ariel Kennan**

[0:00](https://youtu.be/jeVnawryyTM?t=0) Well, thank you so much to the Innovate US and Center for Civic Futures team for inviting me to share with you all. I'm going to talk about digital identity and up front how we can help support access, security, and privacy in benefit systems, and ideally how that will help you on the back end for some of the data integration and disambiguation of the users. So I'm from the Beck Center for Social Impact and Innovation. We're a nonprofit, nonpartisan center at Georgetown. We're focused on improving digital service delivery. We run a national network called the Digital Government Network, which has over 8,000 public practitioners. I encourage you to join us. So, you know, public benefits agencies are tasked with keeping people's information and data safe, while also protecting government systems and resources. And really, this is no small feat. And you really need support to help achieve that goal. There are many angles we can consider to help us do this. We can think about things like human-centered application design, cybersecurity standards, and privacy-enhancing technologies and practices. But one angle is how to consider how digital identity approaches can help solve for access, security, and privacy, and how digital identity can actually carry forward into further data matching and integration.

[1:30](https://youtu.be/jeVnawryyTM?t=90) Ideally, if we have better data on the front end, the easier it will be to distinguish individuals on the back end. Digital identity has been a core research theme for us here at the Beck Center for the past several years. And I'd like to thank my colleagues at the Beck Center, the Center for Democracy and Technology, the National Institute of Standards and Technology, and Public Policy Lab, who have been partners in this research, and much of the content I'll be sharing with you today. So, we're going to cover digital identity basics, making digital identity easier for applicants and beneficiaries, as well as tailoring through risk management approaches. So, let's dive into some of the basics. At the highest level, when we're talking about digital identity, we're talking about how we can tell someone else or a service provider, like a government agency or a bank, information about who we are online. When you want to access that service online, you may need to create a specific digital identity. For example, you might make a username and account to place a shopping order. That username and account represents you in that interaction. But we all likely probably have different representations of ourselves across the internet and different digital environments. So, there's three core concepts we're going to cover today. Proofing, authentication, and federation. The first is proofing. Importantly, creating an account or username doesn't mean a service provider knows who you are in the real world. So, identity proofing is all about answering the question, who are you? And provides means to increase confidence that a digital identity belongs to a real person and is being used by that intended user. You may have encountered this yourself and prompts yourself things like sharing information like your name, address, your social security, questions about your credit history, or prompts to upload an image of your face and photo ID. Identity proofing can be thought of as

[3:32](https://youtu.be/jeVnawryyTM?t=212) a three-part process. This graphic outlines the steps technically involved in identity proofing from collecting identity information about someone to validating their identity evidence, and then also linking the evidence back to the person providing it through verification. Proofing shows up in a few different ways in public benefits applications. We're seeing in a lot of unemployment insurance applications biometrics. This often looks like the live selfie and photo of an identity document. These present both usability and equity concerns as biometrics are not effective equally across all demographic groups. There's also some new security risks as a result of deep fakes and degenerative AI. Knowledge-based verification is very common. These are questions about your credit history and other personal details. These also present security and equity issues. A lot of the questions that are asked in KBV have actually been exposed in prior data breaches. Although it's commonly seen in some state instances, NIST no longer recommends it due to security concerns. You also may see other options like video conferences with trained identity verifiers or in-person identity verification.

[4:50](https://youtu.be/jeVnawryyTM?t=290) And then authentication. If identity proofing tried to answer who you are, authentication is about finding out if you're the same you. So whether or not you were asked to go through proofing to access a system, you might still have to create an account to access a service and then come back to it later. So in more technical terms, authentication determines the validity of one or more authenticators used to claim a digital identity. And you'll see on the slide here an illustration of some of the different kinds of authenticators that may be used alone or in combination. When they're used in combination, we're talking about multi-factor authentication, which is likely a term you've heard and also something that we regularly face in a lot of other service interactions in our own accounts. As a note, always using accounts and authenticating users has real pros and cons. Accounts provide obviously a single input. People can get back to their information and access their applications that are in progress and see the status of them. For staff on providing customer service, it has information in a single place.

[6:02](https://youtu.be/jeVnawryyTM?t=362) And in the case of our conversation today, data integration and identifying individuals ideally can be easier. We are seeing issues with access and barriers to access. This is a screenshot from the peak Colorado application, which we think is a great example of showing the user's real benefits of why you should create an account. But most importantly, it also gives them an option so when you're applying initially, you can actually get your application started without an account so that this isn't a barrier to access to someone who's just trying to initiate that relationship. We've been seeing more states take this approach and we really encourage it. And lastly, on our key concepts is federation. So, federation is how do I share information about you? It provides a convenient way to leverage digital identities that users have already established in order to access additional services. This, in many cases, looks like benefits applications integrating state single sign-on services. And as of last year, state single sign-ons are integrated into benefits portals in 19 states. But a few notes on single sign-on. There's really some trade-offs. So, one is that single sign-ons make it that you have one account to access many services, which can be very valuable from a usability gain, especially for people who are eligible for multiple benefits programs. It can also apply a higher level of security across different services, but with less burden on each of those services. However, single sign-on can really create a barrier. So, having that optionality where needed. It can also block users from low level things that they need to do, which might not actually require that level of security.

[7:53](https://youtu.be/jeVnawryyTM?t=473) Also, when it's in a single service or operated by another agency, it might mean still that there's some duplication of questions answered and other things that may look confusing to a user. And also, I really want to say that federation and single sign-on create a lot of ease, but it's really important that we continue to ensure that privacy is maintained, consent is requested, and that single sign-on doesn't actually become a way of surveilling our beneficiaries and users. So, there are a lot of different ways that states are integrating these services. There's really no one single path to how to do this, and we've seen really states looking at different points in the process, exploring the optionality, relying on third-party services as needed, and really looking at different approaches on how they integrate digital identity throughout their benefit systems. We'll talk more about some of the approaches you can use to actually tailor this for your needs, your agency, and your community. But I do want to talk about making digital identity easier for applicants and beneficiaries, which if you're making this easier, ideally, it's getting you better data on the other side. So, we did research in three states last year with our partners at Public Policy Lab, where we talked to beneficiaries about how they were experiencing digital identity technologies when accessing SNAP, Medicaid, child care, and unemployment insurance. The report is a really beautiful multimedia report.

[9:38](https://youtu.be/jeVnawryyTM?t=578) It has a series of insights and recommendations. I'm just going to touch on a few of them today, but I encourage you to check it out and spend more time with it if this is something that your agency is working on. So, a few of the key insights. One is that beneficiaries really feel like they're resigned to sharing personal data to access these essential benefits, despite harboring some really deep concerns about which entities, government and private, will actually access that information. It's very unclear to them where that information goes and how it will be used. Both multi-factor authentication and biometrics come with a lot of challenges, usability issues, and doubt from the users, and also thinking about some of the instability in access to digital devices, email, etc., that may be facing some of your clients. And also that not all friction is equal. I thought that was really interesting that actually sometimes the friction is welcomed and helps and sometimes it harms. So, additional layers of security can actually make people feel that their information is being protected, but it can also actually present a barrier and is seen as disproportionate to the level of risk presented. So, what are a few of the things that you can do to help resolve some of these challenges? Some of our recommendations include to really help build trust by providing simple explanations of how people's data will be used. This can include things like explaining privacy policies and data sharing and retention practices and easy to understand plain language. It can include clearly explaining how the platform integrates and shares data with other systems, as well as making an effort to minimize the information you collect to only the necessary data points that are really required for determining identity or eligibility.

[11:34](https://youtu.be/jeVnawryyTM?t=694) Options are key. So, when you're thinking about multi-factor authentication, you're thinking about proofing, giving people multiple pathways to do that. One size fits all is not something that works. People really need choices so that they can tune to their situation. And lastly, using a risk-based approach to evaluate whether a specific interaction or transaction actually even requires an account or proofing. And so, I'm going to tell you more about some tools that can help you get started on that risk management process. So, let's talk about tailoring and how we can use tools available to us under risk management to make sure that we're actually meeting the needs of our agency and the communities that we serve. So, NIST or the National Institute of Standards and Technology provides excellent frameworks for supporting your agency in access, security, and privacy, including the Privacy Framework, Cybersecurity Framework, and Digital Identity Guidelines. These have actually all been created or updated in the past few years to tune to emerging needs and technologies. We do not have time to go into them in depth today, but I encourage you to spend time with them if you're not already familiar.

[12:46](https://youtu.be/jeVnawryyTM?t=766) But introduced in the base volume of the fourth version of NIST Digital Identity Guidelines, the Digital Identity Risk Management or DIRM process is a tool to help teams managing online services effectively understand and manage risks, understand what types of risks can be addressed by identity management functions, and tune those functions to the specific service context. A risk management approach, as opposed to a more text box style compliance approach, actually allows agencies to select processes and protections that work for them and their community. This can really help you make sure that your identity controls are not arbitrary, that you've tuned them to those specific needs and risks, as well as document decisions that you've made. This is a great process to undertake when you're launching a new application, you're adding new features, you're integrating something like a single sign-on, as well as when you're considering reusing account and proofing data in any capacity in a way that it might be reused and shared. And as you take on a DIRM process, you're going to want really a multidisciplinary team from your agency and thinking everyone from technical to program to even frontline staff to be included. I'm not going to walk you through the DIRM process in detail due to time today, but over the past year, we've developed a tool for assisting agencies in their DIRM processes.

[14:13](https://youtu.be/jeVnawryyTM?t=853) It lays out all of the steps in an Excel workbook. It's actually developed to support agencies with integrated SNAP and Medicaid applications, but could be used for other programs and purposes as well, and also can really start as a baseline of the process that you are very welcome to add on to. To close up, I pulled from many of the publications and resources that we've put out over the last couple of years. You can find them on the Digital Government Hub under our digital identity topic. We also have a data and data integration topic, and so encourage you to check those out. Thank you so much for allowing me to be here and present to you today. Please feel free to be in touch with any questions and also in the conversation later. And please feel free to join the Digital Government Network. So I'll pass back over to Rob.

**Rob Asaro Angelo**

[15:06](https://youtu.be/jeVnawryyTM?t=906) Thanks so much, Ariel. First of all, great presentation. Thank you. It's been great. It's been very valuable being involved with the group there. And I really appreciate you giving us something to take away from today, not just your great presentation, but the links on there provide real tools and resources that you can use in your agency. So that's a great takeaway for folks. We were very involved in the identity here at the New Jersey Department of Labor through a very tumultuous time, like many of my colleagues. So it's great having that community of practice, number one. And like I said, not just putting out studies about it, but actually giving real things that folks can use. So I really appreciate that. Dealing with our ID verification is one of the hardest things we've had to deal with over my eight years as Commissioner in the Department of Labor. It involves so many different functions, so many different program areas. I really appreciate you talking about SSO, which seems great, but there's so many different users across the state government and you end up being in conflict with maybe another agency or what they think it should be. So just really, and the other thing I was going to say before we move on to Chris is that hold your vendors to account.

[16:09](https://youtu.be/jeVnawryyTM?t=969) That goes for all things, obviously, but this is a point of friction for your users and your claimants, whatever program you're doing. So we really had, I'm not going to mention vendors, that's not what I'm here for, but we had a good relationship with ours. We were sort of learning together as we went through the COVID rush, trying new things. And it was just, I think for all your things you're doing, always make sure that you're holding them to account, but also offer them advice. You know, they're not always experts on how they deal with this stuff. And they've been really open to hearing from us and hearing from our cyber fraud unit. We have a lot of learnings back and forth, so I hope you get the same out of yours. Okay, before we move on to Chris Correas, I want to give a plug for the Code for America Summit. I'm excited to be going. So if you haven't gotten your tickets yet, make sure you get on there and go. There's going to be a bunch of amazing panels. So I know Chris didn't want to do that. It's not his presentation. I'm doing it for him. But Chris, go ahead with your presentation. I appreciate you being here so much.

**Chris Correa**

[16:59](https://youtu.be/jeVnawryyTM?t=1019) Oh, that's great. Thank you so much, Rob. Yeah, and you saved me some words. I appreciate that. My name is Chris. I lead engineering at Code for America. We build technology that helps government deliver services more effectively. And yeah, just really grateful, both for some of those summary comments, Rob, and also Ariel, the students did such a great job describing identity proofing, authentication, how you confirm somebody is who they say they are And just want to note that effective identity proofing at the front door. and stronger user and client data up front will undoubtedly improve the kind of user disambiguation problem that I'll be digging into here. But really, I'll be trying to dig into, like, what happens when a person already has multiple records, perhaps across multiple systems, and none of those systems know that they're the same person. That's the part of the journey I'll be trying to address today. And interestingly, like, Ariel introduced some concerns around equity and security that show up here, too, just in a different form. So let's jump into it.

[18:00](https://youtu.be/jeVnawryyTM?t=1080) I'm just going to start by showing you all some data. So these records are heavily de-identified, but reflect a real-world scenario we encountered where a single person was associated with 12 different records over the years across two different systems. So each individual record includes at least one difference from the others. So I'm going to encourage you all in chat just to take a look here and share, like, what are some of the differences that you notice or something that you see across these rows that might make it challenging to actually connect these records? Yeah, so each has its own unique ID associated with the person. There are missing data. Yeah, there's some date transposition issues. So the date of births are not consistent. Yeah, there are no values, but there's also missing data in disguise, right?

[19:04](https://youtu.be/jeVnawryyTM?t=1144) So, like, you see a social security number. That's all zeros. The addresses are not consistent. They change for good reason, but they're also structured differently. Okay, you all get the sense of, like, what the challenge is that we have here is that there's no easy way just to do an exact match in this case and link all these records, even though, in fact, they all do belong to the same person. Yeah, sorry, Rob. I know it is a bit triggering, but this is the world that we live in sometimes. This is what we're trying to get to, where a number of disparate records on the left can be resolved into real people on the right, real entities. That's why we sometimes call this entity resolution, so that every record can be attributed to a correct, valid, real-world person. In this case, there are two people that are reflected in 11 or 12 different records.

[20:06](https://youtu.be/jeVnawryyTM?t=1206) So how do we get from that messy data table to this kind of outcome? There are a range of tools and approaches to consider, and so I'm not going to go into great detail here in the interest of time, but here's some of the approaches you can think about when faced with challenges of linking person records within or across agency data sources. So up top are a couple ideas really related to data governance. So there are ways to normalize your data. That will help with any kind of matching algorithm that you ultimately deploy. There is data enrichment, which in a previous example, we saw that many of those 12 records were missing information. And so data enrichment is the process of filling those gaps using external reference data. And one warning caveat here, earlier Ariel mentioned some of the equity concerns with knowledge-based verification. The companies who provide data enrichment services are typically using the same databases that we use for knowledge-based verification. So the people who struggle with knowledge-based verification at the front door might also be the same people whose records are hardest to enrich downstream.

[21:13](https://youtu.be/jeVnawryyTM?t=1273) But nevertheless, this is a strategy for making your records more complete, which can assist with the record linkage problem. And finally, there are different approaches to actually matching these records. You can use deterministic rule-based exact matches, and this can often get you pretty far. During the pandemic, Code for America supported some pandemic EBT efforts. We had to match student records across multiple data sources to determine which families were eligible for assistance. And we initially tried a rules-based exact match approach, but we were able to successfully link 84% of the records we wanted to match. That's not nothing. But the remaining records required more sophisticated matching algorithms and relaxed criteria. And so that takes us to fuzzy matching. There are a lot of different algorithms and techniques here. Most commonly, a string similarity algorithm can match very similar text. And this is good enough to note the similarity between, for example, Anne or Annie, or some of those kind of very similar addresses that might have had slight differences.

[22:15](https://youtu.be/jeVnawryyTM?t=1335) There are phonetic algorithms to evaluate how close fields are based on how they sound. And then I just want to note that names can be particularly challenging. So there are dedicated commercial software solutions that are trained on billions of names and are effective in matching across scripts or transliterated names. They can handle common nicknames. They are aware of cultural differences in name structure and sequence. And these tools are often available within some database platforms, are often embedded in probabilistic entity resolution But just noting that if you're trying tools. to match on names, that can be particularly challenging. There are a whole family of tools that can help with that. And finally, there are what I'm calling probabilistic entity resolution approaches. These are systems that deploy statistical models to weigh all the available evidence. They can chain attributes across intermediate records to identify clusters of real world entities. They might weigh the rarity of evidence. So for example, they might be aware that a common name, such as Robert Black, provides less discriminative value than matching a highly unique name.

[23:20](https://youtu.be/jeVnawryyTM?t=1400) They can often self-correct in real time. So as new records enter the system, it can recalculate the entity graph and actively split records, merge records, and so on, on an ongoing basis. So that as you include more data, these solutions become more effective. I can quickly walk through what this looks like at scale with an example. There's lots I want to get to, so I'll just be quick here. But as I mentioned before, we have implemented or advised an entity resolution to support nutrition assistance eligibility. We've helped states research program participation gaps across Medicaid and other programs in one state. We've implemented an entity resolution to streamline SNAP caseworker workstreams by linking documents and deduplicating case records. And also, several years ago, we started working with a court system to automatically seal court records for individuals so they could increase their access to jobs, housing, and education. Now, the eligibility requirements limited this service to individuals who only had a few convictions, did not have any open cases, and had remained conviction-free for a period of time since their last conviction.

[24:23](https://youtu.be/jeVnawryyTM?t=1463) So in automatically deciding if a case met that criteria, the software actually needs to understand all of the cases associated with a person who is accused or convicted. As you might have guessed, this was pretty challenging. There were many courts, many different data entry practices over the years, and there was no reliable identifier connecting individuals across cases. So we worked with the court to configure an entity resolution solution. In this case, we used the sensing software that was able to link records across court databases in time. We discovered that those 3 million individual person records in the court systems actually represented about 1.2 million real people. And once we have that person-level information, we can deploy the software to automatically determine eligibility on a case-by-case basis. This is fundamentally the same pattern you'd use for cross-benefits program coordination. You can replace courts with other kind of program and agency names. The architecture would look pretty similar. But notably, before we could really deploy this in partnership with the state court, the judiciary wanted proof that this could actually solve the problem.

[25:28](https://youtu.be/jeVnawryyTM?t=1528) So we partnered with university researchers to create some test data, a ground-truth data set, and compared our automated results against human analysis by paralegals and other experts who had knowledge about the process and relevant systems. So we did a validation study. The automated process for entity resolution was faster, but not perfect. And during this early prototype, the experts did find one case where the entity resolution solution But here's the part that surprised some folks. made a mistake. The automated system wasn't just faster at matching those records. It was actually more accurate than the human experts doing the same work. Ultimately, this validation study is what gave the judiciary confidence in the solution and ultimately enabled automatic relief for hundreds of thousands of individuals. Now, just one note here is that there's often an assumption that manual review and human-in-the-loop is a safe option, and automation with emerging technology is a risky choice. That is often and usually the case, but not always.

[26:31](https://youtu.be/jeVnawryyTM?t=1591) And because we had a ground-truth data set that allowed us to interrogate and validate the matching, we could build shared trust in the solution. And that brings me to one of the most important recommendations I can give you if you're considering a record linkage process is create a ground-truth data set. You could start today. It doesn't have to be huge. But this is just an answer key. It's a representative sample of your own messy data where experts, especially caseworkers, QC staff, or others can help you determine the right answers, the known matches, the known non-matches. And you can start this before you choose any tool or technical approach. It costs nothing but time. But if you have this data set, it will empower you to actually evaluate tools against the reality of your data, and will help you build confidence with some objective measures. So those are some of the solutions.

[27:33](https://youtu.be/jeVnawryyTM?t=1653) I want to talk quickly about your systems, because a common mistake I see is folks jumping to the tool question. Should we use this approach or that approach, this software or that software, before understanding what your data and your situation can really support? So your strategy for disambiguation should reflect the qualities of the data that you are working with. Here are six of the considerations I think about. This isn't exhaustive, but it's a good start. And so as I walk through a few of these, think about your own data that you might be interested in linking, either within or across systems. Where do you land on each one? The first item on the left is stable identifiers. So what percentage of your records have reliable state ID, social security number, date of birth, other kinds of what I call stable, non-changing identifiers? This is the single most important dimension, because it really determines the ceiling for any kind of deterministic exact rule-based matching. If there are inconsistencies in the data, this is where data normalization can help.

[28:36](https://youtu.be/jeVnawryyTM?t=1716) So that's where that format consistency comes in. Field completeness really reflects how much data is missing. If it's a lot, this might require data enrichment And then all the considerations on the right side or other strategies. of this screen are where I start to think about these larger, more sophisticated resolution solutions. One is identity variation. And by this, I don't mean typos. I mean legitimate name differences that can occur. If name is an important criteria for matching, to what extent will marriage or other life events, transliteration, cultural naming conventions, there are a lot of factors that might result in variation in how somebody's name is presented. And sometimes the populations with the highest variation might be the folks that you're actually most in need of coordinating services for. Record density simply reflects how many records per person do you expect. If it's two or three, pairwise matching works. It might be really straightforward to deploy exact matches or fuzzy matching. But for the many different representations of a person, like the example we looked at in the beginning, you might need an algorithm to build entity clusters, not just pairwise scores.

[29:40](https://youtu.be/jeVnawryyTM?t=1780) Because matching record A with record C might require information in record B. And then finally, temporal span. Linking records from across years is a different kind of problem. So over time, there are lots of opportunities for information to change. So this can also amplify some of the other challenges on this list. Think about which of these six dimensions is the biggest kind of obstacle or opportunity in your data right now. These are just some of the factors that will help determine the right strategy or collection of strategies for your situation. That leads me to my next recommendation, which is simply rather than jumping to a solution that will be fastest to implement or jumping to the most sophisticated solution, just think through the data that you have. Create that ground truth data set, interrogate it, reflect on these different kind of dimensions so you can focus on where the opportunities are that are best going to help you and the folks that you're serving. Those dimensions of data quality tell you what's possible. I just want to talk about what's acceptable because every matching system makes mistakes, I promise, and the kinds of mistakes matter. When you're disambiguating records and linking these records, you have to make a choice about how you're going to choose to measure accuracy. Here are a couple common terms you can think about. One is precision, which is an accuracy metric that answers, of the matches we made, how many are correct? So high precision means you're avoiding any false positives. Recall is a different metric that answers, of all the true matches out there, how many are we actually finding? So high recall means you're avoiding any missed connections. And you can tune the technology, every solution, to increase the precision or increase the recall, but these are genuine trade-offs. It's not a limitation of any technology, it's a statistical property of classification. You can't maximize both at the same time. So two questions should be driving your choices. What happens when you get

[31:46](https://youtu.be/jeVnawryyTM?t=1906) a false match? What happens when you miss a true match? Who bears the cost? What opportunities are lost? I'll share some quick classic examples. So for example, in medical screening for serious conditions, initial tests are optimized for recall. In other words, they don't want to miss out on any true positives, and that comes at the cost of a lot of false positives. In that case, that's a choice that makes sense. A spam filter for email is the opposite. There you want to optimize for precision because you want to be really confident in email spam before it gets moved to your inbox. You don't want to miss an important message. So that's where you might optimize for precision. But we see these trade-offs all the time in human services as well. Here's what a simple decision framework might look like. So first, if the worst outcome is erroneously linking two different people, then your metric should focus on precision or the percentage of matches that are valid. And sometimes exact matching with strict thresholds might work here, especially if you have strong data governance or can invest in some of those data quality, normalization, enrichment, and so on strategies.

[32:49](https://youtu.be/jeVnawryyTM?t=1969) If the worst outcome is missing a possible link between two records, that might mean fuzzy matching with lower thresholds for a positive match might be aligned with your goals. This might mean you accept some overmatching. But that could be acceptable, particularly if there's human review that can mitigate the overmatching risks. And if both are generally costly, and this happens all the time, if both errors are expensive or risk harm, you should really consider a layered approach. This will often require sophisticated entity resolution purpose-built software. In this case, the F1 metric is something that you can focus on. This resolves the tension between precision and recall by combining them into a single number, calculating the harmonic mean between the two. And so in your domains, really think about what feels most costly, a false positive or overmatching or a false negative undermatching. Which column are you in? Let me leave you with a quick summary. Here's some of the steps you can take right now or tomorrow if you're trying to introduce or improve record linkage. First, how are you defining accuracy? What we just talked about. That should drive every technical decision that follows. Just be really clear about that.

[34:02](https://youtu.be/jeVnawryyTM?t=2042) Next, use those dimensions of data quality we talked about to help you understand what solutions are aligned with the data that you're trying to link. And then finally, create some data. Find your most challenging cases, bring in caseworkers and others with deep knowledge of your system. This is how you can start to evaluate approaches and build trust in the solution that you land on. The QR code here links to information about how Code for America is leveraging AI to unlock government capabilities. Please reach out if you have any questions or ideas. I'm really happy to take any questions on any of the content or topics here. Thank you.

**Rob Asaro Angelo**

[34:36](https://youtu.be/jeVnawryyTM?t=2076) Thank you so much, Chris. I know that our audience here always appreciates good case studies, so I appreciate that kind of real-world information on there. And by the way, when I said that spreadsheet was triggering before, I didn't just mean as a long-time government and campaign worker trying to data match things. As somebody with a hyphenated last name whose name is always misspelled and mispronounced, there's probably as many variations of my name on that sheet as there are people on there. You know, it's just always something that's frustrating from the user

**Chris Correa**

[35:04](https://youtu.be/jeVnawryyTM?t=2104) side and from the government side. So, appreciate it. Yeah, it's a real problem. We, in that same data set where we had those 12 records, we found somebody with 56 different representations of a name. So, it can be, it can get challenging at times for both the, for both the clients and folks that we're trying to serve. And of course, for those of us who are trying to improve these

**Rob Asaro Angelo**

[35:24](https://youtu.be/jeVnawryyTM?t=2124) services by using that data. Right. And just real quick to my colleagues on here who are from human services or benefits groups. And, you know, I think that ID verification is always something that's, you know, it brings up a lot of anxiety for people. Right. And it's something that we hear about from our legislators a lot. Oh, we hate this process. Why do they have to do this? I think it's always important to say, not only are you trying to fight fraud and fraudsters, you're trying to protect their benefits. Make it individual and specific because somebody is getting into someone else's unemployment account. When the real person goes to get there, there won't be any benefits for them. So, I would make sure that on an optic side or, you know, PR side, I always talk about we're saving the person's individual benefits when we're doing that work. That's why that little extra step is so important. So, thank you both for your presentations, Arlan and Chris. One of the first questions on the Q&A, I want to hit that because there's going to be a lot more on there. And so, I don't know the answer to, I know all the vendors say, oh, we got this, don't worry. But what is the impact of AI deep fake for facial fingerprint or other identity? I don't know who wants to take that, Ariel or Chris.

**Ariel Kennan**

[36:26](https://youtu.be/jeVnawryyTM?t=2186) Yeah, happy to take this. I mean, the same way we're able to use some of these tools for fun and making silly videos, people could actually make a video that looks just like me right now, that's me as the live selfie as I'm trying to verify for something like unemployment insurance. There's a great article, I'll try to find the link to it here in a minute, from 404 Media that came out last year, that's actually about their companies that are making fake IDs. So, they're digital fake IDs. You can change the color of the carpet and the flooring on them, which had been one of the make it safer things from the identity verification companies had been like, we were always looking at different carpet and does it look real and all that kind of stuff. Well, you can generate a lot of that stuff with AI now. And so, I think it comes back to needing the either multi-factor authentication, so some of those proving, but also multi-factor when you're looking at verifying somebody's identity, that it goes through multiple points. And that you're looking also at where the traffic's actually coming from. If it's from a different country, for example, it's likely not a legitimate actor all the time. And so, being able to look at different vectors basically around that. But I think an emerging thing as the models get more advanced, especially the ones that are more image generating, that they can create a lot of different things. I've seen some really cool demos though, from companies that are, when you're doing the live selfie, they're putting in other light patterns that are unique, color signals, things like that, that help disrupt. So, it's a different generation, but you're able to actually match the generated thing to the generated thing for legitimacy.

**Chris Correa**

[38:20](https://youtu.be/jeVnawryyTM?t=2300) Great. Chris, anything to add there? No, that's great. That's a great overview. Thank you very

**Rob Asaro Angelo**

[38:27](https://youtu.be/jeVnawryyTM?t=2307) much. Great. And hopefully, folks, this call, this is the concern of your vendor, and you're not going to be the one doing the different lighting and color binding things that Ariel talked about. Just real quick, you had a case that was our personal journey in New Jersey verification was really like most people during COVID. And what we had, we had a large backlog of folks that we had flagged through our normal processes, through the information processes, and it was getting into the tens of thousands. And we didn't have the manpower to sift through that. We knew that the majority of them were fraudulent for various reasons, but we knew there had to be false positives there too. Now, in the past, if it was 1% or 2%, it'd be no big deal because we could deal with that by hand. But now we're talking about hundreds of thousands of claimants. We're talking about thousands upon thousands of folks who weren't able to receive their benefits. That's when we moved to a digital identity verification vendor, mainly to get us to that backlog. And we made the policy choice, which you talked about being so important in there, that everybody had to go through the same process. If there wasn't, then we weren't going to have a risk score and then throw them through the process. If we want to have a really truly equitable system, that everyone was going to go through the process the same way. Because if not, we'd be sort of picking winners and losers just by having to go through that process.

[39:43](https://youtu.be/jeVnawryyTM?t=2383) And I think it's something that sticking by that helped us, helped me, with legislators and other folks, stakeholders who were upset about the process in general, but also was a specific use case to us. We're trying to clear a backlog that we knew was mostly, we were confident was fraudulent, but we knew there was a significant number of folks in there who were false positives or frauds, so we had to move through there. Yeah, I really love the question here from, sorry if I get the pronunciations wrong, Cecesco Soali, about the importance of personal data. And I think that as users now in the digital world, we're so used to our personal information just being out there and available. I mean, I say this all the time to whether it be some of my family, who's like worrying about this or that, and it could be a double full-time job to protect your data in the digital world. I've, quite frankly, I've given up. I hate saying that, but it's just true. I try to make sure that the accounts that matter or have financial information are as secure as possible, but it is something that is, I guess the way that this person puts it, is sort of devaluing my own personal information. So how would you go about educating the public about the importance of their personal information so they begin to value platforms, actually steer away from abusing their data? I'm going to throw that to Chris, who's muted. There you go. Yeah, I mean, it's a great,

**Chris Correa**

[40:58](https://youtu.be/jeVnawryyTM?t=2458) I don't have an easy answer for this one. I think it's an important observation. I do think, to Rob's point, a lot of folks are feeling, I know I talk to a lot of folks who are not feeling optimistic about the ability to kind of protect personal information. I don't really have any, it's a little bit outside my expertise on like how to educate the public, but I do think it's

**Ariel Kennan**

[41:24](https://youtu.be/jeVnawryyTM?t=2484) a really important consideration and effort. Ariel? Yeah, I mean, I think one thing that was really interesting from going out and talking to beneficiaries about their experience with digital identity was to see how sophisticated actually many members of the public are. And granted, this research self-sampled to people who were willing to talk about technology, so they probably have a little bit higher digital literacy. But I do think now that some of the operating systems, so that like Apple, for example, and Google, you have a built-in password manager in your phone, which is great for helping protect accounts and security. It creates unique passwords, it stores them. That used to be like the thing that was left for the technical nerds that we used one password or last pass or whatever. But now that it's actually embedded in the phones, a lot of the people we talk to were using password managers. And so it just becomes a default tool. They create their own set of problems if you lose that device or you lose access to your main account. But if you're keeping that main account super secure, ideally that's also the case. Also to give Apple a shout out, I think they've done a great job of having better notifications in the phone when your location's being tracked, when your data is being shared, when you're in the app store, even warning you upfront that this app uses this data about you. And so I think having it built in from default into the operating system, but I also think if we're thinking about those privacy-first protections in government systems and making it really transparent how your data is going to be used and how we're going to keep it safe is, I think, really high value to the people we serve. Great answer. I'm going to go

**Rob Asaro Angelo**

[43:08](https://youtu.be/jeVnawryyTM?t=2588) backwards here. Question from David. Have there been any significant federal policy efforts to standardize data forms and formats? I'm going to throw this to Ariel, but I think the answer is right, but not specific. It's not enough, that's for sure.

**Ariel Kennan**

[43:22](https://youtu.be/jeVnawryyTM?t=2602) I am not going to speak well to specifics, but yes. I mean, actually quite a long time ago, golly, coming up on, I don't know, 15, 20 years ago, federal HHS really started investing in data interoperability and helping support how that works at the federal level and systems requirements to support it. But knowing at the state and local levels, that's still really challenging. And so thinking through how do we continue to make it actually work where the services, at the point at which services are delivered and administered. Yeah, I don't think

**Chris Correa**

[44:04](https://youtu.be/jeVnawryyTM?t=2644) there's like a, I don't know if I'd go as far as a federal policy efforts, but there are a lot of standards and best practices that are deployed, which are really helpful. It can be really helpful to the extent that vendors and states have real levers to actually implement these

**Rob Asaro Angelo**

[44:22](https://youtu.be/jeVnawryyTM?t=2662) standards and increase interoperability. I know we're not going to name a specific vendor or tool as far as, but are there favorite software tools you might recommend? Is there anything generic we can say here, Chris, or maybe some kind of open source thing or basically

**Chris Correa**

[44:38](https://youtu.be/jeVnawryyTM?t=2678) a type of tool? Yeah, I can share a couple ideas. I think it's actually pretty easy to get started with both free and open source tools and some of the commercial products. There's usually a free license or limitation. We mentioned like the Sensing product that we use in some states, you can process a hundred thousand records for free using that tool. But there are also free and open source solutions that can address something like the data normalization challenges and also entity resolution. So, in the UK, the Ministry of Justice actually released an open source Python tool called Splink, which is an entity resolution tool. It's really well documented. Even if you don't deploy it, the documentation teaches a lot of the core concepts and can help you test out and move forward some entity resolution initiatives. And then there are a lot of open source libraries and services that help along the way. And so one that comes to mind is a software program called LibPostal. It's open source. It's a statistical machine learning model that will help you parse addresses. It's based on a global database of, I think, billions of addresses. And so it will take some unstructured address data and return some structured normalized address data. So, when I talked about data normalization earlier, that's a good example of a tool that you can use just to clean up and standardize how that information is formatted.

[46:11](https://youtu.be/jeVnawryyTM?t=2771) And of course, there are a lot of services, too, where you can send that data to an endpoint and

**Rob Asaro Angelo**

[46:15](https://youtu.be/jeVnawryyTM?t=2775) get that kind of service as well. Great. Thanks, Chris. I see a question here. Have you seen many states use Login.gov as an SSO? Just real quick, from my own personal experience, in the Department of Labor, our Arts and Department of Labor, which is funded by the U.S. Department of Labor, they have a specific use case for Login.gov when it comes to unemployment insurance. That's sort of changed over time. So, we have not gone the Login.gov route. Unfortunately, we love the idea that you're offering USPS in-person verification, but the DOL in this administration changed that. You had to use Login.gov to go down that path, and we just didn't see the value in changing everything up for that, quite frankly. It just would be too disruptive. But just, I don't know what your experience, Ariel, or others, when you hear about other agencies. I always hear mixed bags. I think it really depends on how the agency is using it to determine whether or not they're happy with it. Yeah. I don't know the current

**Ariel Kennan**

[47:10](https://youtu.be/jeVnawryyTM?t=2830) number of states, but as you said, Rob, it's mostly unemployment insurance. Also, there is a DOL grant program. Actually, I just saw that that was extended, but it does put some new requirements on the data and the service. I think also, I've heard some feedback from states on kind of overall interoperability and some of the specific use cases that states might be needing to address. I had talked to a state a while ago that was actually needing to have a service that minors could use, and Login.gov does not actually support that currently, and so they were having to do some workarounds around it. One of the really great benefits of Login.gov was the partnership with the Postal Service as a route to in-person verification, which obviously not everyone needs, but a lot of people need, and most people live very close to a post office, and so it really does offer a great option for doing proofing.

**Rob Asaro Angelo**

[48:12](https://youtu.be/jeVnawryyTM?t=2892) Great. Actually, I have a personal question for you, Chris. In your presentation comparing the software on the judicial example, what was the difference in that two and a half percent between the human checking and the program, that the program was getting right, the humans were getting wrong? I'm just curious. You mean the difference in accuracy?

**Chris Correa**

[48:33](https://youtu.be/jeVnawryyTM?t=2913) Yeah. Well, I can tell you that the scale that we're working on, it was maybe two percent, maybe a little under two percent, but that translated to 60,000

**Rob Asaro Angelo**

[48:45](https://youtu.be/jeVnawryyTM?t=2925) determinations that we think that- I'm sorry. Was there a standard thing that led to that discrepancy? Was there something that the humans were consistently missing or vice versa, or was

**Chris Correa**

[48:55](https://youtu.be/jeVnawryyTM?t=2935) it just- Oh, yeah. No, I don't think there was one. There wasn't one thing that was missing. I think the real challenge is just that it can be difficult to chain evidence across multiple records. In other words, if we look for an individual, it might take two or three steps of logic to infer that, oh, actually, there's somebody living at the same address at this time who had this name. I don't have their date of birth or social security number, but actually, another record related to that record. It's kind of like the evidence chain problem that humans will typically go a couple steps deep on, but maybe not investigate all the different possibilities. I think that's really, for both entity resolution and AI overall, that's some of the advantages is that for these repeated tasks that just involve a lot of information search and synthesis, there's some advantages.

**Rob Asaro Angelo**

[49:50](https://youtu.be/jeVnawryyTM?t=2990) Great. Thank you. A question from the Q&A, Saurabh Chakraborty. Talk about different kinds of uses of data for identity, whether it be for giving us benefits and services, or I think looking at some kind of research. When setting up technology to identify people and link records for the purpose of administering benefits, are there considerations to keep in mind to allow for assessment as well on the other end of the spectrum?

**Chris Correa**

[50:13](https://youtu.be/jeVnawryyTM?t=3013) Yeah. I mean, I think we've supported state agencies in doing both, right? So to kind of use this technology to streamline or improve benefit or service in some way, and also to analyze program participation, for example, across different agency or program data sets. If the question's asking if you can do both at the same time, I think yes. I think the real, what we often see, again, to take it to this precision or recall question, the incentives or the kinds of errors that you're willing to make in one use case are different than the other. And so as a general statement, you might really value precision more in terms of administering benefits, for example, but in order to kind of analyze program participation, you might want to relax the standards and make sure you're making the best guess and so on. And that's fine. If you're doing things at the same time, I think being able to like track the rationale, most of these tools will have some kind of observability where you can see why the match was not made or why it was made and some kind of confidence level.

[51:17](https://youtu.be/jeVnawryyTM?t=3077) And so if you store and track that information, you can end up using that same kind of output for the kind of service delivery as well as using that same kind of output for the kind of service delivery as well as using it for the assessment. You might have like different thresholds for what you consider a match or not in those two

**Rob Asaro Angelo**

[51:34](https://youtu.be/jeVnawryyTM?t=3094) different use cases. Great. Thanks, Chris. I see a question here from Stephen, and I think it's a great question that we deal with a lot, about SNAP recipients who want to file a complaint about fraud, but we're worried that it was going to track back to them and they would somehow be denied benefit or get in some kind of trouble. That's something that we've run across here. I mean, we have a very public phone line to call about fraud, but we've certainly seen folks who are complaining they were defrauded and we do a real serious check and subpoenas and everything. But that's a rarity. I'm just curious if you have any knowledge of how other states handle this when folks are complaining about fraud. Is it anonymous? It's also a challenge for how much we're allowed to tell this person who's making this anonymous complaint about what we're finding. Do any best practices there come up in any of your convenings, Ariel?

**Ariel Kennan**

[52:22](https://youtu.be/jeVnawryyTM?t=3142) I don't know that I have known examples that I know of this. I do think that agencies using privacy-preserving techniques and thinking about how you do collect data about callers, about people who submit web forms, and follow up appropriately on investigations.

**Rob Asaro Angelo**

[52:46](https://youtu.be/jeVnawryyTM?t=3166) Great. Thank you. We've got to wrap up here, but I'll see one quick question here from Brendan about giving an example using Mint and the app would allow them to grant access to different bank accounts and could use this in government departments as well with data-sharing agreements. Listen, I mean, what you're talking about is sort of the holy grail for a lot of us in this world, but in the end, so many programs we'd like to be able to share data with, there's restrictions on that from a federal or state statute reason, or there's other kinds of barriers. I mean, it seems like it was a constant thing with us. We're a huge data storage here in the Department of Labor in New Jersey for wage records and a bunch of other stuff. We're always looking to add to that from other departments, and so you sort of hit or miss by which department you're going to. I presume that everyone has that same kind of issue and that variety in their state as well, but certainly I've always felt that, and our statute actually says if it's data that's for government use, we're allowed to share it with other entities in our state government with very few exceptions. So I really would push back on the initial reaction from a government department to say, oh, no, we can't share that data with another government agency, I mean, obviously, but I think in most cases, there's a way to do that internally.

[53:56](https://youtu.be/jeVnawryyTM?t=3236) But now we're at the end time here. I want to thank Chris and Ariel so much for your time today, for having real use cases, real takeaways for our audience here today. Everyone will get a link for all the stuff that's available for replay, and make sure you join us on our next session. Thank you all for being here today, and I appreciate our presenters and look forward to seeing you all next week. Thanks, everybody.

* * *

Original page: https://innovate-us.org/the-identity-challenge-tackling-user-disambiguation-and-data-integration-across-programs · Licence: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), as stated on innovate-us.org