InnovateUS Hosts Workshop Series for the Public Sector on Program Evaluation

This October, in response to learner feedback, InnovateUS launched its first multi-part workshop series, which enabled participants to learn about program evaluation in depth. The three-part program evaluation series attended by over 270 learners was led by Dr. Deborah Stine, Founder and Chief Instructor of the Science and Technology Policy Academy, and provided a thorough overview of program evaluation for the public sector, from the essentials of the six fundamental steps of program evaluation, to the complexities of navigating stakeholder and community engagement. 

Dr. Deborah D. Stine is the Founder of the Science & Technology Policy Academy, which provides independent consulting, policy analysis, program evaluation, freelance writing, teaching, and coaching services. A Fellow of the American Association for the Advancement of Science (AAAS), she previously served as Executive Director of the President’s Council of Advisors on Science and Technology in the Obama White House, a Science & Technology Specialist at the Congressional Research Service, a study director at the National Academies of Sciences, Engineering, and Medicine, and a professor of the practice in engineering and public policy at Carnegie Mellon University. She has a BS in mechanical and environmental engineering, an MBA, and a Ph.D. in public policy analysis and public administration. 

Part 1: Understanding Program Evaluation Essentials

Thursday, October 5th, 2023 

To kick-off the session, Dr. Stine discussed how program evaluation can empower local and state employees. In an interactive activity, participants were provided a prompt to insert into ChatGPT which would uncover real-world examples of how program evaluation has helped government employees in the participants’ states. The participants shared their findings, and Dr. Stine provided three examples of her own. 

Dr. Stine then explained some of the reasons behind program evaluation, including the following:

• Accountability - Often required by funding organizations as part of funding conditions. 

• Developmental - Making the case for more funding to potential funders 

• Knowledge - Reviewing an existing program to identify improvements to increase effectiveness, equity, and efficiency. 

Dr. Stine then outlined the 3 E’s and I’s for assessing past, present, and future programs. The goal is to understand how well a program is working in the six domains listed below.  These program evaluation findings can then provide a basis to modify a program in the future or make the case that additional resources are needed to reach a societal goal. 

3 E’s

• Effectiveness in reaching program outcomes and societal impact 

• Efficiency in getting the best bang for the societal buck

• Equity in treating all stakeholders fairly

3 I’s

• Indicators of need

• Implementation and delivery 

• Improvements to increase potential of reaching societal goals

Types of Program Evaluations 

Dr. Stine explained that programs can be evaluated both internally, to test the health of and improve an existing program, or externally by a third party, typicallytas a condition of funding already received or to make the case for futrure funding. 

These program evaluations are facilitated through two kinds of systems:

1. Logic Models that assess the inputs, outputs, and outcomes of the program 

2. Qualitative or Quantitative Data Collection and Analysis (or both, known as ‘mixed methods.”)

The Six Steps of Program Evaluation

Program evaluation can be broken down into 6 procedural steps. Dr. Stine explained that the first three steps can be done in any order depending on the status of the program. This cycle should not be seen as a one-off method, but rather a recurring process. Dr. Stine explained “program evaluation is really a big circle, because you are constantly looking how to improve the program toward reaching its societal goals.”

Part 2: Navigating Stakeholder and Community Engagement in Program Evaluation

Thursday, October 12th, 2023

In the second session of this series, Dr. Stine honed in on step 1 of the 6-step program evaluation framework from the first session: engaging stakeholders. Dr. Stine explained that for some people, engaging stakeholders and communities can be a challenging proposition when evaluating programs. Stakeholders are defined as any group or individual who can affect or is affected by the actions, decisions, policies, practices or goals of an organization. Whereas community is defined as a group of people sharing common characteristics, interests or geography. 

Who are the stakeholders?

Stakeholders fall into one of the following four categories. 

1. Implementers - This group includes program directors, managers and staff, volunteers and consultants who have developed the program being evaluated.

2. Decision Makers - This group consists of policymakers, agency leadership and funders.

3. Those served or affected by the program - This group covers clients, family members and neighborhood organizations.

4. Partners - This group could consist of academic institutions, advocacy groups, professional associations, skeptics, opponents of the program, staff of related agencies or competing agencies 

Dr. Stine walked participants through an important process known as stakeholder mapping, which allows program evaluators to refine the often extensive list of communities and stakeholders that are created in defining a program and its reach. Mapping stakeholders allows program evaluators to focus on the most important and substantive stakeholders. The image below demonstrates the four stages of the mapping process, from analyzing the stakeholders to developing tailored communication. 

Part 3: Crafting Logic Models and Developing Indicators

Thursday, October 19th, 2023

In the third and final session of the program evaluation series, Dr. Stine focused on creating logic models. Logic models are systemic and visual representations of relationships between the resources available to operate a given program (inputs), program activities (outputs), and program results (outcomes). Logic models external factors, assumptions of the program, specific situations or circumstances, and priorities for the creators of the program.  Logic models can look at existing programs to see how well they are working or potential new programs to show how they are designed to reach a societal goal.

Why is developing logic models important? 

Dr. Stine provided the following reasons as to why logic models are critical tools in program evaluation and the role they play in determining a program's effectiveness, efficiency, and equity.

Logic models:

1. Guide evaluation 

2. Clarify program structure and theory of change 

3. Facilitate communication to stakeholders and communities 

4. Aid in resource allocation 

5. Support continuous improvement 

6. Enhance transparency and accountability 

7. Facilitate training and onboarding 

8. Help identify potential gaps or redundancies 

Dr. Stine then focused on the key components of a logic model and walked participants through the development process. This included a discussion on each component in the visual and an in-depth breakdown of selecting, developing, and gathering data to support logic models. 

Dr. Stine closed her workshop with a discussion on the importance of applying ethical principles to program evaluation. Some of the ways to ensure a good standard of ethics in program evaluation include the following:

• Ensure that stakeholder involvement is composed of diverse groups and communities and that cultural sensitivities are carefully navigated throughout.

• Maintain transparency throughout the program evaluation process, including the logic model structure and decision-making processes.

• Avoid harm where possible. This includes avoiding perpetuating inequalities and looking for unintended negative consequences of the program you are evaluating 

• Distribute resources fairly, considering the most marginalized and vulnerable groups and communities.

• Maintain privacy and confidentiality when conducting program evaluation, this includes data and personal anecdotes from participants and stakeholders.

• Be truthful and honest, avoid setting unrealistic outcomes for programs and be pragmatic about the resources available to you. 

Did you miss out on the workshop series? Watch the recordings here!