In this video, I'll be talking about standards of evidence. Just how rigorous is your evaluation of impact? And to describe this, I'm going to be using a framework that I borrowed and adjusted from the Nesta Foundation in the United Kingdom. Okay, so as we talk about the level of rigor, precision, quality to the data and the evidence we're gathering, I'm going to lay out five levels of rigor. Level 1 is the lowest standard of evidence, it's the least rigorous. Level 5 is the highest standard of evidence. And then I'm going to describe what I call level Q. I'll get back to that in a moment, I just want to make clear that it's 1, 2, 3, 4, 5 and then there's this other thing, Q. So, level 1. Level 1 means you don't actually have any data. You have logic, you have a good theory, but you don't actually have data to back up your claims. So we'll talk about this and we'll talk about how valuable level 1, is if you have a really well articulated logic model. So that's a concept I've talked about in an earlier video, we'll dig into it more. Level 1 is about having logic, about a theory of change, but you don't have data. Level 2 means you have data, but you only have data on outputs. So your outputs are your activities, or the organization's activities. If I were talking about my outputs as a professor, I might talk about the number of students I teach. Of course we don't know if they are learning anything, but we could count the number of students I teach. Or we could count the number of readings I assign. The challenge here is we don't know that people are actually reading them. So outputs give you pretty basic information about your activities. They don't really get you to impact. Still, you can learn something at level 2. Level 3 means that you have evidence of a change in outcomes. So you have some outcomes, you know what they were before your company, or your organization, or your program was in action. And then you can look and see wow, afterwards have changed. But you don't actually have proof of causality. So the challenge here is, you don't really know if your organization or company is having the impact you intend, right? So maybe birth weights are up for children. That's a good thing. But is it because of your nutrition program? You don't really know. Maybe graduation rates are up, more kids are graduating from high school. That's a good thing. Is that because of your educational technology? You don't know. Maybe recidivism is down, fewer people are returning to prison once they get out, also a good thing. Is that about your company's employment practices? You don't know. So that's level 3. You've got evidence of change over time, very helpful, can't really pinpoint causality. Level 4 means that you have evidence of a change in outcomes and you have proof of causality. This is when you conduct a true experiment, or what's called a randomized control trial. And here you can actually show proof of causality. I know that our program, our organization, our company, has caused these changes. Level 5 is, you actually can replicate this change process. You've got evidence of causality and you can show that it generalizes. So it generalizes to multiple settings, multiple countries, multiple populations. You might do this through a multisite research design of a meta analysis. Okay, so those are the five levels of rigor. You're getting more and more rigorous, better and better data, more and more evidence of causality, evidence of generalizability, as you go from level 1 to level 5. And now we get to level Q. I'll talk in a moment about why I call this level Q. Level Q is when you have evidence that the outcomes and impact caused by your program or company are cost efficient. In level Q, you're essentially doing a cost-benefit analysis. Or to use a term that we'll talk about in more detail. A term that's used a lot in the impact space, you have evidence of your SROI, your social return on investment. So when you do an SROI analysis, you're trying to figure out, what's my bang for the buck? How much does it cost to produce these outcomes and is it worthwhile? So you got to figure this out. You're making a lot of assumptions, you're building in a lot of data to try to figure out, well, what's the dollar value of the outcomes I'm creating? And what's the dollar value of the inputs I'm putting in? And what does that ratio look like? I call this level Q, because the rigor with which you do these analyses is really a question. You might have great data that allows you to calculate SORI with a lot of precision and a lot of rigor. So maybe you have level 5 data that you can use to feed into the equation where you're going to figure out, what's my cost-benefit ratio? Another possibility is you have really mediocre data and you're making lots of guesses and lots of assumptions. And in that case, you're SROI is not as meaningful. You can come up with a number of what's the social return on investment for your program or your company, but it's not as credible. So there's a real question about what the level is and we'll talk about that more when we get into this topic in a future video. So, the one thing I want to say in concluding this video is, as a professor, as a researcher, hey I think higher rigor is better. And higher rigor is better. It's also more costly and more time consuming. So there maybe times when you make the calculation, it's not worth it for me to do the most rigorous research design to assess impact. We'll talk about that as we go through these different levels. We'll talk about what you learn at different levels. We'll talk about why you might want more rigor. But we also recognize there are times when all that rigor, that's great for scientists. Maybe that's great for professors. You might not go there, to that high level of rigor. So in the next videos we're going to be looking at, what are each of these levels and how do you gather data, make the analyses for each of those levels?
