Now we're going to talk about different sources of coverage data. Starting with population-based surveys, as I mentioned earlier, population-based surveys are the primary source of coverage data in low and middle income countries. These are almost entirely in-person surveys for the moment, although people are working on methods for phone surveys to be used in the future. Population-based surveys are based on households, as we've talked about, they can estimate some coverage denominators. In other words, the population in need of an indicator of an intervention. They can estimate coverage numerators for some interventions and practices, the population in need that is receiving the intervention or practicing the behavior, and they provide data on other important variables that can be used as stratifies to understand which groups are or are not receiving interventions, geographic variables, the place of residence, Region, District, and so forth of the household, whether the household is living in an urban or a rural area, the wealth quintile, the level of wealth of the household, level of education or sex of the respondent, and ethnicity and there's a number of others as well. As we've discussed, population-based surveys are particularly useful for measuring coverage indicators because they sample households in such a way that all households and household members are eligible for inclusion. Essentially we're taking a picture of the entire population, this is what gives us a representative estimate of coverage in that population. Let's talk about sources of existing coverage data. Demographic and Health Survey are really important source of coverage data, these surveys have been funded by USAID since the 1980s. There currently implemented by ICF, which is a US-based company which collaborates with National Institutes of Statistics in individual countries to implement these Surveys. DHS has conducted over 300 population-based household surveys in 90 countries since 1984, this is as of the end of 2019. DHSs typically include large sample sizes, especially the more recent surveys, they are almost always nationally representative and they are usually able to be dis-aggregated to subnational level, so regional or in some cases district levels, again, particularly the more recent surveys that have larger sample sizes. In most low-income countries, and many middle-income countries, DHS is an important source of coverage data. DHS's generally have good quality data, they have established rigorous protocols for sampling, for collecting data, for cleaning, and analyzing that data. They usually collect most or all standard indicators for reproductive, maternal, newborn, child, and adolescent health and nutrition, and what I mean by standard indicators are, for example, indicators for the Sustainable Development Goals, Indicators for countdown to 2030, the WHO core coverage indicators. In some countries, DHS also supports AIDS or Malaria Indicator Surveys, those are AIS or MIS, which collect specific data on prevalence and coverage of HIV and AIDS and Malaria. They also conduct service provision assessments or spas in certain countries and those are surveys of health facilities that collect information on facility readiness and the services that are provided at those facilities. One limitation of DHSs, as with many household surveys, is that there is a fairly long lag time between when data is collected and the availability of clean datasets for analysis or the availability of the report and data collection itself can actually take a bit of time. It can take a number of months because they have these very large sample sizes. That can mean that you're having data for 2019 you may get but data only in 2020, you have to wait a while. DHS has produced an updated guide to DHS statistics that provides really useful information on their indicator definitions, how they calculate indicators, and so that can be quite useful if you're analyzing existing DHS data and their data are free and they're open access. They also have something called stack compiler, which will give you estimates of various indicators for your countries of interest without you having to analyze the data and we'll talk more about that later in the course. This figure or this map shows DHS' that had been conducted as of 2019 and basically, the darker green means that there have been more DHS surveys, lighter green means fewer surveys, great means they haven't had any. You can see that Sub-Saharan Africa and South Asia are very well-represented there are many DHS surveys in those settings and particularly in certain regions of Sub-Saharan Africa, also to some extent in Latin America, and to a lesser extent in Central Asia and other regions. This slide provides an overview of some of the topics that are covered in the DHS. DHS has four questionnaires, a household questionnaire that's administered one per household, a woman's questionnaire that's administered to eligible women in the household, women aged 15-49 years, a man's questionnaire that's administered to eligible men, and a biomarker questionnaire for surveys in which biomarkers are collected. You can see in each column the different topics that are covered in the questionnaires. I'm not going to go through them in detail here, but I would really encourage you to review the questionnaires. They are quite long and cover quite a bit of ground. As I mentioned, the DHS Program conducts a number of surveys that collect coverage data on reproductive, maternal, newborn, child, adolescent health and nutrition indicators. This is just a screenshot of the DHS program page that shows all of the surveys conducted by country and then by year, and so you can see DHS surveys, Aids Indicator Surveys, malaria indicator surveys, and key indicator surveys that all of which collect coverage data. Another important source of household survey data are MICS, Multiple Indicator Cluster Surveys. These are funded and run by UNICEF since 1995. The MICS has conducted 300 population-based surveys covering 116 countries over six rounds of data collection. The MICS is somewhat different from the DHS in that there's a strong emphasis on country ownership, and so there is a little bit more variability in the content of the survey, although they do have Core questionnaires and the implementation of the survey and analysis from country to country. It is a little bit more variable in terms of data quality, although the MICS team works hard to really have high standards for the quality of data collection. The focus of the MICS is on reproductive, maternal, newborn, child, and adolescent health and nutrition and human rights. Most of the MICS surveys measure SDG health indicators or the earlier ones measured MDG health indicators. As with DHS, there's a long lag time between data collection and the availability of the report or clean datasets, which is just in part the nature of doing household surveys and does with DHS, data sets are free and open access after sign-up. This slide shows countries that have had MICS surveys as of 2017. The countries that have sort of that light yellow highlighting are the countries that have had at least one MICS and so you can see that they cover quite a lot of ground, including almost all of Sub-Saharan and Northern Africa, many countries in the Middle East, Central Asia, South Asia, parts of Latin America. Most countries, most LMICs will have at least one MICS. This slide, like the previous one for DHS, shows the different questionnaires and topics. Somewhat similar to DHS, MICS has a household questionnaire, it has a woman's questionnaire for women 15-49 years, it has a man's questionnaire. In the DHS, questions about children are nested within the women's questionnaire. Women are asked about their children who are aged less than five years. For the MICS, they have a separate child questionnaire that is administered to the mother or the primary caregiver of the child if the mother's not in the household. You can see the different topics that are covered here that are quite similar to those in the DHS. This screenshot just shows a page from the MICS website showing all of the different MICS surveys that have been conducted. This particular part of the page shows the most recent surveys which are all in the survey design phase. But if you go to the MICS website, you can filter by various characteristics and you can see the surveys that have been conducted in your countries of interest or during the time period of interest. This slide just shows a screenshot from the MICS website showing what are currently the most recent surveys. They actually on the website have a very long list of all the surveys they've ever done or that are currently being planned, and you can filter it by various characteristics, so you can see the surveys for your country of interest or your region of interest or your time period of interest. The availability of coverage data for RMNCAH&N indicators has increased over time as there has been more focus on intervention coverage. It's a little bit small, but in the top map, you can see the availability of coverage data for seven particular indicators of interest over the 2005-2011 time period. Essentially, the countries that are colored in green had population-based data on all seven Indicators. Orange had five or six, and then the pink and the red had fewer indicators. Then the bottom map shows the same seven Indicators, data availability for the same seven indicators for the 2012-2017 period. Basically you can see it on the bottom map, there's less orange, more green, maybe a little bit less pink and red. The availability of data has been increasing despite the cost of doing the surveys. We do have more coverage data available to us now. Other survey datasets, I have alluded to these, but just to mention them specifically, we have the AIDS Indicator Survey, the AIS, that covers HIV and AIDS prevalence, well, HIV prevalence, HIV and AIDS knowledge, attitudes and behaviors. It's implemented by the DHS program. Malaria Indicator Survey, also implemented by the DHS program, looks at prevalence of parasitemia and coverage of various malaria interventions including IPTp, vector control interventions, and so forth. Other non-DHS surveys, so the SMART, which stands for Standardized Monitoring and Assessment of Relief and Transition is a nutrition-focused survey that was originally developed for use in emergency situations. So for rapid assessment of nutritional status, food insecurity, and mortality in emergency situations. They've been conducted since 2002. Some countries have adopted these and conduct them annually to get national and sub-national measures, particularly of nutritional status. They are restricted access, so they are typically owned by the country. They're implemented by the country, owned by the country, and so as opposed to DHS, they are not available on the SMART website, but you can go to that website and review the methodology and the various tools that they make available for implementation of the survey. Then we want to mention PMA2020, Performance Monitoring and Accountability 2020, which is funded by the Bill and Melinda Gates Foundation, implemented by the Gates Institute at Johns Hopkins. That is a survey that focuses particularly on family planning, adolescent health, maternal newborn health, also other topics such as nutrition and water and sanitation. It's implemented in 11 countries. What's different about that survey is that they conduct several rounds of the survey in the same cluster before changing cluster. It is a little bit faster and a little bit less expensive. It is also open access. You have to set up an account on their website, and they have more information about the survey on their website. Now I want to switch gears from talking about household survey, so population-based sources of data, to talking about routine datasets. When I'm talking about routine data, I'm talking about data typically that's collected via the government, through for example, the health management information system, civil registration, disease-specific program reporting, and so forth. There's lots of different sources of routine data in countries. A common complaint or concern has been the quality and standardization of those data, but there have been a lot of investments recently that aim to improve the quality of routine data and increase their use. For example, many countries are now using DHIS-2, the District Health Information System 2, to collect and compile routine data to improve the standardization of those data. There are also a lot of efforts to improve the quality of routine data and rethink what is collected in routine data systems. If the routine data are available, the indicators of interest are available and are of adequate quality, they can provide important and useful information for countries, programs and evaluators. Most health systems produce their own estimates of intervention coverage using their routine data. There are data that are collected usually at their health facilities through registers and records when individuals come in for care. These coverage estimates are most often available for preventive interventions like immunization, vitamin A, or measures of service contact or utilization, so ANC or facility delivery. The reason they're most commonly available for those types of interventions is the denominator is easier to estimate. Indicators of coverage of treatment are less common because the denominators are more difficult to estimate. You may get estimates of the number of children diagnosed with malaria or the number of children treated for malaria, but the proportion of all children in the country with malaria treated with an ACT is going to be difficult to estimate because you don't know how many children with malaria are out there who didn't come to the health facility. The numerators for routine measures of coverage, again, come from facility records of who came for care or what they received. The denominators are usually estimated from census data, so they are usually projections from the most recent census. The quality of those denominators varies depending on the quality of the census, how recent it was, how much population movement there has been, and also the level at which you're estimating. Usually the national level estimates are the best because you don't have to worry too much about internal population movements. It's not going to affect the national level estimates of population, but once you get down to small sub-national units like districts, and you're pretty far out from the census, you may have a significant amount of error. Numerators can also have significant error in either direction. Immunization coverage indicators are notorious for this for overestimating the number of children who received immunization. Sometimes because kids get double counted, or children who aren't actually in the target age range get immunized. But we can also undercount the number of people receiving interventions, especially if lots of individuals are getting care in the private sector and therefore not being captured in the routine system, or maybe for something like, let's see, ORS, oral rehydration solution, or taking iron supplements in pregnancy, they may just go to the pharmacy to get those. Again, it doesn't get recorded in the HMIS. One very important advantage of routine data is that they are typically much more timely than surveys. We talked about the amount of time that it takes to get survey data. Routine data are usually collected on a quarterly or sometimes monthly basis and they are often released. Estimates are released yearly, sometimes you can even get quarterly, estimates released publicly, and they're available at district level which population-based surveys may not be. Routine data are really useful for planning, they may be useful for improving programs or trying to understand short-term trends. But the quality may not be good enough for impact evaluation. For an impact evaluation, where you are trying to determine whether a program had an effect on health status had an impact, and you're trying to usually use that to determine whether to continue the program or to scale it up or to replicate it in another setting, you really want to be hanging your hat on really good quality data, because otherwise you risk drawing conclusions that are potentially incorrect. Maybe you determine the program doesn't have an impact when it does or vice versa. For program evaluation purposes, you typically want to use data that you are very confident in the accuracy of. That is not always the case with routine data, although it can't be so important to assess if you have routine data you're interested in potentially using, you want to assess the quality of that data and then consider for what purpose you want to use it, whether it's for an impact evaluation or whether it's for shorter term planning. Just to say a word about DHIS-2 which I mentioned earlier, we now have over 60 countries that are using DHIS-2, to manage their health information system. Using DHIS-2 is not a panacea, so it's not going to magically improve the quality of routine data. You still have problems with lack of standardization. You still have problems with the quality of the inputs into the system. If there's issues with the registers, if people aren't being recorded when they're coming for care, you still are going to have a problem with your numerators collected using DHIS-2. Here's an example of routine coverage data used for monitoring. This is from Tanzania, where the Ministry of Health uses an RMNCAH scorecard. This particular scorecard is from the last quarter of 2018 and so you can see there are three columns. Each column shows two indicators and then there's three rows, one or three rows that we're showing here. One for all of Tanzania, one for Arusha region, and one for the Dar region. In each cell, the top half of the top triangle, if you will, the top half of the cell, it represents one coverage indicators at the bottom half of the cell represents another. It's color-coded depending on whether the country, sort of the regions target has been achieved for that particular indicator and then you can see little triangles pointing up or down to show whether there's been an increase or not from the previous period. This is produced every quarter and it provides very timely information to the ministry and other stakeholders to understand and how they're doing and where sort of more emphasis, more effort is needed. This is from some work that we did in Mali comparing routine and population-based coverage data. This is for DPT3, so coverage about least three doses, DPT vaccine or for later years at least three doses of pentavalent vaccine. The solid lines represent the DHS based estimates of immunization coverage and those dotted lines represent the estimates at immunization coverage based on the routine health information system. Each line here is a district, the districts are color-coded by which region they're in. Just a couple of things I want to point out here. One is that as I mentioned before, you can see sort of the levels of immunization coverage from the routine data. Those dotted lines tend to be in most cases higher than the survey-based test units. Also in many cases they're over a 100 percent, which is common for me for routine immunization coverage estimates, but the other thing I want to point out is that you see in the routine data a lot of year-to-year variability. These lines are going all over the place. Whereas for the DHS data, we only have three data points. There were surveys conducted in 2001, 2006, 2012, and we'd sort of don't know what happened in between those surveys. We've interpolated, a line in between those data points, but we actually don't know what happened between 2006 and 2012, so the routine data may be better capturing that variability between those time points. I think this slide is a good illustration of the advantages and disadvantages of both of these sources of data. Just to sum up again, the strengths and limitations of routine data for obtaining coverage estimates. A major strength is that routine data are collected continuously, they are available in a much more timely manner than household survey data. They're often available at a much lower level at district or even health facility level, and they're much less timely, much less costly and time-consuming to collect than surveys. They do have a number of limitations though, so it's difficult to obtain accurate denominators. The quality of the data and the completeness of the data may be poor. It only captures those seeking services, usually from the public sector. It's not going to capture all of your indicators of interest. It may be difficult to access the data in some countries, it's up to the Ministry of Health who they will allow access to the data and it's subject to information error, there may be error in the information recorded. Let's look at the strengths and limitations of household survey data. The strengths are, it provides population-based data. It looks at the entire population and tells you what is the level of my coverage indicator in the whole population. This surveys are generally nationally representative, at least the big ones like the DHS and the mix, you can get national estimates for the overall population and it captures data or information that's not collected in routine data, including things like behaviors. For example, exclusive breastfeeding or other feeding behaviors. Use of bednets that are not things that are delivered at health facilities and are typically are not things that are collected through the routine health information system. The limitations of survey data that it requires very rigorous methods to implement well to get good quality data. As we've talked about it a lot, it's expensive, it's time consuming, it is still subject to information error. So we talked about there are some things that respondents may not be able to report on well, so you can't collect everything through a household survey. It may have limited country buy-in and support, so routine data are owned by the country. Survey data, depending on who was doing the survey, the country may or may not agree with the conclusions or especially if they weren't involved in the planning and implementation of the survey. And surveys are often done every three to five years. They're not done quarterly, they're usually not even done yearly. And so you don't have this very timely data that you get from routine data. It's really representing a single point in time or a single period in time. A final note about Demographic and Health Surveillance System data, which is another potential source of coverage data. DHSS is a surveillance platform that is implemented in a small area of the country, and that continuously monitors demographic indicators and some health and indicators in the whole population in that area. DHSS's are typically run by research institutions that may be affiliated with the Ministry of Health or by academic institutions and because they do ongoing surveillance with quarterly or by yearly rounds of data collection, they have up-to-date information on that whole population with lots of data points so they can support longitudinal analyses, the information is typically available in a fairly timely way. The downside is that it's not representative at the national level, it's really representative only at this very small area of the country. The data may not always be readily available, so it depends on who collects and owns the data. Some of the data owners may be protective of sharing that data. And a criticism of DHSS sites is that over time they can become dissimilar to surrounding areas because they're often used as sites for intervention trials or other types of data collection activities or studies that may change the population. May give them more access to particular health interventions for a particular health care, or may result in changes in their attitudes towards health services. So for more information on DHSS sites, the in-depth network is a good resource. But in general, unless you're interested in coverage only in the little area covered by the DHSS site. This is generally not a good source of coverage data for evaluations of health programs or for other purposes, again, except in the area in which these data are collected. Then other sources of coverage data. So we've mentioned lots of different population-based surveys, the DHS, the mix, the smarts PMA, I want to mention that there are also other country-specific surveys. So there are lots of one-off surveys that are done by the Ministry of Health, by the National Institute of Statistics, by various academic institutions. These are quite variable in terms of their quality and their contents. The access to these surveys tends to not be straightforward. It depends on who owns the data and how willing they are to share. The National Institute of Statistics for National Institute of Health is generally a good resource in a country if you want to understand what surveys have been done there besides the DHS and the mix, they usually are aware of the major household surveys that have been done recently and may be able to tell you more about those. This slide is just a summary of the sources of coverage data that we've talked about. This is not completely exhaustive. There may be a few sources that are not covered in here, but this is most of them. So when you are in a country evaluating a program or for another reason thinking about doing a coverage survey, these are the sources of data that you would want to think about when you assess the availability of existing data and whether it's really necessary for you to do your own coverage survey. And just again, as we close out a reminder that measuring intervention coverage requires population-based measurement. So measures of intervention coverage, you want to understand what the coverage of the intervention is in the whole population. And so population-based ways of collecting that data are the preferred method. But we have talked about other sources of data that may be useful in different circumstances.
