[BLANK_AUDIO] I promised to report on a study that we did on motivated underreporting. This is a project that I did together with Roger Tourangeau and Stephanie Eckman. Roger Tourangeau and I, we got a grant from NSF to study misreporting in surveys. In general, I'm not going to report all of these results, but our research question has three related forms of measurement error that we are interested in: screener questions, filter questions, and repeated questions in panel surveys. For each of these types we do see evidence of motivated misreporting. The mechanisms, we think, behind the scenes are that respondents may answer these questions in ways that reduce the burden of the interview. So, in order to have to do the interview they make themselves ineligible in screening interviews. Think of respondents who are shy and don't want to offend the interviewer and want to decline the participation in a survey. They could simply say, "Oh no, I'm not part of the age category", or, "We don't have a small child", if that is what is looked for. Likewise, we talked about filter questions before, they could avoid triggering filter questions in these formats where they see that each filter question is followed by a series of followup questions. We talked about that in prior sections of this class. Now, of course you could argue, this can also reflect interviewer motivation to reduce the burden. Interviewers are judged and evaluated by their response rates, and so it might be an incentive for interviewers to suggest to the respondent that, "Well, just answer a few questions, you might not be eligible to do this survey", and therefore avoid nonresponse in suggesting, implicitly, to the respondent that they make them self ineligible. To study this, as I mentioned, we got funding from the National Science Foundation, the grant numbers are listed here. We funded from these grants two telephone surveys that were done in the U.S. The first one had a target population of 35 to 55 years of age. We had about 2,500 households screened, and within those, we had 958 eligibles. Inspect the sample a bit. We had voting records and, on purpose, selected cases that could fall into this age range at a higher rate. And then we had about 750 completed interviews, and within the survey three experimental manipulations. I'll come to those in a second. The second study was a face to face survey done more recently. Our attempt here was to replicate the findings we found on the phone. You have to read those papers if you want to see the results, I'm only going to report on the phone survey here. So the experimental manipulation was a manipulation of screening wording. The first experimental condition was, "Is anyone in the age range 35 to 55?" The second one, so imagine all of the screened households were randomly divided into one of these three conditions, so one third of the cases got that direct question. The next third got the compliment, which was all house of members less than 35, the question to the household, or over 55, so screening out if you will. And then we had a full household roster in the control condition which would usually be done, and, in addition, we ask for age, sex and race and ethnicity of all household members. The survey got an advanced letter that revealed the target population or hid the target population and this experimental condition was crossed with the other. They were also an interviewer manipulation, again crossed with the other conditions, and we wanted to test the interviewer contribution to this undercoverage phenomenon. And in this interviewer manipulation we varied the payment scheme. Some interviewers got a bonus for completed screener interviews, $2 in size. Others got a bonus for the completed interview. So they would have had an incentive, monetary incentive for respondents to actually select into the survey rather than just the incentive to finish the screener and be done with the survey. And the last condition was no bonus at all. We had twelve interviewers who worked in each of these conditions. So here are the results. They are separated by the three types question in the screening wording, the letter, whether they reveal the age range of the target population or not and the bonus scheme. In the columns you see the screener completion rate, the eligibility rate, the interview completion rate, the response rate, and the actual yield, so how many cases do we in the end have to analyze. And let me start with the yield. So what is interesting is that the roster, the method that's usually done, has the highest yield. So you end up with the highest number of cases. However, it has the lowest response rate. And that is, because you have higher eligibility. And even so there is the higher eligibility you find the cases that you need but you have lower completion rates. More people decide to not be respondent actively, but because you start out high, you still have a higher yield. Another way to say this is, you move coverage error or undercoverage error into a nonresponse error, which in a way, is a fair thing, because that is a statistic we often report in surveys. The other two experimental conditions you can see here. We have higher rates of completed screeners, which makes sense. These conditions are much easier. It's a quicker question to do, where as the whole household roster people might not finish the entire screening. But, it's easier to hide, "No, no. I'm not in that age group" or "Everyone is between these two age ranges", which would be this condition here. But then of those that did say they are eligible, a higher rate completed. Now, it is also interesting that we see effects of the advanced letter on the completion rate, though not on the eligibility rate. And finally, on the interviewer payment scheme, we didn't see big effects of payment on the overall response rate and the overall eligibility rate, but a slight effect on the screener completion. So interviewers that got additional money to complete the screener, completed screening at higher rates. An interviewer that got a bonus for completing the interview, completed interviews at a higher rate. There was a lot of variance across the interviewers. So, there's a strong interviewer effect on how they react to these bonus schemes and how they perform the task overall, which was much larger than any other variance component in here. So there's a trade-off with the screening questions. We have, you know, nonresponse and measurement error and undercoverage that you need to consider here when you have this screener question in place. And likewise, you have a trade-off between the accuracy of the information, a burden, and a perceived threat on the side of the respondent. If you want to read more about these questions, the references are out here and on the Coursera website. You will see the actual listings of the papers and can download them from there.
