It is a well-known fact that reported turnout in post-election surveys is higher than official turnout rates. This overestimation of turnout in surveys may be due to different factors: coverage error, overrepresentation of voters (nonresponse bias) and overreporting of actual abstainers (measurement bias). Unfortunately, there is only very few data available to test how much these factors contribute to the overall turnout bias. What is needed for such an analysis is validated data, i.e. one has to know if a survey respondent correctly reported her turnout or if she lied about it. Even rarer than this validated data, though, is turnout information about non-respondents. Using a unique data-set from the canton of Geneva, Switzerland, Pascal Sciarini and me are able to examine the components of the turnout bias. The data include not only information about reported and validated turnout among survey respondents, but also the validated turnout among non-respondents. We use these data to analyse several aspects of turnout bias, which so far lead to three publications.
In a first paper we focus on vote overreporting and its link with survey participation and political participation. We argue that turnout is typically a topic with a connection between response propensity and measurement error, as factors related to political involvement (usual voting behaviour) are likely to influence both survey participation and misreport. To take this link into account, we run a Heckman selection model in which the outcome equation estimates the probability of overreporting conditional on the probability of participation in the survey. In most other data, though, no information about nonresponse is available, so that we offer a solution that may still correct for the nonresponse bias. Applying this correction to U.S. data shows that not correcting for response propensity may indeed lead to inaccurate estimates of the determinants of overreporting and, ultimately, of turnout.
Sciarini, P. and A.C. Goldberg (2016). Turnout bias in postelection surveys: Political involvement, survey participation and vote overreporting. Journal of Survey Statistics and Methodology 4(1): 110-137.
Our second paper starts with a more general analysis of the components of turnout bias. Using two different data-sets, we show that voter overrepresentation accounts for a larger share (60-65%) of total turnout bias than misreporting. This speaks against a widespread view that measurement error is more important. In a second step we then concentrate on nonresponse bias and analyse individual predictors of survey participation. We are even able to break down the nonresponse problem into its two main components – contact and cooperation – and can thus see which type of person is lost at which point of the survey process.
Sciarini, P. and A.C. Goldberg (2017). Lost on the way. Nonresponse and its influence on turnout bias in post-election surveys. International Journal of Public Opinion Research 29(2): 291-315.
In a third paper, we examine the effect of non-response follow-up (NRFU) surveys on turnout bias. We again use two datasets on validated voter turnout data, collected across two different ballots. Both studies include a short follow-up mail survey on so-called “central” questions for nonrespondents, in addition to the main CATI/web surveys. The results demonstrate that collecting extra information from initial non-respondents is worth the effort. The NRFU survey substantially increases representativeness with respect to sociodemographic and especially participation variables. This results in better estimates of turnout determinants in the final (combined) sample than is seen from CATI/web respondents only. Importantly, the increase in response rate and the decrease in nonresponse bias comes at almost no price in terms of measurement errors as vote overreporting is only slightly higher in the mail follow-up survey than in the main CATI/web survey.
Goldberg, A.C. and P. Sciarini (2019). Who gets lost, and what difference does it make? Mixed modes, nonresponse follow-up surveys and the estimation of turnout. Journal of Survey Statistics and Methodology 7(4): 520-544.