Experimental Effects of Advance Postcards, Survey Title, Questionnaire Length, and Questionnaire Content on Response Rates and Incentive Costs in a Mail Non-Response Follow-Up Survey

Introduction and Research Questions

Non-response follow-up (NRFU) studies are used to gather data on sample units that fail to respond to a survey. Unit non-response bias is then estimated by comparing the original survey’s respondents to non-respondents who respond to the NRFU. Minimizing non-response is a top priority for NRFUs, since their purpose is to obtain responses from people who previously did not respond.

Several aspects of NRFU study design are important for promoting response. The mode of data collection should differ from the main study to avoid correlated non-response, which makes paper mail-in questionnaires a fitting NRFU mode for studies originally conducted online, by telephone, or in person. For the same reason, the title or topic and sponsorship of the study as presented to participants may be differentiated from the main study. Meta-analysis of many studies indicates that a survey’s topic can be particularly impactful if respondents find it interesting (response odds ratio 2.4; Edwards et al. 2002), but this may be harmful because what is interesting to one sample subgroup may discourage another from participating. The same meta-analytic evidence also indicates that advance letters should be used to announce the study because they have a substantial effect on response (odds ratio 1.5), as does a short questionnaire (odds ratio 1.9). To further quantify the tradeoff between questionnaire length and response rate, more than 40 years ago it was estimated that each additional question reduced the response rate to mail surveys by half a percentage point, and each page reduced the response rate by 5 points (Heberlein and Baumgartner 1978), though obviously this relationship must be bounded. Somewhat more recently a systematic review of 292 randomized controlled trials estimated that the odds of response for a 3-page questionnaire were half the odds for a 1-page questionnaire (Edwards et al. 2002).

These are powerful effects. Yet these lessons do not always translate as directly into practice as we might wish, for two connected reasons. First, the effects of particular design elements may vary across contexts, creating uncertainty about the benefits of a particular design choice for a particular study. Indeed, meta-analysis shows considerable heterogeneity in the effects of advance letters (De Leeuw et al. 2007) and other design elements (Edwards et al. 2009). Second, study goals such as a high response rate and a long questionnaire are in direct tension. Using a 1-page questionnaire constitutes such an enormous opportunity cost that researchers may reasonably ask whether the effect would be as large in the specific application to their study as it appears to be more generally.

The motivation for the current study was to test the effects of advance mailings, questionnaire length, and study topic (in study title and questionnaire content) on the response rate to a mail survey, in the context of a non-response follow-up to the American National Election Studies (ANES) Time Series Study. The research questions (RQ) were as follows:

RQ1: Questionnaire content. The ANES is a study of adult U.S. citizens. Its questionnaire content is heavily political. For this study’s purposes, non-response correlated with political behavior, opinions, or attitudes is a serious concern. For the NRFU, this raises the question of how strongly political questionnaire content is associated with non-response and whether more ANES non-respondents can be included by using a non-political NRFU questionnaire.

RQ2: Questionnaire length. The ANES NRFU, unlike some NRFUs, includes both ANES respondents and ANES non-respondents, raising the question of whether effects of study design on response rates differ for ANES respondents (whose NRFU response propensity is very high) and ANES non-respondents (whose NRFU response propensity is relatively low). It is the lower-propensity non-respondents who are critical to reach with the NRFU. Thus, though we anticipate that questionnaire length will be inversely related to response rate, we ask if this relationship is different for ANES respondents and non-respondents.

RQ3: Study title. What is the effect of alternate titles on the response rate? Topics of interest promote response, suggesting that a study title about matters that are timely, such as Covid-19, or important to most people, such as families, could be higher than a study that is more generic, such as a National Study of Households.

RQ4: Advance mailing. Does an advance postcard increase the response rate to a brief mail study, for prior respondents or non-respondents? Advance letters promote the study’s legitimacy, which is beneficial for many studies, but is it still beneficial for a study that already uses numerous design elements known to promote response: a low-burden study (short questionnaire) with a pre-paid monetary incentive, multiple follow-up mailings, questionnaire re-sent after non-response, sent by first-class mail, and sponsored by a university?

RQ5: Costs. Do these design elements affect the financial cost of administering the study? The ANES NRFU used a $5 prepaid incentive and, after initial non-response, made a $20 post-paid incentive offer. Thus, promoting initial response has the potential to save money by reducing the number of participants who receive the escalated incentive.

Methods and Data

Data come from the ANES 2020 Non-Response Follow-Up Study (NRFU; American National Election Studies 2021). The study was conducted by mail between January 28 and June 1, 2021. The sample included both respondents and non-respondents to the ANES 2020 Time Series Study (ANES; American National Election Studies 2020). The ANES was a survey of adult U.S. citizens selected using address-based sampling, interviewed using a mail push-to-web design. The ANES selected one adult U.S. citizen per sampled household after a brief initial screening questionnaire, and the minimum overall response rate was 37 percent. The NRFU sample consisted of 4,000 individuals who responded to the ANES, 1173 individuals whose household was screened and who were selected for ANES but did not respond, and 2827 households that were selected for ANES but did not respond at the screening stage (total 8,000). The individuals who completed or who were screened for ANES were invited to NRFU by name; at the non-responding households, invitations used the Hagan and Collier (1983) method, randomly requesting a response from the oldest or youngest male or female. The NRFU study’s nominal sponsor was Duke University. A prepaid $5 incentive was enclosed, and up to 6 invitation or reminder mailings were sent. The last two mailings offered a $20 postpaid incentive. There were 3,779 responses to the NRFU, for an unweighted response rate of 47 percent overall and a weighted response rate of 57 percent overall, 83 percent among ANES respondents, and 34 percent among ANES non-respondents. The weighted NRFU data represent the full ANES sample.^[1]

Independently randomized methodological experiments were integrated in the NRFU study to test questionnaire content, length, title, and advance mailing effects. Half the sample (n=4,001) was sent an advance postcard shortly before the initial questionnaire invitation, while half (n=3,999) omitted this mailing. Half the sample (n=4,001—again, randomized independently) was invited to the National Study of Households while half (n=3,999) were invited to the National Study of Households, Families & Covid-19. Questionnaires were randomized for length and content into four types (each n = 1,943 to 2,056): a one-page non-political questionnaire, a one-page questionnaire with political questions, a 2-page questionnaire with political questions on the second page, and a two-page questionnaire with political questions on the first page. The questionnaires are available on the ANES website at www.electionstudies.org.

Results

The advance postcard had no discernable effect on the response rate for the NRFU overall or among ANES respondents or non-respondents. These results are shown in Table 1, where the response rates to NRFU were about 57 percent overall, 83 percent for ANES respondents, and 34 percent for ANES non-respondents, regardless of the advance mailing condition; differences for the mailing conditions were 0.2 percentage points or less.

The survey title experiment (see Table 2) found that the shorter, simpler title—National Study of Households—yielded a higher response rate than the longer title, but no differences reached statistical significance. Overall, the response rate was 2.2 points higher with the short title (p = .080) and 3.2 points higher for ANES non-respondents (p = .095).

Questionnaire length showed large effects overall and for ANES non-respondents, as shown in Table 3. Overall, the response rate was 3.7 points higher for the one-page questionnaire than the two-page questionnaire (p=.003) and 6.9 points higher for the one-page questionnaire among non-respondents (36.9 compared to 29.9 percent, p<.001). For ANES respondents, the difference of 1.5 was in the expected direction but not significant.

Questionnaire content showed no detectable effects (also Table 3). Response rate differences between the political and non-political one-page questionnaires were about 1 point or less and were nonsignificant. For the two-page questionnaire, response rate differences associated with the position of political content on the first or second page were 0.8 to 1.7 points and were nonsignificant.

There were no significant differences in total incentive payout costs. Between one- and two-page conditions, where the response rate difference reported above was significant, there was an average cost difference of three cents (not shown in tables).

Conclusions

This study found advance postcards were not effective in promoting response to the NRFU, suggesting similar studies in the future can omit this mailing. The short title had a slightly higher response rate than the one also referencing families and Covid-19, suggesting the short, plain title was better. Political content, whether by inclusion, position, or exclusion, had no discernable effect on survey response, even among ANES non-respondents, indicating that such content need not be avoided to promote response. However, questionnaire length was strongly and inversely related to response, especially for the critical sample group of prior non-respondents: for this group, the 2-page questionnaire depressed the response rate by almost 7 points compared to the 1 page questionnaire, indicating the price of more questions is high for the sample group where NRFU response is most important.