The Mode of Invitation for Web Surveys

E-mail is a common invitation mode for Web surveys. This invitation mode is cheap, is easy to automate and personalize, and provides easy access to the survey (a clickable URL). However, there are a growing number of problems with e-mail as an invitation mode, including concerns about spam, the rapid turnover (churn) of e-mail addresses, the fact that many sample frames do not contain e-mail addresses, privacy concerns about providing e-mail addresses when solicited, and the inability to deliver a prepaid incentive.

The use of e-mail invitations in Web surveys may be one factor in the relatively low response rates compared to mail surveys (Lozar Manfreda et al. 2008; Shih and Fan 2008). One often-employed strategy to counteract this nonresponse problem is the use of pre-notification. In mail surveys, research has consistently shown that pre-notification in the same mode (i.e., advance letters) tend to increase response rates (see Couper 2008; Dillman et al. 2009). However, research exploring pre-notification for Web surveys suggests that pre-notification in another mode may be more effective (see Bosnjak et al. 2008; Kaplowitz et al. 2004; in contrast see Porter and Whitcomb 2007). Using pre-notification in another mode like a postcard or mailed letter may attract greater attention and more legitimacy to the subsequent e-mail invitation (see Bosnjak et al. 2008).

The main purpose of the study presented here was to test whether an invitation in another mode (e.g., by a mailed letter) could be an alternative to the common e-mail invitation in a Web survey. We expected that a mailed invitation letter may have greater legitimacy than an e-mail invitation. However, there is a media break, i.e., respondents have to type a URL and a password. In addition to this variation of invitation mode we tested the effect of a mailed letter as pre-notification in our Web survey. In contrast with most studies that use student samples, we used a random sample of the general population. In such student surveys, a relationship already often exists with the sender, potentially making e-mail more effective. In our study, no prior relationship existed.

Participants were recruited with the aid of the German General Social Survey (ALLBUS). This survey was conducted between spring and summer 2010. The ALLBUS is a biennial, representative survey of the German general population that has been in place since 1980, and covers topics such as attitudes, behavior, and social structure of residents in Germany. As a data collection mode, ALLBUS uses computer-assisted personal interviews (CAPI) in private households. In 2010, the final ALLBUS sample encompassed 2,827 persons overall. The sample was drawn by the community offices in Germany that register residents so the mail addresses of all respondents are known. The response rate was about 35 percent (see http://www.gesis.org/en/allbus/study-profiles/2010/ for further information).

At the end of the ALLBUS interview all respondents were asked whether they use the Internet. There were 1,869 Internet users (66 percent of respondents). All Internet users were then asked whether they were willing to participate in a scientific follow-up survey conducted by GESIS Leibniz Institute for the Social Sciences^[1]. Among those asked, 1,492 persons (80 percent of the 1,869) expressed willingness to participate. In a following question, this group was asked whether they would provide their e-mail address. Altogether 667 persons provided their e-mail address (45 percent of the 1,492). Compared to those persons who did not provide their e-mail address (n = 825) these 667 persons are higher educated (51 vs. 38 percent, χ² (2) = 26.93, p < 0.001) and are less likely to be female (43 vs. 53 percent, χ² (1) = 15.68, p < 0.001). These people are also more likely to agree with the statement “most people can be trusted” (26 vs. 21 percent, χ² (1) = 4.35, p = 0.004).

All 667 persons who provided their e-mail address were then randomly assigned to one of four groups (see Figure 1). The first two groups were sent a mailed letter as pre-notification and the invitation mode was either a mailed letter (first group) or an e-mail (second group). In the third and fourth groups there was the same differentiation in the invitation mode but there was no pre-notification. This design allows us to test the effect of a mailed pre-notification and the effect of mail versus e-mail for the survey invitation.

Figure 1  Overall process and experimental design.

All invited persons were directed to an identical online questionnaire with a survey length of about 15 minutes. The survey was fielded in April 2011 with a time delay of between 158 and 311 days from the initial ALLBUS interview to the invitation to the online follow-up survey. Only one reminder was sent to all nonrespondents. The mode of the reminder was the same as the invitation mode.

Results

The response rates for the different groups varied between 40 percent and 57 percent (see Table 1).

A mailed letter, whether as pre-notice or invitation, is more effective than e-mail alone (51 percent vs. 40 percent, χ² (1) = 3.71, n.s., χ² (1) = 3.93, p = 0.047). However, the effect of the pre-notification occurs only in the e-mail invitation group (57 vs. 40 percent, χ² (1) = 8.55, p = 0.003); in the mail invitation mode there is no effect of pre-notification (51 percent in each group). The interaction between pre-notification and invitation mode is statistically significant (see Table 2) when controlling for covariates of willingness to provide an e-mail address.

Discussion

As discussed at the outset, there are pros and cons to using e-mail for invitations to online surveys. Asking for e-mail addresses is a sensitive question; less than half of those asked in the ALLBUS interview provided their e-mail. Those who do so are more trusting than those who do not. We find that a mailed (paper) letter, whether as a pre-notice or an invitation, is more effective than e-mail alone.

One limitation of this study is that this experiment is conditional on stated willingness to participate in a follow-up survey and conditional on providing an e-mail address. We did not include those who did not provide an e-mail address. In the 2012 ALLBUS we plan to test the effectiveness of not asking for an e-mail address and then sending a mailed invitation to an online survey. But our initial findings suggest that mail can be just as effective as e-mail for online survey invitations.