Examining the Effect of Prenotification Postcards on Online Survey Response Rate in a University Graduate Sample

Camille B. Lalasz Center for Research Design and Analysis, University of Nevada, Reno

Michael J. Doane Center for Research Design and Analysis, University of Nevada, Reno

Victoria A. Springer Center for Research Design and Analysis, University of Nevada, Reno

Veronica B. Dahir Center for Research Design and Analysis, University of Nevada, Reno

Introduction

The Internet has become an increasingly popular avenue for data collection. Online surveys have many advantages, such as cutting the costs of printing and mailing surveys and reducing time for data collection and entry (Kaplowitz et al. 2004). However, online surveys have disadvantages (Schonlau et al. 2002); most notable being the low response rates relative to other modes of data collection (Hoonakker and Carayon 2009). With growing interest in the use of online surveys (Sue and Ritter 2011), research needs to clarify ways to increase response rates for this survey mode.

The literature identifies several factors that increase response rates for other survey modes, including prenotification (Fox et al. 1988). Notifying participants of a study in advance may increase response rates by alerting individuals that they have been selected as participants and increasing the legitimacy of the study (Fox et al. 1988). One way to inform participants of a study is via mailed introductory postcards. This form of prenotification has been found to increase response rates for both mail (Whiteman et al. 2003) and telephone (Iredell et al. 2004) survey modes. It is less clear whether introductory postcards function similarly for online surveys. What research does exist is inconsistent, with some studies reporting no difference between mail and e-mail prenotification on online survey response rates (Porter and Whitcomb 2007), while other studies suggest that mail prenotification is even less effective compared to e-mail prenotification (Schaefer and Dillman 1998). Other research has focused on differences between introductory postcards and no prenotification at all, generally finding that postcards have the potential to increase response rates for online surveys (Kaplowitz et al. 2004). Overall, findings are inconclusive. Given the costs associated with mail prenotification, additional research on the effectiveness of introductory postcards for online survey response rates is needed.

The purpose of this study was to test whether using an introductory postcard increased the response rate for an online survey among recent alumni from the University of Nevada, Reno (UNR). The Center for Research Design and Analysis (CRDA) at UNR conducts an annual survey of alumni one year after graduation to document graduates’ employment and/or educational standings and to assess how well major programs prepare graduates for life after college. Degree programs, departments, and colleges at UNR use alumni feedback to determine how well educational goals are being met and determine if changes should be made. For this study, one group of randomly-assigned alumni was sent an introductory prenotification postcard, while the second group did not receive a prenotification.

Method

Participants

A total of 1,358 individuals graduated from UNR in May 2009 and were eligible to participate in the survey conducted in spring 2010. UNR provided the following personal information for each graduate: name, degree, mailing address, and e-mail addresses. (Alumni had between one and five e-mail addresses on file.) The university did not have a mailing address on file for six alumni, bringing the total number of eligible participants to 1,352.

Procedure

Potential participants were randomly divided into two groups. Group 1 (n=673, 49.6%) received a high contrast black and white introductory postcard which briefly described the survey and informed participants that they would be sent an e-mail invitation within the next two weeks. The postcard (see Figure 1) was designed based on best practices (Dillman 2000; Nederhof 1988), including the use of a dominantly black cover. Group 2 (n=679, 50.2%) did not receive an introductory postcard.

Two weeks after the postcards were sent out to group 1, an initial e-mail invitation was sent out to all potential participants. This e-mail informed potential participants of the method (i.e., brief, online survey); focus (i.e., postgraduate career, educational achievements, and assessment of their UNR education); important purpose (i.e., used by departments to evaluate their effectiveness and improve existing programs and to address legislative, Board of Regents, and accreditation interests); and confidential nature of the survey. Potential participants also were given a web link to view the survey instrument and past responses, a link to a FAQ page, and contact information for the lead investigator.

Figure 1 Prenotification postcard used for university graduate sample.


SP-Vol 7_Jun_Lalasz_fig1.jpg


Up to four reminder invitations were sent out every two weeks to those individuals who had yet to participate in the survey. These invitations contained the same information as the initial invitation but were worded differently in accordance with best survey practices (Dillman et al. 2009). Invitations were sent out to each one of five possible e-mail addresses associated with each potential participant.

Results

The purpose of this study was to determine whether postcards increased response rates for an online survey of university alumni contacted 1 year after graduation. Prior to exploring this question, preliminary analyses were conducted.

Preliminary Analyses

Thirty-six of the 673 introductory postcards sent to group 1 members were returned by the U. S. Postal service due to incorrect addresses. Because these individuals did not receive prenotification of the survey as intended, they were removed from all subsequent analyses, and group 1 was reduced to 637 potential participants.

We postulated that the likelihood that a potential participant would be successfully contacted via e-mail invitation may increase as the number of e-mail addresses for the individual increased as multiple e-mail addresses should improve the odds that at least one is active. Consequently, a t-test was conducted to determine if there was a mean difference in the number of e-mails associated with participants who received introductory postcards (group 1) and those who did not (group 2). Results indicated that there was not a significant effect by group, t(1314)=–1.83, p>0.10, with group 1 (M=2.01) and group 2 (M=1.92) showing no difference in mean number of e-mail addresses associated with group members. To explore the equivalence of groups, we also tested for differences in degree type (i.e., undergraduate, graduate) and state of residence (i.e., Nevada, non-Nevada). Chi-square analyses indicated that there was not a significant difference in degree type, χ2 (1, N=1,316)=0.01, p>0.10, or state of residence, χ2 (1, N=1,316)=0.98, p>0.10, between group members.

Analysis

Just over 37% (489 out of 1,316) of alumni participated in the survey. Of the 637 potential participants who were sent a deliverable introductory postcard, 242 (37.99%) participated and 419 (62.01%) did not. Of the 679 potential participants who were not sent an introductory postcard, 247 (36.38%) participated in the survey and 432 (63.62%) did not. A chi-square analysis was conducted to determine if response rates varied by group. The relationship between these variables was not significant, X2 (1, N=136)=0.366, p>0.10, indicating that those who were sent an introductory postcard were not more likely to participate in the survey.

Discussion

This study suggests that postcard prenotification was not fruitful for increasing the online response rate in this population for the specific purpose of this survey. Though it is often impossible to know the characteristics of a given population before a survey is administered, there were no explanatory differences that emerged between participants who did and who did not receive a postcard prenotification, such as multiple e-mail addresses, degree type, or state of residence. Given the equivalence of the participants who received introductory postcards and those who did not, it is appropriate to conclude that the use of the prenotification postcards did not result in a significant increase in the response rate for our online survey. This finding is important as it contrasts other studies that report an increase in response rates for online surveys with mailed prenotification (e.g., Kaplowitz et al. 2004).

These null findings warrant a further discussion of the population of this study. The respondents to this online survey were university alumni. Thus, the mail addresses that were used for the postcard prenotification may not have been the most accurate addresses for some potential participants. For instance, the mail addresses provided to the university by the former students at the time their contact information was collected just weeks before graduation, may have been their parents’ address (a more permanent address) rather than their current UNR address or future address. For this reason, our population for this specific study is unique, and the fact that our results contrast with other studies about mailed advanced notification (e.g., Kaplowitz et al. 2004) is not that surprising. When surveying more transient populations (e.g., recent college graduates), it seems that an e-mail advanced letter might be more productive and cost effective.

We suggest that future research should explore this and other factors that may disentangle this divergent finding. For example, the effectiveness of mailed prenotification may depend on the accuracy of the population’s mail and/or e-mail contact information. Specifically, mailed prenotification may increase response rates for populations with less reliable e-mail addresses by ensuring that individuals are notified of the study (Dykema et al. 2013) but be ineffective for populations with accurate e-mail addresses. Individual differences may also correspond to increased likelihoods of responding when mail prenotification is used. For instance, potential participants who conduct the majority of their correspondence through electronic media may not be responsive to prenotification through the U. S. Postal service. Future research should also delve further into the effect of prenotification content. The potential gain related to mail prenotification may be related to the perceived legitimacy of the mailing itself. Accordingly, a full page letter on university or other official letterhead may increase online response rates whereas informational postcards may not. Further, an appropriately designed “teaser” may drive interest while a bland or information-dense note may stifle participant engagement. Pilot testing of prenotification design and appeal with nonsurveyed members of the specific target population would provide valuable insight into this strategy. Given the increasing popularity of online surveys and the costs associated with postal mail prenotification, it is suggested that further research be conducted to better understand the circumstances under which mail prenotification increases response rates as well as further exploration into when and with whom an e-mail advanced notification would be more effective.

Author Note

The authors wish to thank Dr. Kerry Kleyman for her assistance in creating the prenotification postcard used for this study.

References

Dillman 2000
Dillman, D.A. 2000. Mail and internet surveys: the tailored design method (2nd ed.). John Wiley & Sons, Inc., New York.
Dillman et al. 2009
Dillman, D.A., J.D. Smith and L.M. Christian. 2009. Internet, mail, and mixed-mode surveys: the tailored design method. John Wiley & Sons, Inc., Hoboken, NJ.
Dykema et al. 2013
Dykema, J., J. Stevenson, L. Klein, Y. Kim and B. Day. 2013. Effects of e-mailed versus mailed invitations and incentives on response rates, data quality, and costs in a web survey of university faculty. Social Science Computer Review 31(3): 359–370.
Fox et al. 1988
Fox, R.J., M.R. Crask and J. Kim. 1988. Mail survey response rate: a meta-analysis of selected techniques for inducing response. Public Opinion Quarterly 52(4): 467–491.
Hoonakker and Carayon 2009
Hoonakker, P. and P. Carayon. 2009. Questionnaire survey nonresponse: a comparison of postal mail and Internet surveys. International Journal of Human-Computer Interaction 25(5): 348–373.
Iredell et al. 2004
Iredell, H., T. Shaw, P. Howat, R. James and J. Granich. 2004. Introductory postcards: do they increase response rate in a telephone survey of older persons? Health Education Research 19(2): 159–164.
Kaplowitz et al. 2004
Kaplowitz, M.D., T.D. Hadlock and R. Levine. 2004. A comparison of web and mail survey response rates. Public Opinion Quarterly 68(1): 94–101.
Nederhof 1988
Nederhof, A.J. 1988. Effects of a final telephone reminder and questionnaire cover design in mail surveys. Social Science Research 17(4): 353–361.
Porter and Whitcomb 2007
Porter, S.R. and M.E. Whitcomb. 2007. Mixed-mode contacts in web surveys: paper is not necessarily better. Public Opinion Quarterly 71(4): 635–648.
Schaefer and Dillman 1998
Schaefer, D.R. and D.A. Dillman. 1998. Development of a standard e-mail methodology. Public Opinion Quarterly 62(3): 378–397.
Schonlau et al. 2002
Schonlau, M., R.D. Fricker, Jr. and M.N. Elliott. 2002. Conducting research surveys via e-mail and the web. RAND, Santa Monica, CA.
Sue and Ritter 2011
Sue, V.M. and L.A. Ritter. 2011. Conducting online surveys. Sage Publications, Thousand Oaks, CA.
Whiteman et al. 2003
Whiteman, M.K., P. Langenberg, K. Kjerulff, R. McCarter and J.A. Flaws. 2003. A randomized trial of incentives to improve response rates to a mailed women’s health questionnaire. Journal of Women’s Health 12(8): 821–828.


About Survey Practice Our Global Partners Disclaimer
The Survey Practice content may not be distributed, used, adapted, reproduced, translated or copied for any commercial purpose in any form without prior permission of the publisher. Any use of this e-journal in whole or in part, must include the customary bibliographic citation and its URL.