An Incentive Experiment Designed to Increase Response to a Between-Wave Contact Update Mailing in Two Panel Studies

Katherine A. McGonagle, Robert F. Schoeni, Mick P. Couper and Mohammad Mushtaq University of Michigan

Overview

Since 1969, the Panel Study of Income Dynamics (PSID) has sent its families a “contact update” mailing between waves of data collection in order to keep track of the whereabouts of the families and minimize attrition. Having updated contact information became all the more important starting in 1997 when PSID changed from annual to biennial interviewing. All things being equal, the longer the time between data collection waves, the greater the likelihood that sample persons have moved, and the greater the difficulty in locating movers (Couper and Ofstedal 2007; Duncan and Kalton 1987). Families are now sent the mailing the year before production interviewing, and those that update or verify their address and telephone information receive a $10 postpaid check. Field effort for the families that responded to this mailing in recent waves is much lower in the form of reduced tracking, refusal conversion efforts, and number of calls to finalize the case, underscoring the cost effectiveness of the mailing.

Given these advantages, in 2008 we designed a study to improve the response rate of the contact update mailing (McGonagle et al. 2009, in press). A main finding of that study was that response rates were substantially higher in a subgroup of nonresponding families that was sent a second mailing. The study did not provide evidence with regards to the importance of an incentive or optimal amounts for such an incentive.

This paper reports on the initial results of a second study designed in 2010 in advance of 2011 production interviewing to examine whether response rates are affected by whether an incentive is provided, and to test the relative effectiveness of different incentive amounts. Because PSID is conducted by telephone, a second outcome of interest is whether a new telephone number was provided. This study includes families that participate in the PSID, and for the first time, young adults aged 18 and older who participate in the PSID – Transition to Adulthood study (TA). TA is a panel study that follows individuals who were first interviewed as children in the PSID Child Development Supplement once they turn age 18 and collects a variety of social and economic content.

Background

A substantial literature exists on the benefits of providing incentives in exchange for participation in surveys (e.g., see Laurie and Lynn 2009; Laurie et al. 1999; Singer et al. 1999), with research based on longitudinal studies generally finding evidence of a positive association between incentive amount and response rate (Fumagalli et al. 2010; Martin et al. 2001; Rodgers 2002). Moreover, studies find that responsivity to incentive amounts differs by sociodemographic characteristics of sample members (Mack et al. 1998; Martin et al. 2001), and that the positive effects of incentives at one wave may persist over time, reducing cumulative nonresponse over multiple waves (Mack et al. 1998; Scherpenzeel et al. 2002).

There is limited experimental research on alternative between-wave contact strategies. Fumagalli and colleagues (2010) found a positive association between incentive amount and return of an address change card. In our 2008 study, we found no effect of a prepaid versus postpaid $10 incentive on return of the address update card, but we did find a positive effect of a second mailing sent to a subgroup of families who did not return the address card in response to the first mailing. The current study is designed to examine the utility of a four decade long practice in the PSID of providing incentives for a between-wave contact update mailing by varying the provision of an incentive and the amount in two samples of respondents.

Sample members within each study were randomly assigned to receive a postpaid incentive of either $0, $10, or $20 in exchange for returning the contact update postcard with a verification or update of their address and/or telephone number. This paper reports on initial findings with regard to response rates to the mailing and provision of new telephone numbers by these different conditions for each study. Hypotheses for the findings and next steps for analysis are described.

Methods

Conditions

Sample members eligible for the 2011 PSID and TA interviews (n=8444, n=1678, respectively) were randomly assigned to one of four experimental conditions: 10% received no contact update mailing; 10% received a mailing but no incentive; 40% received a mailing and a $10 postpaid incentive (which is the PSID status quo amount) and 40% received a mailing and a $20 postpaid incentive. Production outcomes for families in the condition that did not receive the mailing will be reported in a future paper once 2011 interviewing has concluded. Families in the three mailing conditions were sent the initial contact update mailing in August 2010, approximately seven months before the start of 2011 production interviewing. Those not responding within two months were remailed the materials in October 2010.

Results

Table 1 shows that 67.7 percent of respondents in PSID and 59.6 percent in TA returned the contact update postcard. The $10 and $20 conditions yielded significantly greater postcard returns than the $0 condition in both studies. This effect was very large in the TA study, with return rates of 23–25 percentage points higher in the $10 and $20 vs. $0 conditions. The $20 condition produced significantly more postcard returns than the $10 condition in the PSID only.

Table 1 Percentage returning card and providing new numbers

Total $0 incentive $10 incentive $20 incentive
PSID
% Returning card 67.7 58.9 67.1a,b 70.7a,b
• First mail 54.5 42.3 54.0 58.4
• Remail among Nonresponders 29.0 28.8 28.5 29.5
% Providing new phone # 12.5 9.9 12.7a 13.1a
• First mail 9.3 6.4 9.2 10.1
• Remail among Nonresponders 7.2 5.8 7.7 7.1
TA
% Returning card 59.6 38.8 61.5a 63.5a
• First mail 44.8 29.0 46.5 47.6
• Remail among Nonresponders 26.8 13.8 28.1 30.4
% Providing new phone # 16.8 10.9 16.7c 18.5c
• First mail 11.4 7.1 11.7 12.4
• Remail among Nonresponders 9.6 5.4 9.4 11.6

a10 and $20 are significantly different from $0 at p<0.01; b$10 and $20 are significantly different from each other at p<0.01; c$10 and $20 are significantly different from $0 at p<0.05.

The remail was associated with a substantial number of postcard returns in both studies, with 29.0 and 26.8 percent of initial nonresponders returning a card in PSID and TA respectively. The remail had the weakest effect on nonresponders in the $0 condition in TA with 13.8 percent responding.

The rate of new telephone numbers was 12.5 percent in PSID and 16.8 percent in TA. The $10 and $20 conditions yielded significantly more new telephone numbers than the $0 condition in both studies; $20 yielded more new telephone numbers than $10 but the difference was not statistically significant in either study.

Discussion

The results of this experiment provide evidence consistent with Fumagalli et al. (2010) in finding that a monetary incentive compared to no incentive yields higher return rates of a between-wave contact update, especially in the TA sample. Further, in our 2008 study, a remail significantly increased return rates over a single-mail protocol. In the current study, a remail to nonresponders is associated with a substantial increase in return rates, and especially new telephone numbers, even for those receiving no incentive. Together, these findings support the use of a monetary incentive in combination with a remail for nonresponders.

Of particular note was the fairly substantial return rate in the $0 condition for those in PSID compared to TA. It may be that the higher response rate in PSID reflects the participation of these families over many prior waves which have built up feelings of loyalty and commitment to the study, and possibly an expectation of receiving future incentives. This hypothesis is consistent with other research demonstrating the persistence of the positive effects of incentives over waves of panel studies (Mack et al. 1998; Scherpenzeel et al. 2002). TA participants, in contrast, have limited experience with PSID data collection, and their low rate of return in the $0 condition may reflect an undeveloped sense of loyalty and/or a lack of expectation about receiving an incentive.

Once 2011 production data are available, we will examine whether the higher postcard return rate and consequently higher rate of updated telephone numbers that are due to the more generous incentives pay off in lower tracking rates and fewer calls to final disposition.

While we must wait for the results of 2011 production interviewing to make definitive conclusions, our initial results suggest that the longstanding PSID practice of a $10 incentive for returning the postcard may be a cost-effective strategy for obtaining updated contact information, and that the additional gains from increasing the incentive to $20 may not be worthwhile.

References

Couper and Ofstedal 2009
Couper, M.P. and M.B. Ofstedal. 2009. Keeping in contact with mobile sample members. In (P. Lynn, ed.) Methodology of longitudinal surveys.Wiley, New York. pp. 188–203.
Duncan and Kalton 2000
Duncan, G.J. and G. Kalton. 2000. Issues of design and analysis of surveys across time. International Statistical Review 55(1): 97–117.
Fumagalli et al. 2010
Fumagalli, L., H. Laurie and P. Lynn. 2010. Experiments with methods to reduce attrition in longitudinal surveys. Institute for Social and Economic Research Working Paper 2010–04. University of Essex.
Laurie et al. 1999
Laurie, H., R. Smith and L. Scott. 1999. Strategies for reducing nonresponse in a longitudinal panel survey. Journal of Official Statistics 15(2): 269–282.
Laurie and Lynn 2009
Laurie, H. and P. Lynn. 2009. The use of respondent incentives on longitudinal surveys. In: (P. Lynn, ed.) Methodology of longitudinal surveys. Wiley, New York. pp. 205–233.
Mack et al. 1998
Mack, S., V. Huggins, D. Keathley and M. Sundukchi. 1998. Do monetary incentives improve response rates in the Survey of Income and Program Participation? Proceedings of the American Statistical Association, Survey Research Methods Section, pp. 529–534. American Statistical Association, Washington, DC.
Martin et al. 2001
Martin E., D. Abreu and F. Winters. 2001. Money and motive: effects of incentives on panel attrition in the survey of income and program participation. Journal of Official Statistics 17: 267–284.
McGonagle et al. in press
McGonagle, K.A., M.P. Couper and R.F. Schoeni. in press. Keeping track of panel members: an experimental test of a between-wave contact strategy. Journal of Official Statistics.
McGonagle et al. 2009
McGonagle, K.A., M.P. Couper and R.F. Schoeni. 2009. An experimental test of a strategy to maintain contact with families between waves of a panel study. Survey Practice, June 2009: http://surveypractice.org/2009/06/29/panel-contacts.
Rodgers 2002
Rodgers, W. 2002. Size of incentive effects in a longitudinal study. Proceedings of the Survey Research Methods Section of the American Statistical Association, pp. 2930–2935. American Statistical Association, Washington, DC.
Scherpenzeel et al. 2002
Scherpenzeel, A., E. Zimmermann, M. Budowski, R. Tillmann, B. Wernli and A. Gabadinho. 2002. Experimental pre-test of the biographical questionnaire, Working Paper, No. 5-02. Neuchatel: Swiss Household Panel. Downloaded April 26, 2011 from http://aresoas.unil.ch/workingpapers/WP5_02.pdf.
Singer et al. 1999
Singer, E.S., N. Gebler, T. Raghunathan, J. Van Hoewyk and K. McGonagle. 1999. The effect of incentives on response rates in face-to-face, telephone, and mixed mode surveys: results of a meta-analysis. Journal of Official Statistics 15(2): 217–230.


About Survey Practice Our Global Partners Disclaimer
The Survey Practice content may not be distributed, used, adapted, reproduced, translated or copied for any commercial purpose in any form without prior permission of the publisher. Any use of this e-journal in whole or in part, must include the customary bibliographic citation and its URL.