Incentives for Cell Phone Only Users: What Difference Do They Make?

Robert W. Oldendick and Dennis N. Lambries University of South Carolina

Providing incentives to potential respondents has been shown to be one of the more effective means of encouraging participation in surveys across modes of data collection. The increasing number of cell-phone only (CPO) households has sparked an interest in best practices for interviewing cell phone respondents. One aspect of interviewing cell phone respondents that has not been adequately addressed is the question of the need to provide some type of remuneration to those who complete the interview on a cell phone. In this research, we provide evidence that suggests that providing such remuneration does not have a significant impact on either response rates or results.

Although the literature on this topic is not extensive, there have been several studies of this topic (Dillman 2007; Singer 2002; Trussell and Lavrakas 2004). For example, Brick and his colleagues (2007) found in a national survey of cell phone households that a $10 incentive significantly improved participation over a $5 incentive. Similar studies by Diop and his colleagues (2008a, 2008b) found that a $10 incentive produced a higher rate of production compared to no incentive or to a $5 incentive. These earlier studies were based on unscreened dual-frame designs that included all cell phone users – that is, those both with and without landlines – while our research focused on those in cell phone only households, which may account for some of the difference in the findings.

In the early 2000s, most initial approaches to contact cell phone users presumed that some type of incentive or remuneration would be given. This was usually done as a reimbursement for possible charges incurred by the respondent, although in some cases they may have functioned as incentive payments. One of the conclusions from the Cell Phone Sampling Summit II was that “survey respondents reached on their cell phone should be properly reimbursed for their time on a research call” (Lavrakas and Shuttles 2005). As the cost structure of cell phone billing has changed and more users have moved to virtually “unlimited minutes” plans, does reimbursement of cell phone users remain a sine qua non?

To test the impact of incentives on response rates for cell phone users, we conducted experiments as part of two dual-frame telephone surveys of the adult population of South Carolina done in November and December 2009. When a number was reached, a series of screening questions determined whether the person was contacted on a cell phone, was 18 years of age or older, a resident of the state, and did not have a landline residential telephone.

The survey introduction requested potential respondents to participate in “a confidential study of public opinion” being conducted by the University. All procedures in the “incentive” and “non-incentive” conditions were the same, except at the end of the survey introduction those in the incentive condition were read the statement, “If you are eligible and complete this study, you will receive a $10 Wal-Mart gift card.” (A more extensive description of the procedures used in this study is available from the authors).

Table 1 provides the final call dispositions for 1500 numbers dialed in each of these conditions. As expected given the independent samples in each condition, the percentage of not-in-service and ineligible (business, changed number, household with a landline, respondent under 18, or not a resident of the state) are quite similar, with the largest differences (between conditions in Survey I) being in the percentage of not-in-service numbers (32.7 percent vs. 36.8 percent) and the percentage of those contacted who were not in CPO households (15.4 percent vs. 13.1 percent).

Table 1 Call Dispositions – Cell Phone Samples, Incentive and Non-Incentive Conditions.

Survey I
Survey II
Incentive Non-Incentive Incentive Non-Incentive
No Answer/Busy 39 (2.6) 35 (2.3) 39 (2.6) 38 (2.5)
Answering machine 240 (16.0) 245 (16.3) 262 (17.5) 268 (17.9)
Not in service 490 (32.7) 552 (36.8) 480 (32.0) 472 (31.5)
Business/Changed 44 (2.9) 44 (2.9) 23 (1.5) 37 (2.5)
Language/Ill 6 (0.4) 5 (0.3) 10 (0.7) 11 (0.7)
Has LL/Not cell 231 (15.4) 197 (13.1) 221 (14.7) 221 (14.7)
Under 18/Not res. 59 (3.9) 85 (5.7) 86 (5.7) 77 (5.1)
Unable to complete 123 (8.2) 107 (7.1) 107 (7.1) 136 (9.1)
Refusals 181 (12.1) 162 (10.8) 199 (13.3) 158 (10.5)
Completions 87 (5.9) 68 (4.5) 73 (4.9) 82 (5.5)

More importantly for our purposes, the percentage of completed interviews did not vary substantially across conditions, ranging from 5.9 percent in the Survey I incentive condition to 4.5 percent in this survey’s non-incentive condition. The standard response rates (AAPOR RR4) for these four conditions were:

Survey I, Incentive Condition: 17.8
Survey I, Non-Incentive Condition: 16.2
Survey II, Incentive Condition: 15.0
Survey II, Non-Incentive Condition: 16.9

The slightly higher response rate in the expected direction in Survey I is offset by the difference in the opposite direction in Survey II. In neither case was the difference in response rates between conditions large enough to make the case that providing an incentive produced a substantial difference.

The lack of difference between the two conditions is also reflected in the demographic composition of the two samples. Table 2 shows the background characteristics for each of the cell conditions, as well as those for the landline samples. As would be expected given the characteristics of those living in CPO households, the cell phone samples reached a much higher percentage of younger people, minorities, men, and those from lower income families than did the landline samples. When the characteristics of the respondents in the cell phone conditions are compared, none of the differences is significant at the 0.05 level. The only difference that approaches significance is that for age in Survey II, which had a higher percentage of those under 30 in the non-incentive condition and a higher percentage of 30–44 year olds in the incentive condition.

Table 2 Demographic Comparisons - Cell Phone Conditions and Landline.

Survey I
Survey II
No Inc Incent Landline No Inc Incent Landline
AGE
 18–29 36.7 29.6 6.3 46.3 30.1 6.7
 30–44 30.0 34.6 18.8 18.8 35.6 18.1
 45–64 30.0 28.4 42.3 31.3 30.1 42.1
 65 or over 3.3 7.4 32.7 3.8 4.1 33.2
 (N) (60) (81) (640) (80) (73) (630)
p* NS <0.10
RACE
 Black 23.2 34.1 25.2 40.7 31.5 23.5
 White 68.3 56.1 71.6 51.9 65.8 72.7
 Other 8.3 9.8 3.2 7.4 2.7 3.8
 (N) (60) (82) (647) (81) (73) (637)
p* NS NS
SEX
 Male 52.9 52.9 41.2 52.4 49.3 43.8
 Female 47.1 47.1 59.8 47.6 50.7 56.2
 (N) (68) (87) (673) (82) (73) (657)
p* NS NS
EDUC
 Less than high school 11.5 9.8 10.7 17.5 6.8 11.3
 High school graduate 27.9 30.5 26.3 37.5 30.1 28.7
 Some college 37.7 41.5 27.9 22.5 32.9 27.7
 College graduate 23.0 18.3 35.0 22.5 30.1 32.3
 (N) (61) (82) (642) (80) (73) (638)
p* NS NS
FAMILY INCOME
 Less than $50,000 67.9 63.9 56.8 65.2 61.3 53.6
 $50,000 or more 32.1 36.1 43.2 34.8 38.7 46.4
 (N) (53) (72) (521) (69) (62) (503)
p* NS NS

p*=chi-square test for differences between the incentive and non-incentive cell phone conditions.

Similarly, there were very few differences in responses on the substantive questions in these surveys between the cell phone groups. Of the 29 closed-ended items in Survey I, significant differences between the groups were found on three, each involving participation in the arts; of the 24 closed-ended items in Survey II, no differences were significant to the 0.05 level, and three were significant at p<0.10.

Overall, the results of this study in terms of response rates, the demographic composition of respondents, and the substantive results provide little indication that providing an incentive to participants makes an appreciable difference. While there are other considerations involved in the decision as to whether to provide remuneration to such participants, this decision does not appear to have a significant impact upon response rates or the results.

Finally, we recognize some of the limitations of this study. First, it focused on cell phone only users, who may be more likely to have unlimited minute plans and be less concerned about remuneration. This factor may be key to understanding the differences between the results of this study and those of previous research that found significant effects for incentives, in that cell phone only users may be more likely to have extended or unlimited minutes plans and be less concerned about the cost of the call. Second, only one type of incentive was offered, and increasing the value or altering the type of incentive may produce different results. Third, the study is based on a limited number of cases. It is also a single state study and these results may not generalize to other geographic areas. Despite these limitations, we found the results convincing enough to change our standard survey practice for cell phone surveys from offering remuneration to participants to not doing so.

References

Brick and his colleagues 2007
Brick, J.M., P.D. Brick, S. Dipko, S. Presser, C. Tucker and Y. Yuan. 2007. Cell phone survey feasibility in the U.S.: sampling and calling cell numbers versus landline numbers. Public Opinion Quarterly 71(1): 23–39.
Dillman 2007
Dillman, D.A. 2007. Mail and internet surveys: the tailored design method. John Wiley and Sons, New York.
Diop and his colleagues 2008a
Diop, A., Y. Kim, J.L. Holmes and T.M. Guterbock. 2008a. Prince William county cell-phone pilot study: a supplement to the 2007 Citizen Satisfaction Study. University of Virginia Center for Survey Research.
Diop and his colleagues 2008b
Diop, A., D. Kermer and T.M. Guterbock. 2008b. Prince William county citizen satisfaction survey: report of results 2007. University of Virginia Center for Survey Research.
Lavrakas and Shuttles 2005
Lavrakas, P.J. and C.D. Shuttles. 2005. Cell phone sampling summit II: proposed statements on “Accounting for Cell Phones in Telephone Survey Research in the U.S.”
Singer 2002
Singer, E. 2002. The use of incentives to reduce nonresponse in household surveys. In: (R.M. Groves, D.A Dillman, J.L. Eltinge and R.J.A. Little, eds) Survey nonresponse. John Wiley and Sons, New York.
Trussell and Lavrakas 2004
Trussell, N. and P.J. Lavrakas. 2004. The influence of incremental increases in token cash incentives on mail survey response: is there an optimal amount? Public Opinion Quarterly 68(3): 349–367.


About Survey Practice Our Global Partners Disclaimer
The Survey Practice content may not be distributed, used, adapted, reproduced, translated or copied for any commercial purpose in any form without prior permission of the publisher. Any use of this e-journal in whole or in part, must include the customary bibliographic citation and its URL.