Landline and Cell Phone Response Measures in Behavioral Risk Factor Surveillance System

Mohamed G. Qayad Centers for Disease Control and Prevention

Carol Pierannunzi Centers for Disease Control and Prevention

Pranesh P. Chowdhury Centers for Disease Control and Prevention

Sean Hu Centers for Disease Control and Prevention

Gwynett M. Town Centers for Disease Control and Prevention

Lina S. Balluz Centers for Disease Control and Prevention

Introduction

It is widely recognized that response rates of landline household random digital dialing (RDD) surveys are declining (Galea and Tracy 2007; Montaquila et al. 2007). Many factors contribute to the decline of response rates, among them new technologies which allow potential respondents to screen callers using call blocking and caller ID. Estimates for 2009 (Blumberg and Luke 2009) indicate that nearly one fourth of all U.S. households were wireless only. An additional one in eight households rely on cell phones for the majority of telephone calls. This could also be a source of both nonresponse and under coverage in landline RDD samples. As a result, researchers are adopting mixed modes to compensate for both under coverage and nonresponse (Dillman et al. 2009).

Although trends toward cell phone usage might simply direct researchers to incorporate cell phone RDD samples, there are barriers to the adoption of cell phone samples (Brick et al. 2007; Pew Foundation 2006; Steeh 2004). Cell respondents are more difficult to reach than are landline respondents, due to safety concerns (such as driving), technologies which allow potential respondents to screen calls before answering, including caller ID; call blocking; and the proportion of phones which are used exclusively by minor children, who are not eligible respondents in many surveys. This produces a larger number of unproductive calls. Finally, the Telephone Consumer Protection Act (2010) requires survey centers to manually dial cell phone numbers rather than using the automated dialing systems employed in most RDD landline surveys. Recent meta-analysis and earlier studies show that cell phone surveys are more expensive than similar surveys conducted using landline (Guterbock et al. 2010; Keeter 2007).

The Behavioral Risk Factor Surveillance System (BRFSS) is a state based RDD for non-institutionalized adults over 18 years, conducted by state health departments in collaboration with the Centers for Disease Control and Prevention (CDC). The survey has been a unique source of data for health behaviors, chronic disease conditions, and the use of preventive health services for states since 1984 (http://www.cdc.gov/brfss). Now active in all 50 states, three U.S. territories, and the District of Columbia (D.C.), the BRFSS is the largest telephone health surveillance system in the world with over 400,000 interviews completed each year. Traditionally conducted as an RDD landline survey, the BRFSS started collecting data by cell phone from 2008. In 2009, cell phone RDD samples were included in 48 states, D.C., Guam, Puerto Rico, and the Virgin Islands.

Many surveys begin by using screening questions to ascertain whether persons reached by telephone are eligible respondents. Screening questions may also be used to select from among eligible adults within households. Introductory scripts which describe the survey and introduce the respondent to the survey topic have been shown to have effects on response rates (Currivan et al. 2007). Both landline and cell phone questionnaires of the BRFSS include screening questions at the outset of the survey to ensure that respondents meet eligibility criteria. The BRFSS screens landline households by ensuring that the number reached is a landline phone within a residence. Respondents are then selected at random from the number of adults who reside within the household. Cell phone respondents are screened to ensure that the interviewer has reached an adult who lives within a private residence. Cell phone respondents are further screened to eliminate persons who could also be within the landline sample. Therefore, if a potential respondent reached by cell phone indicates that he or she also maintains a landline telephone in his or her residence, he or she is screened out of the sample. As a result of this additional screening question, many potential respondents reached by cell phone do not qualify to be interviewed.

The literature suggests that cell phone respondents will be more difficult to reach and that they will be less cooperative than respondents reached by landline phones (Link et al. 2007). Although much is known to support the proposition that cell phone respondents are difficult to reach, little is known concerning their propensity to cooperate in survey research once they have been successfully contacted. This research describes the overall and state level response and cooperation rates of the landline and cell phone only samples of the BRFSS.

Methods

Data presented here are taken from the 2009 BRFSS. All states, U.S. territories and the District of Columbia were included in the landline sample, but two states (Tennessee and South Dakota) did not collect data from their cell phone only household residents. Both the landline and cell phone samples are supplied by Marketing Systems Group (2010). A sampling of 100 number banks is generated from all area codes and telephone exchanges within each state. The 100 number banks with at least one published residential number (one-plus) are disproportionately sampled with number banks which do not have at least one published residential number at a ratio of 1.5:1, resulting in a disproportionate stratified sample (DSS). The cell phone samples are selected randomly from dedicated 1,000 cell phone banks. The sampled numbers are dialed manually.

States conduct the BRFSS separately and submit data in a standard format to the CDC. Standardized disposition and data files for landline and cell phone surveys are submitted separately by the states. BRFSS uses the American Association for Public Opinion Research (AAPOR) standards for reporting interview outcomes.

Using the landline and cell phone disposition files for the 48 states that collected both landline and cell phone data, we calculated several response measures: response rate4 (RR4), cooperation rate2 (COOP2), differences between landline and cell phone response and cooperation rates, and percent of completed interviews of the total sample for the U.S. In addition, we calculated the percent refusal/break-offs (selected respondents who did not answer the first core question or who answered at least the first core question); percent of contacts not selected or without determining their eligibility; percent reached through answering device/ voicemail and percent with telecommunication barriers to be reached.

We also calculated percent of the sample that completed the screening questions. Among those who completed the screening questions, we calculated the percent of interviews completed, refusal/break-off rates and percent of respondents who did not complete the interview by state in both surveys.

Results

As indicated in Figure 1, rates of completion for cell phone numbers were a little less than half of those for landline phone numbers (4.3% and 10.1%, respectively). Calls to cell phone numbers were more likely to reach answering devices/voicemails (14% for cell phone and 4.2% for landlines), and refusal/break-off rates were also higher among cell phone numbers compared to landline (17% vs. 11.8%), respectively. Telecommunication barriers for the landline and cell phone only household samples were 0.4% and 3.3%, respectively. Landline numbers were also more likely to be non-working.

Figure 1 Distribution of major dispositions of landline and cell phone samples, BRFSS 2009.


SP-Vol_6_Oct_Qayad_fig1.jpg


As Figure 2 illustrates, the sample for landline households included over 4.2 million phone numbers. The cell phone sample included over 400,000 cell phone numbers. Although only 5% of cell phone numbers were successfully screened, compared to 15% of landline numbers, 75% of all screened cell phone numbers produced completed interviews. This number compares favorably with the completion rate of 66% among landline numbers that successfully completed screening questions. Moreover, the refusal/break-off rate among cell phone respondents was also lower after screening, with 15% refusing or breaking off the interview, compared to 20% of landline numbers. The percent of eligible respondents who did not complete the interview, referred as others, was also lower in cell phone compared to landline respondents (10% and 14%, respectively).

Figure 2 Number and percent completed and refused/terminated among those completed screening questions in cell phone and landline BRFSS 2009.


SP-Vol_6_Oct_Qayad_fig2.jpg


Because the BRFSS is completed in each state individually, median state level cooperation and response rates are provided (Table 1). Overall, the median RR4 among the states was 52% for landlines and 39% for cell phones.

Table 1 Response and cooperation rates for cell phone and landline, and differences between cell phone and landline response and cooperation rates by state, BRFSS 2009.

State RR4 response rate
Cooperation rate/COOP2
Cell phone Landline Differences
(cell-landline)
percentage points
Cell phone Landline Differences
(cell-landline)
percentage points
AK 55.2 61.6 –6.4 80.1 77.9 2.2
AL 33 50.1 –17.1 56.4 69.43 –13.0
AR 49.5 49.1 0.4 86 66 20
AZ 23.2 37.1 –13.9 60.9 59.1 1.8
CA 21.1 38 –16.9 47.7 51 –3.3
CO 43.5 60.2 –16.7 85.8 81.7 4.1
CT 21.2 42.2 –21 69.3 70 –0.7
DC 20.6 40.8 –20.2 70.9 66.5 4.4
DE 36.3 44.8 –8.5 68.6 74.2 –5.6
FL 26.5 50.7 –24.2 69.8 80 –10.2
GA 36.5 54.6 –18.1 78.2 83.1 –4.9
GU 55.9 71.2 –15.3 92.4 89.2 3.2
HI 25.4 44 –18.6 66.3 67.6 –1.3
IA 58.5 60.8 –2.3 84.6 76.5 8.1
ID 51.5 51.6 –0.1 88.1 70.7 17.4
IL 29.4 50.1 –20.7 67.5 74.5 –7
IN 39.9 45.8 –5.9 80 63.6 16.4
KS 39.2 58.5 –19.3 70.5 76.2 –5.7
KY 39.7 68.5 –28.8 76 86.8 –10.8
LA 39 51.9 –12.9 81.5 70.8 10.7
MA 29.4 46.4 –17 69.6 78.1 –8.5
MD 20.5 38.6 –18.1 65.5 63.2 2.3
ME 38.5 53.6 –15.1 79.9 74 5.9
MI 38.6 65.1 –26.5 80.5 79.6 0.9
MN 36.7 61.7 –25 73.6 84.9 –11.3
MO 41.9 55.8 –13.9 81.5 76.1 5.4
MS 46.8 47.3 –0.5 77.8 69.8 8
MT 49.1 61.8 –12.7 73.6 78.2 –4.6
NC 35.2 52.7 –17.5 71.5 79.8 –8.3
ND 40.5 56.9 –16.4 81.1 73.6 7.5
NE 53 65.3 –12.3 82.2 79.6 2.6
NH 34.8 48.3 –13.5 74.5 73.2 1.3
NJ 24.2 44.9 –20.7 59 77.8 –18.8
NM 43.7 58.1 –14.4 78.6 74.9 3.7
NV 25.7 47.9 –22.2 71.4 69.9 1.5
NY 24.1 37.7 –13.6 68.8 61 7.8
OH 32.6 47.7 –15.1 71.4 70.1 1.3
OK 49.3 57.4 –8.1 77.8 76.6 1.2
OR 39.7 52 –12.3 90.9 85.5 5.4
PA 29.9 45 –15.1 69.5 66.8 2.7
PR 51.7 69.2 –17.5 93.1 89.7 3.4
RI 22.3 41.9 –19.6 62.4 64.3 –1.9
SC 35.7 60.2 –24.5 67.8 79.65 –11.85
TX 34.7 44.5 –9.8 74.9 66.7 8.2
UT 43.9 64.8 –20.9 83.9 81.3 2.6
VA 37.6 51 –13.4 79.6 82.4 –2.8
VI 45.9 54.3 –8.4 78.1 69.7 8.4
VT 45.3 60.1 –14.8 86.5 80 6.5
WA 29.4 46.5 –17.1 84.3 70.1 14.2
WV 43.2 62.1 –18.9 81 77.5 3.5
WI 53 57.1 –4.1 81.7 82.7 –1
WY 60.3 52.4 7.9 84.9 72.2 12.7
Median 38.6 52.0 13.4 77.8 74.5 3.3

State Level Differences

Nationwide, landline respondents had higher levels of completion of screening questions (Figure 2). This remained true for all states, as illustrated in Figure 3. Table 1 also illustrates the differences in cooperation rates between cell phone and landline samples by state, with some states showing stronger cooperation for landline respondents (Alabama, Florida, New Jersey, Kentucky, Minnesota, and South Carolina), while the majority of states show higher levels of cooperation for cell phone respondents (as indicated by the negative value of the differences between the landline and the cell phone). Thus, the trend noted in the nationwide data is also observed in the majority of states. Contrary to the findings in the cooperation rates, landlines had higher RR4 than cell phones for all states, except Wyoming.

Figure 3 Percentage point differences in completing screening questions between landline and cell phone, BRFSS 2009.


SP-Vol_6_Oct_Qayad_fig3.jpg


Note: Positive values indicate that landline has higher percent than cell phone.

Table 2 shows the percentage of completes, refusal/break-offs and eligible respondents not completing the interviews for all screened respondents in both cell phone and landline samples in 2009. Although the data indicate that differences between completion rates and refusal/break-off rates for screened respondents may vary among states, the findings indicate higher completions in cell phones than landline in all but nine states. The refusal/break-offs and eligible respondents not completed the interview also show that landline numbers had higher percentages than cell phones numbers in the majority of states, which contributes to lower completion rate in the landline among screened respondents.

Table 2 Cell phone and landline outcomes after screening by state, in percentages.

State Call outcomes after screening
Completed interviews
Refusal and break-offs
Did not complete interview
Cell phone Landline Cell phone Landline Cell phone Landline
AL 73.4 62.8 20.3 27.2 2.9 9.4
AK 81.1 69 8.3 18 10.1 12.7
AZ 52.2 52.1 31.7 32.5 15.9 15.4
AR 87.9 59.8 6.9 25.9 4.8 13.9
CA 87.6 85.6 7.6 13.7 4.6 0.7
CO 84.8 72.2 11 13.8 2.9 13.7
CT 57.5 59.3 24.7 23.1 17.2 17.5
DE 98.6 65.1 1.4 19.9 0 14.3
DC 64 55.8 22.7 26.2 13.3 17.9
FL 72.2 69.3 16.8 12.3 9.3 18
GA 79 70.6 17.1 11 3.3 18.2
HI 63.8 52.9 15.6 18.2 20.2 28.4
ID 88.5 64.2 8 24 3.2 11.5
IL 53.2 62.6 19.5 20 26.5 17.4
IN 81.6 56.3 13.2 28.7 5.2 14.6
IA 85.3 69.8 10.8 19 3.9 10.9
KS 71.6 69.4 21.5 20.2 6.2 10.2
KY 73.9 81.6 15.6 12.7 10.5 5.6
LA 81.9 65.5 13.2 24.7 2.4 9.2
ME 80.3 67.2 14.1 20.7 5.1 11.5
MD 57.3 53.5 25.8 30.1 16.9 16.4
MA 70.9 66 23.2 14.7 5.9 19.1
MI 72.1 69.1 16.6 17.2 10.8 13.7
MN 59.5 72.5 19.2 12 21.1 15.5
MS 76.4 61.3 14.9 21.5 7.3 15.9
MO 79.5 66.9 12.8 18.2 7.4 14.7
MT 70.8 71.8 16.5 17.7 11.7 10.1
NE 83.4 72.7 11.1 16.5 5 10.5
NV 81 64.8 14.1 24.7 3.2 9.9
NH 65.3 66.5 21.2 23.1 13.5 10.4
NJ 67.9 62.9 29 14.8 3.1 21.9
NM 75.7 67.8 15.6 18.9 7.9 13.2
NY 74.9 52.1 19.3 28.8 4.8 18.6
NC 73.5 71.8 16.9 15.4 8.8 12.4
ND 84.4 66.6 10 21.4 4.1 11.7
OH 69.1 62.2 24.5 23.6 6.4 14.1
OK 74.5 66.3 15.4 16.5 9.5 17
OR 86 72 10.9 10.7 3.1 17.4
PA 71.2 59.4 19.6 26.1 7.9 14.1
RI 53.6 57 29.8 28.9 16.6 14
SC 64 70.5 19.4 15.4 15.8 13.8
TX 72.5 57 17.5 24.9 9.4 17.7
UT 79.6 74.3 10.1 14.9 8.8 10.6
VT 87.2 71.7 8.2 15.7 3.4 12
VA 80.2 70.1 14.1 11.8 5.8 18
WA 85.3 60 11.4 22.7 1.7 17.2
WV 85.8 78.6 9.7 12.8 3 8.3
WI 77.9 70.6 15.2 19.4 5.5 9.9
WY 85.4 65 10.5 22.2 4.1 12.5
GU 84.2 79.4 6.6 6.9 7.7 13.7
PR 88.1 78.9 4.9 5 6.6 15.8
VI 79 61 14.7 22 4.9 16.1
Median 77.2 66.6 15.3 19.2 6.3 14.0

Discussion

The findings agree with the current literature that cell phone populations have lower rates of response than do landline populations (Brick et al. 2007; Link et al. 2007; Steeh and Piekarski 2008). As the data shows, cell phone only numbers respond less often to telephone calls and in completing the screening questions and have higher refusal and break-off (selected respondents who did not answer the first core question or who answered at least the first core question) rates relative to landline samples. The findings here are comparable to those reported in the AAPOR Cell Phone Task Force Report in 2010 (AAPOR 2010) which estimated cell phone refusals and break-offs at five percentage points higher than landline refusals and break-offs.

Contrary to the findings above, the completion rates, cell phone refusal and break-offs are lower after the respondent completes the screening questions. This partially explains the higher cooperation rate of the cell phone respondents than the landline, 78% and 75%, respectively. Because the proportion among the cell phone samples that completes the screening questions was lower than that of the landline sample (5% vs. 15%), the overall contribution to the response rate is trivial. Recent research (Frasier and Morrison 2011) also indicated that cell phone respondents who complete eligibility screening were more cooperative than the landline respondents.

The landline response rates were higher than cell phone in all states, except Wyoming. However, the cooperation rates in the cell phone sample were higher in the majority of the states, which is attributed to the reasons mentioned earlier. The consistency of higher response rates in almost all states assures the favorability of the landline mode over cell phone, despite the looming coverage bias in landline.

In addition, the higher completion rates in the cell phone, compared to landline, in the majority of states among respondents who completed the screening questions indicates the cell phone respondents’ tendency to complete the interview when conditions are likely to be favorable to complete. This phenomenon was consistent across the majority of states, which shows that similar conditions among the states are probably driving this favorable situation and merits further investigation. Survey organizations could benefit from such situation in the cell phone if common conditions are found across the states.

The results indicate that the landline survey has higher response rate than cell phone, whereas the cell phone show higher cooperation rate. There are three major challenges that the cell phone survey has to reverse to achieve a response rate comparable to that of the landline: (1) higher rate of calls reaching voicemails, (2) higher rate of contacted individuals not initiating the interview, and (3) higher rate of refusal/break-offs. Despite these challenges and having lower completion of screening questions in the cell phone, the cell phone showed favorable completion rate than the landline among respondents who completed the screening questions. Thus, the success with cell phone populations is dependent on finding the factors responsible for the lower screening completion and the higher completion of the interview among those who completed the screening questions.

Disclosure: The BRFSS survey was reviewed by the Human Research Protection Office at the Centers for Disease Control and Prevention and determined to be exempt from human subject guidelines. Authors have no conflict of interest to disclose.

Disclaimer: The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

References

AAPOR 2010
AAPOR Cell Phone Task Force. 2010. New considerations for survey researchers when planning and conducting RDD telephone surveys in the U.S. with respondents reached via cell phone numbers. Available at: http://www.aapor.org/Cell_Phone_Task_Force_Report.htm.
Blumberg and Luke 2009
Blumberg, S.J. and J.V. Luke. 2009. Wireless substitution: early release of estimates from the National Health Interview Survey, January – June 2009. Available at: http://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless200912.pdf.
Brick et al. 2007
Brick, J.M., P.D. Brick, S. Dipko, S. Presser, C. Tucker and Y. Yuan. 2007. Cell phone survey feasibility in the US: sampling and calling cell numbers versus landline numbers. Public Opinion Quarterly 71(1): 23–39.
Currivan et al. 2007
Currivan, D., M.K. Farrelly, J. Pais and C. Phelan. 2007. The effects of introductory scripts and respondent incentives on overall participation and ubgroup participation in an RDD telephone survey. Presented to the AAPOR Conference, Anaheim, CA.
Dillman et al. 2009
Dillman, D.A., G. Phleps, R. Tortora, K. Swift, J. Kohrell, J. Berck and B L. Messer. 2009. Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the internet. Social Science Research 38(1): 1–18.
Frasier and Morrison 2011
Frasier, A.M. and H.M. Morrison. 2011. Cell phone versus landline respondents: who’s more cooperative after screening? In Joint Statistical Meeting (JSM), Miami, FL.
Galea and Tracy 2007
Galea, S. and M. Tracy. 2007. Participation rates in epidemiologic studies. Ann Epidemiol. 17(9): 643–653.
Guterbock et al. 2010
Guterbock, T.M., P.J. Lavrakas, C.T. Stamford, T.N. Tompson and R. ZuWallack. 2010. The variable costs of cell phone interviewing: understanding cost and productivity ratios in dual-frame telephone surveys. AAPOR Annual Conference, Chicago, IL. Available at: http://www.aapor.org/AM/Template.cfm?Section=65th_Annual_Conference&Template=/CM/ContentDisplay.cfm&ContentID=2385.
Keeter 2007
Keeter, S. 2007. How serious is polling cell-only problem? Pew Research Center. Available at: http://pewresearch.org/pubs/515/polling-cell-only-problem.
Link et al. 2007
Link, M., M.P. Battaglia, M.R. Frankel, L. Osborn and A.H. Mokdad. 2007. Reaching the cell phone generation. Public Opinion Quarterly 71(5): 814–839.
Montaquila et al. 2007
Montaquila, J., J.M. Brick, M.C. Hagedorn, C. Kennedy and S. Keeter. 2007. Aspects of nonresponse bias in RDD telephone surveys. In: Advances in telephone survey methodology. Wiley, New York.
Marketing Systems Group 2010
Marketing Systems Group. 2010. Genesys Sampling System. Available at: http://www.genesys-sampling.com/home.aspx.
Pew Foundation 2006
Pew Foundation. 2006. The cell phone challenge to survey research. Available at: http://people-press,org/report/276/the-cell-phone-challege-to-survey-research.
Steeh 2004
Steeh, C. 2004. A new era for telephone surveys. Presented at the AAPOR, Phoenix, AZ.
Steeh and Piekarski 2008
Steeh, C and L. Piekarski. 2008. Accommodating new technologies: mobile and VoIP communication. In: (J.M. Lepkowki, C. Tucker, J.M. Brick, E. De Leeuw, L. Japec, P.J. Lavrakas, M.W. Link and R. L. Sangster, eds.) Advances in telephone survey methodology. Wiley, New York.
Telephone Consumer Protection Act 2010
Telephone Consumer Protection Act. 2010. Available at: http://www.fcc.gov/cgb/policy/TCPA-Rules.pdf.


About Survey Practice Our Global Partners Disclaimer
The Survey Practice content may not be distributed, used, adapted, reproduced, translated or copied for any commercial purpose in any form without prior permission of the publisher. Any use of this e-journal in whole or in part, must include the customary bibliographic citation and its URL.