Mail as an Alternative to Random-Digit Dial in a Survey of Young Adults

Luciano Viera, Jr., Rebecca L. Medway, Scott R. Turner, and Sean M. Marsh, Fors Marsh Group, LLC

As landline telephone coverage rates continue to decline, researchers are increasingly turning to list-assisted mail surveys as an alternative to random-digit-dial (RDD) telephone surveys. However, limited empirical work has been conducted to determine whether these list-assisted mail surveys represent an improvement over RDD for surveys that attempt to target specific subpopulations. This paper compares the results of an existing RDD survey of young adults with those of a concurrent list-assisted mail administration. We found that the mail administration resulted in improved coverage of the target population, increased efficiency of the administration, and enhanced representativeness of the target population, without reducing the validity of the key survey metric.

Over the past decade, the quality of the estimates produced by RDD telephone surveys has increasingly been questioned. In particular, the potential for noncoverage error has grown as cell-only households have become progressively more common. Among young adults, cell-only households are especially widespread; as of mid-2010, 40 percent of young adults ages 18 through 24 lived in a cell-only household, compared to 25 percent of all adults (Blumberg and Luke 2010). In addition, various characteristics and behaviors have been found to differ significantly between young adults living in cell-only households and those living in landline households (Blumberg and Luke 2007).

RDD surveys have also become increasingly expensive and inefficient as response rates have declined over the past few decades (Curtin et al. 2005). In RDD surveys of young adults, further inefficiency arises from the difficulty of successfully determining, prior to contact efforts, which phone numbers represent young-adult households; as a result, RDD surveys of young adults may waste resources by attempting to contact households that do not include members of the target population.

In the face of these challenges, survey practitioners are considering alternatives to traditional, landline-only RDD surveys (Link et al. 2008; Sherr et al. 2009). The two most common approaches are to add a cell phone sample to a landline sample or to switch to a list-based, mail sample. A list-assisted mail approach was chosen for this study because of its ability to efficiently identify young adults prior to contact efforts through the use of appended demographic data, while also reaching cell-only households. A cell-phone sample was not employed because of the low response rate and high costs associated with this approach (Brick et al. 2007; Link et al. 2007).

However, limited empirical work has been conducted to verify that list-assisted mail surveys represent an improvement over RDD for surveys of specific target populations. The present study addresses this gap in the literature by assessing whether a list-assisted mail administration yields superior data quality compared to an existing RDD survey of young adults. The study compares the two administrations in terms of

  • coverage of the target population,
  • efficiency of the administration,
  • representativeness of the respondents, and
  • validity of the resulting survey estimates.

Methodology

The RDD Telephone Survey The Youth Poll has been conducted by the Department of Defense (DoD) for over thirty years. The results presented in this article are from the December 2008 administration. The survey is conducted biannually using a list-assisted, landline-only RDD sample of young adults. Interviews are completed using computer-assisted telephone interviewing (CATI). The target population is non-institutionalized, English-speaking youth ages 16 to 24 who live in the United States and have never served in the U.S. Military.

The Youth Poll asks young people about their career and education plans, particularly as these relate to potential military service. The key metric produced by this survey is “military propensity” – defined as the proportion of youth responding that they will definitely or probably be serving in the U.S. Military in the next few years.

The List-Assisted Mail Survey The mail survey was administered in December 2008. It was conducted concurrently with the RDD Youth Poll in order to maximize comparability. The mail survey had the same target population as the RDD survey. The frame included approximately 28 million cases; address, name, and age information were available for each case. The frame was created by the DoD using government administrative lists, as well as a list purchased from a private vendor (see Table 1). The source lists were merged and purged of duplicate listings to create the frame.

Table 1 Case Sources for Mail Frame.

Source Percentage of Cases
Purchased List 65%
Department of Motor Vehicles 19%
Selective Service 15%
Armed Services Vocational Aptitude Battery <1%
Millitary Service Information Requests <1%
Total 100%

Ten thousand cases were randomly sampled from the frame. Over a two-month period, sample members were sent a notification letter, an initial survey package, a reminder/thank you postcard, and two replacement survey packages. The initial survey package included a two-dollar incentive. The survey instrument contained a subset of the RDD questionnaire items, including the military propensity item.

Results

Coverage While the RDD survey sample was randomly drawn from the U.S. landline household population, the mail survey sample was drawn from lists created by multiple sources. As a result, the coverage properties and “randomness” of the mail sample are less clear. Despite this limitation, coverage estimates for the two surveys have been calculated and compared to one another. The estimated coverage rate of the mail survey (73 percent) was higher than that of the RDD survey (67 percent).

The estimated 73 percent coverage rate for the mail survey was computed by comparing the mail frame with the July 2008 Population Estimates Program (PEP) data produced by the Census Bureau. PEP data was selected as the benchmark because it includes estimates of the institutionalized and enlisted populations, which were included in the mail frame but are not part of Current Population Survey (CPS) estimates.

The estimated 67 percent coverage rate for the RDD survey is based on the Centers for Disease Control and Prevention (CDC) July – December 2008 data, which estimated that about one-third of 18 to 24 year olds live in cell-only households (Blumberg and Luke 2009). Due to a lack of available data, this estimate does not take into account the telephone status of 16- to 17-year-olds or the proportion of 18- to 24-year-olds that live in households with no telephone at all.

Efficiency The mail administration was more efficient than the RDD administration (see Table 2). The cost per mail respondent was less than half the cost per RDD respondent. The mail administration was more successful than the RDD administration at targeting the population of interest; about one-fifth of the mail sample resulted in completed questionnaires, while only 1 percent of the RDD sample resulted in completed interviews. The inefficiency of the RDD survey stemmed from the difficulty of reaching the target population. About one-third of the dialed numbers were nonworking; another fifth were tied to households that did not have a young adult living in them. In contrast, a much smaller proportion of mail cases were undeliverable or resulted in self-reports of ineligibility.

Table 2 Efficiency Measures.

Measures Mail RDD
Length of time to field 2.5 months 2.5 months
Cost per respondent 0.46× ×
Sample size 10,000 332,725
Eligible respondents (Percentage of sample) 1,905 (19%) 4,242 (1%)

Respondent Profile Generally, the RDD and mail survey respondents had similar unweighted demographic characteristics to one another and to December 2008 CPS estimates. However, educational enrollment and attainment were considerably higher among mail survey respondents compared to RDD respondents and CPS estimates. This difference was accounted for in the weighting of the mail survey.

Respondents to the two surveys also differed in their phone usage. Mail respondents were considerably more likely than RDD respondents to have only a cell phone. RDD respondents were considerably more likely than mail respondents to have only a landline. Among respondents that had both a landline and a cell phone, mail respondents were more likely than RDD respondents to report using their cell phone to make all or almost all of their calls (61% and 44%, respectively); although these heavy cell phone users are covered by the RDD frame, they are likely difficult to reach on a landline.

Table 3 Phone Usage by Survey Mode (unweighted).

Phone Usage Mail
RDD
(n=1881+) (n=4204+)
Cell phone only 21% 7%
Landline only 9% 22%
Both landline and cell phone 69% 66%
No telephone 2% 5%

+Respondents who did not answer the phone usage questions were excluded.

Note: Sum of percentages may not add to 100 due to rounding.

Key Metric: Military Propensity The key metric resulting from this survey is military propensity – defined as the proportion of youth responding that they will definitely or probably be serving in the U.S. Military in the next few years. The weighted estimate of military propensity resulting from the mail survey was significantly lower than that resulting from the RDD survey (7 percent versus 11 percent). We believe the mail survey estimate was lower because of improved representation of cell-only, cell-mostly, and no-phone youth in the mail survey.

Cell-mostly youth were significantly less likely to report interest in military service in the mail administration than they were in the RDD administration. Military propensity was statistically equivalent in the two administrations for all other phone usage groups.

Among mail survey respondents, youth who used their cell phone at least as much as their landline were considerably less likely to be interested in military service than were landline-only or landline-mostly youth (Table 4). In particular, cell-mostly youth were significantly less likely to be propensed than were youth in other landline/cell usage subgroups.

Table 4 Millitary Propensity by Phone Usage (weighted).

Propensity
Mail
RDD
Percentage (95% Confidence Interval) Percentage (95% Confidence Interval
Overall 7%* (5.7%–8.4%) 11% (10.0%–12.6%)
Landline only 12% (6.9%–17.4%) 14% (10.7%–16.7%)
Landline mostly 13% (3.5%–22.8%) 10% (4.9%–14.4%)
Landline/cell phone equal 8% (4.9%–10.5%) 10% (7.7%–12.0%)
Cell mostly 4%* (2.7%–6.0%) 10% (7.5%–12.4%)
Cell only 8% (4.4%–11.4%) 15% (9.1%–21.1%)
No phone 8% (0%–17.6%) 16% (9.5%–22.4%)

Note:*=p<0.05.

Conclusions and Plans for Future Research

The mail administration appears to have improved upon the weaknesses of the RDD administration, while maintaining valid estimates of the key survey metric. The mail survey was more cost-effective and more efficient than the RDD survey. It also had a higher estimated coverage rate than the RDD survey. Although the military propensity estimate resulting from the mail survey was significantly lower than the one resulting from the RDD survey, this was likely due to the improved coverage of cell-only and cell-mostly youth.

Additional concurrent mail and RDD administrations are currently being conducted to determine whether the mail survey can replace the RDD survey, with a focus on further increasing coverage and efficiency while maintaining valid estimates.

References

Blumberg and Luke 2007
Blumberg, S.J. and J.V. Luke. 2007. Coverage bias in traditional telephone surveys of low-income and young adults. Public Opinion Quarterly 71(5): 734–749.
Blumberg and Luke 2009
Blumberg, S.J. and J.V. Luke. 2009. Wireless substitution: early release of estimates from the national health interview survey, July-December 2008. National Center for Health Statistics May 2009. Available from: http://www.cdc.gov/nchs/nhis.htm.
Blumberg and Luke 2010
Blumberg, S.J. and J.V. Luke. 2010. Wireless substitution: early release of estimates from the National Health Interview Survey, January-June 2010. National Center for Health Statistics, December 2010. Available from: http://www.cdc.gov/nchs/nhis.htm.
Brick et al. 2007
Brick, J.M., P.D. Brick, S. Dipko, S. Presser, C. Tucker and Y. 2007. Yuan. Cell phone survey feasibility in the U.S.: sampling and calling cell numbers versus landline numbers. Public Opinion Quarterly 71(1): 23–39.
Curtin et al. 2005
Curtin, R., S. Presser and E. Singer. 2005. Changes in telephone survey nonresponse over the past quarter century. Public Opinion Quarterly 69(1): 87–98.
Link et al. 2007
Link, M.W., M.P. Battaglia, M.R. Frankel, L. Osborn and A.H. 2007. Mokdad. Reaching the U.S. cell phone generation: comparison of cell phone survey results with an ongoing landline telephone survey. Public Opinion Quarterly 71(5): 814–839.
Link et al. 2008
Link, M.W., M.P. Battaglia, M.R. Frankel, L. Osborn and A.H. Mokdad. 2008. A comparison of address-based sampling (abs) versus random-digit dialing (RDD) for general population surveys. Public Opinion Quarterly 72(1): 6–27.
Sherr et al. 2009
Sherr, S., D. Dutwin, T. Triplett, D. Wissoker and S. Long. 2009. Comparing random digit dial (RDD) and united states postal service (USPS) address-based sample designs for a general population survey: the 2008 massachusetts health insurance survey. Paper presented at Annual Conference of the American Association for Public Opinion Research. Hollywood, FL.


About Survey Practice Our Global Partners Disclaimer
The Survey Practice content may not be distributed, used, adapted, reproduced, translated or copied for any commercial purpose in any form without prior permission of the publisher. Any use of this e-journal in whole or in part, must include the customary bibliographic citation and its URL.