Mobile-Only Web Survey Respondents

Peter Lugtig Department of Methods and Statistics
Utrecht University

Vera Toepoel Department of Methods and Statistics
Utrecht University

Alerk Amin RAND Corporation

Abstract

Web surveys are no longer completed on just a desktop or laptop computer. Respondents increasingly use mobile devices, such as tablets and smartphones to complete web surveys. In this article, we study how respondents in the American Life Panel complete surveys using varying devices. We show that about 30 percent of respondents sometimes complete surveys on a mobile device, and about 12 percent always use a mobile device. We study the characteristics of these “mobile-only” web survey respondents and find that they share many characteristics of typically hard-to-recruit survey respondents. They are more likely to be non-white, young, and not have a higher education. In terms of voting behavior, we find no differences between the groups of survey respondents who use different devices. This suggests that biases in political polls conducted through web-surveys are unlikely to occur when mobile-only respondents are underrepresented.

Mobile-Only Web Survey Respondents

Survey researchers have long been concerned about “mobile-only” survey respondents. In the 2000s, random digit dialing surveys included landline phones only. The steady decline of landline coverage rates in both the United States and Europe and rise of mobile phone subscriptions has led survey researchers about a decade ago to start using dual-frame samples containing both landline and mobile phone numbers (Brick et al. 2007; Link et al. 2007). Nowadays, cell-phones often constitute the large majority of sampled telephone numbers (Pew Research 2016).

Where cell phones have now largely replaced landline phones as a method of communication in polling, cell phones nowadays feature enough capabilities to potentially replace the (desktop) computer as well in web surveys. As of July 2015, 73 percent of Americans owned a computer, 68 percent a smartphone, and 45 percent a tablet computer (Pew Research 2015).

In the last couple of years, mobile phones have evolved from a medium of auditive communication into a smart multimedia device. Most smartphones feature a high-speed mobile Internet connection, which enables the use of all kinds of new information and communication technologies.

Earlier studies have documented that in Europe an ever increasing number of Internet panel survey respondents are completing surveys on their tablets and smartphones. We found that in 2014 in the Dutch Longitudinal Internet Studies for the Social sciences (LISS) panel, 15 percent of respondents completed surveys on a tablet and 5 percent on a mobile phone. Others have found similar rates in different panels in different countries (Lugtig and Toepoel, 2016; Mavletova 2013; Struminskaya et al. 2015).

This paper aims to study what defines the group of mobile-only web survey respondents in the United States. With mobile-only, we in practice mean both tablet and cellphone respondents, although we will break down our analysis to show differences between the devices. The remainder of this article is structured as follows:

We will first describe data from the American Life Panel (ALP), which we will use to illustrate device use in one large U.S.-based panel. We will describe the proportion of respondents using different devices over time, both at each of seven measurements within the panel, as well as longitudinally. Then we study the characteristics of different types of respondents and focus on differences in voting behavior between the groups to see whether including or excluding the mobile-only group in web surveys would lead to biases in political polls.

Data

The data in this study stem from the RAND American Life Panel (ALP). The ALP is a nationally representative panel survey, consisting of 6,000 survey respondents. The panel started in 2003 with a probability-based survey in the context of the Health and Retirement Survey, carried out at the University of Michigan, and is now maintained by RAND Corporation. Regular refreshments of panel respondents are added to the panel to ensure the ALP is representative of the whole U.S. population. All ALP-respondents are recruited offline. More details about the recruitment and representativeness of the panel can be found on the website of ALP (https://alpdata.rand.org/).

In this paper, data from seven waves are used which all asked respondents questions related to the midterm congressional elections held on November 4, 2014. The first wave was conducted in May 2014; later waves started on September 28, October 5, 12, 19 and 26. Data in each wave were collected over a one-week period. Finally, a post-election survey was held on November 5, 2014. 2,925 respondents from the ALP panel participated in at least three out of seven questionnaires and are used in our analysis. Respondents who participated in fewer than three waves were dropped from the analyses (2,355 cases in total, which included respondents newly recruited to the panel in October 2015). A separate wave-by-wave analysis including all respondents did not show differences in device use between the respondents included and not included in our analysis.

In each wave, we coded the device respondents used to complete the survey using the user agent strings (UAS). UAS contain information on the device, operating system, and browser being used (Callegaro 2013). The strings were manually coded using a script in R 3.3.0 (R Core Development Team 2016), which is available from the authors.

We defined a regular PC or laptop to be a computer which has a fixed-keyboard, and a screen size larger than 6 inches. A tablet is defined as a device with a screen larger than 6 inches, but without a fixed keyboard. A phone is a device smaller than 6 inches that can also be used to make calls over a cellular network.

As covariates, we use variables that we deem to be relevant in explaining differences in the use of devices. We expect male and younger respondents, as well as respondents in paid work, in larger households, with higher incomes, and of non-white ethnicity to use a mobile device more often (Struminskaya et al. 2015; Toepoel and Lugtig 2014).

Results

Table 1 shows that in each wave of the ALP study, about 80 percent of all respondents use a regular personal computer (PC) or laptop to complete the survey. Slightly less than 10 percent of respondents use a tablet, and slightly more than 10 percent a smartphone. Nonresponse in the ALP panel amounts to about 15 percent of all respondents that are invited in every month.

Table 1 Device use and sample percent at seven waves of American Life Panel.

Start date of survey Regular PC or laptop Tablet Mobile phone Total sample size
May 2014 81.3 7.9 10.5 2,119
Sept. 28 78.6 9.2 12.0 2,546
Oct. 5 79.9 8.8 11.3 2,542
Oct. 12 79.2 8.8 11.9 2,536
Oct. 19 78.1 9.0 12.8 2,700
Oct. 26 78.4 8.4 13.1 2,666
Nov. 5 78.2 9.0 12.7 2,722

n (total)=2,972. A few devices did not fall into any of the categories above (less than n=6 at each wave). These included a camera, printer, and a few devices for which the user agent string was not codable.

For our analysis purposes, it is more relevant to study whether respondents use the same device consistently over different waves. We find that about 80 percent of respondents use the same device at all times (see Table 2). 68 percent of respondents always use a PC or laptop for completing questionnaires, 5 percent always use a tablet and 7 percent always a mobile phone. About 20 percent of respondents use multiple devices over the course of the seven waves we study. 11 percent of respondents use both their PC and phone, 7 percent use a PC and a tablet, while 1 percent of respondents uses all three types of devices over a seven-wave period.

Table 2 Longitudinal device use in the ALP.

Use throughout polling period Sample size Sample (%) Average number of waves responding
Always PC 2,027 68.2 6.05
Always tablet 152 5.1 5.58
Always phone 214 7.2 5.75
Mixed PC – tablet 214 7.2 5.88
Mixed PC – phone 316 10.6 5.88
All devices 47 1.3 6.30

n (total)=2,972. Respondents had to participate in at least three waves in order to be included in our analyses. Two respondents who mixed tablet-phone were excluded from the table.

In accordance to earlier findings (Lugtig and Toepoel 2016; Struminskaya et al. 2015), we again find that mobile respondents (respondents who use their tablets or phones to complete the surveys) are less likely to participate in any given wave. Out of a maximum of seven waves, the ‘always PC’ respondents on average participate in slightly more than six waves, whereas tablet only respondents respond in 5.58 waves. Although this difference may seem small, remember that we constrained our sample to respondents who participated at least three times.

We hypothesize that mobile only respondents in the ALP study are very different from PC respondents on a range of demographic variables. Cook (2014), using data from another panel in the United States, indeed finds that younger and female respondents are more likely to complete surveys on tablets. Smartphone respondents are more likely to have only a high school degree and have a lower income than PC and tablet respondents. Both tablet and smartphone users are more likely to identify as Hispanic or African-American (Cook 2014).

Table 3 shows the results of a multinomial logistic regression analysis explaining longitudinal device use, as categorized into the six longitudinal device use patterns we showed in Table 2. The ‘always PC’ group serves as the reference group, which means that the coefficients shown in Table 3 are relative coefficients of differences between groups. We report the average marginal effect (ME) (Mood 2009). These represent the multivariate effects of the predicted probability of being in the ‘always PC’ group compared to the other groups. For example, the coefficient for age/10 in the ‘Mix PC-Phone’ group means that for every decade that respondents become older, they have a 5 percent lower predicted probability to be in the ‘always PC’ group as compared to the ‘Mix Pc-phone’ group. Although this difference appears to be small, we have to keep in mind that when a random person is selected from either the ‘always PC’ or ‘Mix PC-phone’ groups, he or she has a probability of 0.86 to be in the always PC group, as that group is much larger than the ‘Mix Pc-phone’ group. So a shift of 0.05 in these probabilities for every decade that a respondent is younger, should be considered to be a large effect. If an imaginary respondent aged 60 has a predicted probability of 95 percent of being in the always PC group as opposed to the PC-phone group, this probability is about 75 percent for someone aged 20, controlling for the effects of other covariates.

Table 3 Average marginal effects of demographic variables relative to ‘Always PC’ group.

Always tablet
Always mobile
Mix PC-tablet
Mix PC-phone
ME s.e. p-Value ME s.e. p-Value ME s.e. p-Value ME s.e. p-Value
Born in United States 0.01 0.015 0.52 0.02 0.01 0.14 0.03 0.018 0.15 −0.01 0.022 0.79
Ethn: white (ref=other) 0.00 0.016 0.91 0.00 0.011 0.81 0.01 0.023 0.64 0.02 0.018 0.20
Ethn: African American (ref=other 0.06 0.036 0.09 0.08 0.031 0.01 0.07 0.049 0.13 0.01 0.043 0.03
Hispanic 0.07 0.024 0.00 0.06 0.018 0.00 0.07 0.029 0.01 0.06 0.023 0.01
Gender (ref=male) −0.02 0.01 0.01 −0.01 0.008 0.19 −0.03 0.012 0.01 −0.01 0.012 0.41
Married (ref=never married) 0.001 0.013 0.72 0.02 0.01 0.02 0.05 0.019 0.01 0.03 0.016 0.04
Separated (ref= never married) −0.02 0.014 0.11 0.03 0.016 0.06 0.02 0.025 0.55 0.03 0.023 0.21
Low education (ref=middle) 0.01 0.014 0.61 0.01 0.011 0.37 −0.02 0.017 0.18 −0.00 0.017 0.90
Higher education (ref=middle) −0.01 0.011 0.35 −0.03 0.009 0.0 0.02 0.013 0.08 −0.04 0.014 0.01
Low income (ref=middle) 0.01 0.017 0.45 0.01 0.013 0.30 −0.01 0.024 0.70 0.01 0.022 0.69
High income (ref=middle) 0.00 0.012 0.96 −0.02 0.009 0.02 0.04 0.015 0.02 0.02 0.015 0.21
Household >2 0.00 0.014 0.91 0.02 0.012 0.09 0.03 0.022 0.25 −0.00 0.017 0.96
In paid work −0.03 0.011 0.01 0.00 0.009 0.62 0.01 0.013 0.33 0.04 0.014 0.01
Party: republican (ref=indep) 0.01 0.014 0.47 0.00 0.011 0.99 0.00 0.015 0.88 0.02 0.017 0.36
Party: democrat (ref=indep) −0.01 0.01 0.20 −0.00 0.009 0.84 −0.01 0.013 0.34 0.01 0.014 0.75
Party: other (ref=indep) −0.02 0.018 0.27 −0.01 0.015 0.37 0.00 0.029 0.88 0.01 0.028 0.78
Age/10 −0.01 0.004 0.00 −0.03 0.003 0.00 0.00 0.01 0.71 −0.05 0.005 0.00

n=2,972. ME=marginal effect. s.e.=standard error of ME. p=p-value.

The effects shown are always in comparison to the reference group, which is the ‘always PC’ group.

Of all predictors tested, we find that only a few are predictive of device use in the ALP. Small marginal effects for age are found for the ‘always phone’ and ‘always tablets’ group in comparison to the ‘always PC’ group, while there is no significant difference in age between the ‘always PC’ and ‘mix PC-tablet’ groups.

Panel respondents who complete surveys on their phone are not only younger than the ‘always PC’ group. ‘Always phone’ respondents are also less likely to have a higher education (Bachelor degree or higher), are more likely to be married, and more likely to be of Hispanic or African American ethnicity. These characteristics coincide with some of the most important characteristics of the hard-to-recruit (Tourangeau et al. 2014).

However, while we find that the types of respondents who are hard to recruit into surveys are more likely to answer surveys on their phone, we do not find any differences in reported voting behavior between respondents. ‘Always phone’ respondents are no more or less likely to vote republican or democrat than other respondents.

Discussion

This paper studied the device use of respondents in the American Life Panel. ALP respondents are not necessarily representative of all potential survey respondents in the United States. Although ALP aims to be representative of the U.S. population, nonresponse errors may be introduced at the panel recruitment phase, as well as in every wave of the panel. Still, this papers shows that there is a large proportion of respondents that is ‘mobile-only’ or mixes different devices over time to participate in the panel. This finding is likely to extend to other Internet panel studies, as this finding is in line with earlier findings in the United States and Europe.

We find differences in the sociodemographic background of device use, but not in voting behavior. This means that not offering a mobile-optimized web survey, would exclude specific sociodemographic strata of respondents, but not specific voters. It is important however to note that the results presented here are from multivariate analysis. A univariate analysis with only voting behavior in the 2014 midterm elections as a predictor would have resulted in small, but significant differences. The group that always participates by phone is less likely (ME −0.04) to vote republican. Because we also include age and ethnicity as predictors in our multivariate analysis, this univariate effect disappears in multivariate analysis. Always phone respondents are younger and come from ethnic minority groups. These are strata of people who are generally also less likely to vote republican. This reveals that it is important to recruit and maintain hard-to-recruit respondents as panel respondents. As these respondents are often mobile-only, it is important to offer a good user experience to mobile-only web respondents. RAND, which coordinates the ALP, has implemented a mobile-compatible interface in 2014, but as mobile technology changes constantly, it remains important to study how the web survey experience on mobile devices can be improved.

References

Brick et al. 2007
Brick, J.M., P.D. Brick, S. Dipko, S. Presser, C. Tucker and Y. Yuan. 2007. Cell phone survey feasibility in the U.S.: sampling and calling cell numbers versus landline numbers. Public Opinion Quarterly 71(1): 23–39.
Callegaro 2013
Callegaro, M. 2013. Paradata in web surveys. In: (Frauke Kreuter, ed.) Improving surveys with paradata: analytic uses of process information. Wiley, Hoboken, NJ.
Cook 2014
Cook, W.A. 2014. Is mobile a reliable platform for survey taking? Defining quality in online surveys from mobile respondents. Journal of Advertising Research 54(2): 141–148.
Link et al. 2007
Link, M.W., M.P. Battaglia, M.R. Frankel, L. Osborn and A.H. Mokdad. 2007. Reaching the U.S. cell phone generation: comparison of cell phone survey results with an ongoing landline telephone survey. Public Opinion Quarterly 71(5): 814–839.
Lugtig and Toepoel 2016
Lugtig, P. and V. Toepoel. 2016. The use of PCs, smartphones and tablets in a probability based panel survey. Effects on survey measurement error. Social Science Computer Review 34(1): 78–94.
Mavletova 2013
Mavletova, A. 2013. Data quality in PC and mobile web surveys. Social Science Computer Review 31(6): 725–743.
Mokrzycki et al. 2009
Mokrzycki, M., S. Keeter and C. Kennedy. 2009. Cell-phone-only voters in the 2008 exit poll and implications for future noncoverage bias. Public Opinion Quarterly 73(5): 845–865.
Mood 2009
Mood, C. 2009. Logistic regression: why we cannot do what we think we can do, and what we can do about it. European Sociological Review 26(1): 67–82.
Pew Research 2015
Pew Research. 2015. Device ownership over time. Available at http://www.pewinternet.org/data-trend/mobile/device-ownership/.
Pew Research 2016
Pew Research. 2016. Pew will call 75% cellphones for surveys in 2016. Available at http://www.pewresearch.org/fact-tank/2016/01/05/pew-research-center-will-call-75-cellphones-for-surveys-in-2016/.
R Core Development Team 2016
R Core Development Team. 2016. R: 3.2 A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria.
Struminskaya et al. 2015
Struminskaya, B., K. Weylandt and M. Bosnjak. 2015. The effects of questionnaire completion using mobile devices on data quality. Evidence from a probability-based general population panel. Methods, Data, Analysis 9(2): 261–292.
Toepoel and Lugtig 2014
Toepoel, V. and P. Lugtig. (2014). What happens if you offer a mobile option to your web panel? Evidence from a probability-based panel of Internet users. Social Science Computer Review 32(4): 544–560.
Tourangeau et al. 2014
Tourangeau, R., B. Edwards, T.P. Johnson, K.M. Wolter and N. Bates (eds.). 2014. Hard-to-survey populations. Cambridge University Press, Cambridge.


About Survey Practice Our Global Partners Disclaimer
The Survey Practice content may not be distributed, used, adapted, reproduced, translated or copied for any commercial purpose in any form without prior permission of the publisher. Any use of this e-journal in whole or in part, must include the customary bibliographic citation and its URL.