Introduction
As adoption and usage of smartphones first began to increase, a plausible concern emerged that there may be systematic differences between web survey data collected via smartphones and desktops/laptops. The theory was initially validated with some research that suggested that smartphones’ smaller displays may cut off long questions and may not be ideal for displaying certain question formats. Further, finger-typing on a smartphone keyboard and navigating using smaller tabs and next arrows may inhibit a respondent’s ability to comprehend and complete surveys (Lugtig and Toepoel 2015; Peytchev and Hill 2010). A study published in 2017 found that respondents using smartphones were more prone to report being distracted and being away from home compared to desktop survey takers, though data quality was similar between desktop and smartphone users (Antoun, Couper, and Conrad 2017).
However, research using paradata has found that smartphone and tablet users are actually less distracted than desktop/laptop respondents. For example, researchers in one study used JavaScript to monitor background activities and verify whether survey-takers were doing other activities in between questions (e.g., checking email, visiting other websites) and found that smartphone and tablet users were less prone to stray compared to desktop/laptop survey-takers (Décieux and Sischka 2024).
Other recent investigations have concluded that there are no meaningful differences between responses collected via smartphone or desktop/laptop. Two recent studies that investigated device type choice found no systematic effect on data quality gathered from smartphone and PC respondents (Décieux and Sischka 2024; Clement, Severin-Nielsen, and Shamshiri-Petersen 2020). These studies differentiated themselves from other past studies as they allowed respondents to self-select their device type and they were cross-sectional (or cross-sectional like) rather than panel designs. Past research on this topic has relied on data from survey panels or assigned survey-takers to smartphone, PC, or tablet groups.
Established panels may have selectivity biases and panel members may be distinct from other populations as they are more familiar with survey design and taking surveys generally. And though assignment of device type may avoid self-selection bias, it may create an unnatural situation in which respondents may be forced to use a device they do not prefer or with which they are unfamiliar. Whereas assigning device type may produce poorer survey responses and may result in an overestimation of the size of device effects, allowing participants to freely select the device with which they are most comfortable gives more ideal conditions for respondents to provide high-quality answers and more closely allows for observation of real-world conditions of web-based survey data collection.
Smartphone adoption and familiarity have grown, so it follows that research is needed to ensure continued understanding of device type and data quality. This paper aims to add to the existing literature by seeking to answer two questions:
-
Which device type are residential and nonresidential customers more prone to choosing?
-
Does device type choice relate to data quality?
We theorized that residential customers may be more prone to use smartphones, while nonresidential customers may take surveys more often using desktops or laptops. Regarding data quality, we hypothesized that smartphone survey-takers may spend less time on surveys, straight-line more questions, provide shorter open-ended responses, and may be more likely to provide non-substantive responses.
Methods
This study examines survey data collected online from residential and nonresidential energy efficiency program participants in the Pacific Northwest. We administered monthly surveys to residential and nonresidential electric and gas utility customers who received cash rebates or instant discounts for energy efficient equipment purchases or improvements completed in the previous month. The surveys gathered information on respondents’ satisfaction with the rebate/discount programs and on the influence of the programs on respondents’ equipment upgrade decisions. The surveys were similar for residential and nonresidential participants. The residential survey had up to 25 questions and the nonresidential survey had 21 questions, though the length of each survey was dependent on participants’ characteristics and survey responses. Respondents had the option to take the survey in either Spanish or English.
The study period includes surveys administered over a 14-month period, from August 2023 to September 2024. The surveys had been ongoing for several years but the surveys were not initially programmed to capture device type. The nonresidential survey was programmed to capture device type in time to be launched in August 2023 for July 2023 program participants. The residential survey was not programmed in time to capture device type for July 2023 program participants but was programmed to capture this information in time for the survey of August 2023 participants. The study population for the residential survey was all 11,704 residential customers who participated in the program from August 2023 through August 2024. The study population for the nonresidential survey was all 2,686 nonresidential customers who participated in the program from July 2023 through August 2024.
For both the residential and nonresidential surveys, monthly samples were drawn to achieve multiple completion targets, with each target relating to a type of equipment. Targets were established to achieve a specified quarterly confidence/precision target. The sample for each completion target was drawn randomly from all of the previous month’s program participants who received a rebate or discount for that equipment type. However, low nonresidential program participation meant that the samples for many of the nonresidential equipment types were actually a census of all participants who received a rebate or discount for that equipment type.
We administered the survey first on the web, with follow-up phone calls to non-respondents. At the beginning of the monthly survey, we sent a recruitment email to all sampled participants with valid email addresses. The email included a short recruitment message with a survey web link. The recruitment email offered all residential participants a $10 gift card for completing the survey; nonresidential participants were not offered an incentive for taking the survey. We sent up to two reminders to non-respondents starting approximately one week after the initial contact. Participants who did not respond to the survey within approximately one week after the reminder were then queued for phone follow-up. Customers who did not have a valid email address on file were immediately advanced to the phone-administered survey. To limit the scope of this paper to device type choice, only responses collected via self-administered surveys were reviewed in this paper. Phone call follow-up responses were not reviewed or compared.
Respondent’s screen resolution and operating system were captured by the survey tool, and publicly available device-type specifications and product documentation were used to classify device types. Table 1A in the appendix provides a break-out of device type classifications. The survey was tested on both laptop and smartphone screens. Question formats and user experiences were found to be similar; minimal scrolling or additional effort was required for smartphone survey-takers. The primary difference related to Likert grid-style matrix questions. Grid-style Likert questions in the survey required smartphone survey-takers to click a dropdown menu and then select a score; desktop/laptop respondents were able to see all sub-questions (see Figure 1). The nonresidential survey had up to two and the residential survey had up to three Likert grid-style matrix questions.
The residential and nonresidential surveys had email-web survey response rates of about 26% and 20%, respectively (calculated by percent of unique email respondents over the number of unique email addresses invited). We focus on smartphone and desktop/laptop groups in this study as a small portion of survey-takers were classified as having used a tablet device type to respond to the survey.
Results
To gauge whether device type choice was related to rushing, we reviewed the amount of time spent on the survey for each group, length of open-ended responses, and the tendency of each group to straightline grid or matrix-style questions.
Table 2 displays the average response time on each device type. We excluded response times that were determined to be outliers, because during self-administered surveys, respondents may stop and resume surveys at their discretion. We used the interquartile range (IQR) to determine outliers. At the low end, outliers were defined as response times less than the 25th percentile time minus 1.5 x IQR; at the high end, outliers were defined as response times exceeding the 75th percentile time plus 1.5 x IQR. By this definition, there were no low-end outliers. Within each survey group, the average response times were similar for the different device types.
There was some indication that the relationship between device type and length of open-ended responses was different for the nonresidential and residential surveys. Table 3 displays the average open-ended response length in characters for each survey group. It includes short or non-substantive responses in its calculation of average open-ended response length (e.g., “No,” “None at this time”, “No comment”). We also investigated whether excluding non-substantive write-in responses led to a different result. Removing non-substantive responses led to consistent results for both residential and nonresidential surveys.
Residential respondents who took the survey on a desktop/laptop device tended to have longer open-ended responses overall. There was not a statistically significant difference in average open-ended response length for the nonresidential survey, though generally nonresidential respondents who took the survey on a desktop/laptop device tended to have shorter responses.
The review of open-ended response length may not be an ideal proxy for data quality, as there might be a natural floor for question response length (i.e., program participants may not have much to say on a question). Regardless, this finding beckons for further exploration.
The residential survey contained three and the nonresidential survey contained two grid-style Likert questions. We used these questions to investigate straightlining. These questions investigated satisfaction with multiple aspects of participation and participant decision-making, or the influence of several factors on their decision to participate in the program or make energy efficient improvements. We used the measure of straightlining as a proxy for satisficing.
We considered a question to be straightlined if the same response was given for each sub-question in the grid-style Likert questions. For example, respondents were asked to rate their satisfaction on a scale from 1 (not at all satisfied) to 5 (very satisfied) for up to seven items in one of the residential grid-style Likert scale questions. If the same response was selected for all sub-questions (e.g., 5 for all sub-questions, 0 for all sub-questions), the question’s response was considered to be straightlined. Straightlining did not make a response invalid as none of the grids had logically opposed items.
To compare desktop/laptop and smartphone straightlining we counted the total number of straightlined responses for all respondents and divided by the total number of grid-style Likert questions asked to each survey group. Grid-style Likert questions were limited to those that had at least three subquestions.
The portion of desktop/laptop and smartphone grid-style Likert questions that were straightlined was similar for the desktop/laptop and smartphone survey-takers for both the residential and nonresidential surveys. About 26% of smartphone and 25% of desktop/laptop grid-style Likert questions were straightlined in the nonresidential survey. For the residential survey, about 24% of smartphone and 28% of desktop/laptop grid-style Likert questions were straightlined, respectively. Though neither survey group had a statistically significant difference for straightlined responses from smartphone and desktop/laptop respondents, the higher portion of desktop/laptop responses that were straightlined in the residential survey sparked interest and prompted further investigation.
One possible explanation for the difference was that grid-style Likert questions were displayed differently for smartphone and desktop/laptop respondents. As noted earlier, smartphone users were required to click a dropdown menu and then select a score; desktop/laptop users could see all sub-questions (see Figure 1). Desktop/laptop respondents could therefore have more easily opted to straightline, as that display did not require multiple interactions for each sub-question.
There was no statistically significant difference between device type and the average portion of responses that were non-substantive (see Table 4). A non-substantive response was defined as choosing “Prefer not to answer” or “Don’t know.”
Discussion
Smartphone usage has proliferated and with their growth in adoption, more people are using them to respond to survey requests. Over a decade ago in Survey Practice it was noted that 38% of mobile phone users in the U.S. owned a smartphone (Buskirk and Andres 2012). Ninety-one percent of Americans now own a smartphone (Pew Research Center 2024). In this paper we showed that over half of nonresidential and three-quarters of residential respondents chose to complete survey requests using a smartphone. There is a clear need to continue to investigate device effects on web survey responses.
We did not find convincing evidence that smartphone survey-takers provide lower-quality data. Neither response time nor tendency to provide non-substantive responses were related to device type choice for residential or nonresidential respondents. Further, there was no statistically significant difference between the portion of smartphone and desktop/laptop grid-style matrix questions that were identified as straightlined for either the residential or nonresidential respondents. For nonresidential respondents, there was no statistically significant difference between open-ended response length. We did find one difference between the groups. Residential smartphone respondents tended to provide shorter open-ended responses giving some support to the notion that a smaller typing interface may decrease open-ended response length.
Readers should note that this study was observational. We were not able to randomize the assignment of device type. Survey respondents’ device type preference may relate to other factors (e.g., educational background, income level, age, interest in the survey topic) and thus isolating the impact of device type on data quality is not possible from this study. Further, this study was conducted using data from a monthly survey of residential and business customers who had participated in an energy efficiency program in the Pacific Northwest. It is possible that these groups’ behaviors differ from other populations.
The growing consensus, and finding from this study, is that device type generally does not relate to data quality. This finding is likely related to several developments. As smartphone adoption has increased, people have grown more familiar and comfortable with using them. Survey software companies may have perceived growing engagement through smartphones and focused on ensuring survey designers are able to ensure compatibility and user-friendly design, regardless of device type.
This study examined device type choice and data quality for a different population compared to past studies. While other published research generally used panels of respondents or the general population and were mostly drawn from a European or international context, this study reported on residential and nonresidential energy efficiency program participants in the United States. Another difference is that this study reported on two shorter surveys that investigated participants’ experience with a program.
Though most recent research on this topic has shown no systematic differences between data collected from smartphones and desktops/laptops, survey designers must continue to consider the best ways to ask questions and format instruments. There are a wide array of tools and options available for optimizing online survey data collection instruments. For example, survey collection tools may be able to collect audio responses. This and other question types and design choices should be considered to reduce the need for respondents to type long answers, something that may be more onerous for survey-takers who choose to respond using smartphones.
Respondents’ preferences and familiarity with technology will continue to shift with time, along with releases of new software and device types. Researchers should consider how an instrument will be presented on different devices and how this may impact users’ experiences, and potentially, their responses. Testing surveys on multiple device types and operating systems remains a crucial step to ensure respondents have access to equally vetted and easy-to-use tools.
Lead author contact information
Mike Soszynski
mikesoszynski@gmail.com
Acknowledgements
I would like to express my gratitude to Ryan Bliss for his insight and guidance and thank my other colleagues at ADM Associates for their time, support, and encouragement.