Effects of Push-To-Web Mixed Mode Approaches on Survey Response Rates: Evidence from a Randomized Experiment in Emergency Departments

Introduction

Measuring and reporting patient experience with health care through surveys provides useful information to guide consumer decision-making when selecting a provider and contributes to the evaluation of provider performance. For example, results from the Hospital Consumer Assessment of Healthcare Providers and Systems (CMS 2018a) Survey of patients’ experiences of hospital care are publicly reported as measures and star ratings and used in hospital payment (CMS 2018d; Elliott et al. 2016; Giordano et al. 2010). Patient experience surveys also inform publicly reported information about the quality of hospice care and the care received by Medicare beneficiaries who belong to Accountable Care Organizations (CMS 2018a; Hospice 2018).

Given that these survey results inform evaluations of provider performance, it is essential that they accurately reflect the experiences of the patient population by having a reasonably high survey response rate, obtaining a representative sample of respondents, and utilizing statistical methods to ensure representative and unbiased results. In this paper, we examine a patient experience survey of hospital-based emergency department (ED) patients discharged to home after their ED visit (rather than admitted to the hospital), referred to as discharged-to-community (Parast et al. 2019) patients. As previous studies have found very low survey response rates among such patients, less than 20% (Mathews et al. 2019; Weinick et al. 2014), we sought to understand how response rates could be improved in this population.

Previous work in the ED DTC population has shown that single-mode protocols, such as mail-only or web-only, and distribution within the ED do not perform well (1–14% response rate), but sequential mixed-mode approaches may have promise (Mathews et al. 2019; Parast et al. 2019). In this paper, we describe results from a randomized test of nine experimental survey administration protocols, eight of which began with an invitation to a web survey, then followed by a mailed survey, a telephone survey or both. The primary goals of this experiment were to (1) determine whether any of the experimental protocol arms with a web survey component could obtain a higher response rate than a two-stage non-web sequential mixed mode protocol (mail followed by telephone) and (2) identify protocol components (methods of survey invitation) that were most beneficial in terms of improving the likelihood of response.

Methods

Sample and Survey Administration

The study was designed as a randomized experiment in which randomization of patients to survey arms and the sample size targets were designed to achieve adequate power to detect small differences in response rates when comparing mode protocols. A total of 16 large, geographically dispersed hospitals participated in the experiment. To be eligible, a hospital had to have adequate volume (14,000 or more annual ED visits), routinely collect patient email addresses, and have a process to obtain and document consent to contact patients via text message. Hospitals participated voluntarily and did not receive any compensation. Adult (18 years and older) patients were eligible for the study except those who (1) were admitted to the hospital as a result of the ED visit; (2) left the ED without being seen; (3) requested not to be contacted (“no publicity” patients); (4) were excluded due to state regulations; (5) had a primary diagnosis of mental health or substance use (International Statistical Classification of Diseases and Related Health Problems [ICD]-10 codes F01–F99); (6) were transferred to another hospital; (7) were discharged anywhere other than home (e.g., to a skilled nursing facility); (8) had a foreign home address; (9) were court/law enforcement patients (prisoners); (10) spoke a language other than English; or (11) had died in the ED or prior to sampling. Exclusions (1)–(4) were implemented by each participating hospital; exclusions (5)–(11) were implemented by the study team and are listed in order of decreasing frequency.

A total of 26,991 ED DTC adult patients discharged between January 2018 and March 2018 from the 16 hospitals were sampled by the study team and randomized to nine protocol arms within each hospital. Table 1 provides a detailed description of protocol components and timing information for the nine protocol arms. Documented consent to contact patients by text was required for this study. Some hospitals included this consent in their general consent to treat documentation, others employed a separate consent form. Absent consent, patients were not contacted by text. Data collection began between 11 and 64 days (average of 31 days) after discharge, and all survey outreach efforts ended 42 days after the first contact attempt.

The study survey instrument contained 38 questions and is available online (CMS 2018a). A completed survey was defined as one in which at least 50% of the items applicable to all patients^[1] (21 items) were answered. This study was approved by the RAND Institutional Review Board.

Statistical Analysis

We examined response rates using the American Association for Public Opinion Research Response Rate 1 definition (Kaplowitz et al. 2012). To compare protocol pairings, we used logistic regression in which the outcome was individual-level response (0/1) with an indicator for one survey arm within the pair. The two arms being compared were typically different in a single way, such as the number or type of reminders, or a difference in timing. We also examined the response rate by each completion mode and compared characteristics of respondents by completion mode. Throughout, all regression models were adjusted for patient age, gender, Emergency Severity Index (ESI)^[2], primary diagnosis^[3], sampling batch^[4], and indicators for hospital.

To examine the potential for nonresponse bias, we conducted a nonresponse analysis by fitting a logistic regression predicting response (0/1) from available characteristics, tested for differences in nonresponse results by arm by adding an interaction between patient characteristic and arm indicators, and compared respondent characteristics by arm.

Finally, we examined whether patient characteristics were associated with the ability to contact a patient by email or text, using age, gender, ESI, primary diagnosis, sampling batch, and hospital as predictors in a logistic regression. We defined “being able to contact patient by email” as the patient having a non-missing email address provided in the contact information, and consenting to email contact (if required by the hospital). Similarly, for text, we defined “being able to contact patient by text” as the patient having a mobile telephone number provided in the contact information, and consenting to text contact (required by all hospitals).

Results

Table 2 presents response rate results; the overall response rate was 18.6%. The response rate in Arm 5 (Email (x4) + Mail + Telephone) was highest (27.3%) but was not significantly higher than Arm 1, standard mixed mode (25.5%). The response rates for all other arms were significantly lower than standard mixed mode.

Table 3 presents logistic regression results for each of the eight selected comparisons. For each of these protocol comparisons, in addition to differences in the protocol components (e.g., email only vs. email and text), there were also differences in the timing of outreach efforts and differences in the patient population that could be reached using the different protocol components (e.g., mailed survey vs. email). These results show that the following factors increased the likelihood/odds of response: (1) using telephone in place of a second mailing (odds ratio [OR] =2.41), (2) using a text invitation and a second mailed survey instead of a paper invitation (OR=1.34), (3) adding a mailed survey to a web-phone protocol (OR=1.30), (4) adding a second mailing instead of having a third reminder email (OR=1.25), and (5) using a combination of text invitation/reminder and email reminders instead of only email invitation/reminders (OR=1.25).

Table 4 presents the percentage of patients responding by each mode by arm. The reference mode here was Arm 5 (Email [x4] + Mail + Telephone) because it was the only arm with all three survey completion modes. Among arms that included telephone, 14.1%–18.1% of patients responded by telephone; for arms with mail, 7.0%–10.5% of patients responded by mail; and for arms with web, 4.8%–7.5% responded by web. In all arms that included telephone (which was the terminal mode), the majority of responses were by telephone. Overall, the most productive mode in terms of obtaining completed surveys was telephone, then mail, then web. The web mode consistently resulted in the lowest percentage of responses, though Arms 7 and 8 (which also included text messages) obtained a significantly higher proportion of responses by web (>7%) than did Arm 5. These results also reveal that Arm 5 (Email [x4] + Mail + Telephone) had significantly fewer mail respondents and telephone respondents compared to other arms yet had the highest response rate, illustrating that use of an initial web mode has the potential to reduce costs associated with mail and telephone contact.

Table 5 presents respondent characteristics within each completion mode. All characteristics had significant differences in distributions across completion modes. The age distribution of telephone respondents was most similar to the sampled population. Compared to mail respondents, web respondents tended to be younger and more closely reflect the sampled eligible population. For example, while 38.5% of the eligible sampled population were 18–34 years old, only 11.0% of mail respondents were in this age range, as were 18.7% of web respondents and 30.8% of telephone respondents. Thus, among 18–34-year-old patients, the telephone mode most closely reflects the sampled population (although still lower than the true percentage), while the mail mode does not reflect the true percentage, and web mode is in between. In addition, the telephone mode obtained more patients with poorer health and lower education, as well as more racial-ethnic minorities, compared to mail and web mode.

Our nonresponse analyses showed that the likelihood of responding was significantly higher for older patients and for patients with certain primary diagnoses (such as diseases of the nervous system and sense organs, diseases of the circulatory system, and injury and poisoning, relative to patients with a primary diagnosis of infectious and parasitic diseases) and significantly lower for male patients and for low-urgency patients (results not shown). The association between response likelihood and age varied significantly by arm such that older patients were even more likely to respond in arms without a telephone component (results not shown). This result is also reflected in Appendix Table 1, which compares respondent characteristics by arm; arms with telephone have a higher percentage of younger patients responding and more closely reflect the sampled population.

Overall, across all eligible patients in this study, 30.1% could be contacted by email, and 59.3% could be contacted by text. Appendix Table 2 presents logistic regression results examining the likelihood of being able to be contacted by email and by text by demographic characteristics. For both email and text, the ability to contact patients varied greatly by hospital (range of 0.4%–59.7% and median of 28.4% for email; range of 1.3%–85.6% and median of 77.2% for text). Patients aged 25–74 were significantly more likely to be able to be contacted by e-mail compared to the youngest patients (18–24 years old; OR range 1.20–1.43, p<0.01), while patients 85 and older were significantly less likely to be able to be contacted by email (OR=0.54, p<0.001). For text, compared to patients 18–24 years old, patients 45 and older were significantly less likely to be able to be contacted by text (OR range 0.11–0.77, p<0.001). For both email and text, males were significantly less likely to be able to be contacted (OR=0.57, p<0.001 for email; OR=0.83, p<0.001 for text).

Discussion

Results from this study, in which ED DTC patients were randomly assigned to one of nine survey protocols, further demonstrate the difficulty in obtaining even moderate response rates in an ED DTC population. The overall response rate in this study was 18.6%, compared to national average response rates of 27% and 29% for the HCAHPS and Home Health CAHPS surveys, respectively (CMS 2018c, 2018e). However, this low response rate is not surprising given that the ED population is younger, more mobile, and tends to provide incomplete or poor-quality contact information (Elliott et al. 2009; Klein et al. 2011; Weinick et al. 2014). While the testing of eight web-based protocols is instructive, more work is needed to identify ways to improve response rates in this population.

Contact via text message is a promising method of invitation to a web survey, given the finding that more patients could be contacted by text than by email and that text improved response rates over email-only invitation. However, contacting patients via text incurs administrative and legal complications because the Telephone Consumer Protection Act of 1991 requires prior consent to contact via text. The extent to which the hospital-initiated patient consent protocol employed in this study would be feasible among hospitals nationally is not known.

On the other hand, results clearly demonstrated the benefits of including a telephone mode as part of the survey protocol. Telephone mode captures respondents who are not well represented in web and mail modes, including patients who are younger, in poorer health, with less education, and belong to racial/ethnic minorities. Telephone mode increases representativeness and response rates when added to other modes, as found in other health survey settings (Klein et al. 2011). Our results suggest that future protocols for this survey should include a telephone component. Our nonresponse analysis reinforces this conclusion given that differences between the respondent population and sampled population were more extreme using protocols without telephone. While hospital scores calculated using survey results are likely to employ case-mix adjustment to account for some respondent characteristics, there is still interest in having response rates as high and representative of the population as possible (Klein et al. 2011). Mode protocols that include telephone are more likely to achieve this goal than mode protocols without a telephone component.

This study demonstrated that patients who can be reached by email or text differ with respect to age and gender from those who cannot, which is consistent with previous work on this topic (Fowler Jr, et al. 2019). And while a web-only survey using an email and/or text invitation would poorly represent the emergency department patient population, the results presented here reveal that these modes of invitation do enhance contact with a hard-to-reach younger population—but should be followed by mail and/or telephone survey attempts.

Our study was limited to sixteen hospitals that met specific criteria, including willingness to voluntarily participate and to collect and document consent for contact via text. However, the hospitals were geographically diverse, and most importantly, the randomization of patients to modes within hospitals ensures internal validity of comparisons between the experimental arms (Klein et al. 2011). Particularly with respect to results on web survey response and contact via email and text, results are likely to vary depending on the percentage and characteristics of patients within an ED who provide consent to contact and provide email and mobile telephone contact information.

Overall, this study demonstrates that sequential multi-mode approaches involving web, mail, and telephone with multiple methods of web survey invitation will likely obtain the highest response rates in the ED population. Further work is needed to improve response rates. Future research might explore embedding the first survey item in the emailed/texted web survey invitation (Liu and Inchausti 2017), streamlining the introductory screen language, and varying the email/text subject line (Kaplowitz et al. 2012; Trouteaud 2004).

Appendices