Recruiting Hard-to-Reach Populations: The Utility of Facebook for Recruiting Qualitative In-Depth Interviewees

Marc D. Weiner Edward J. Bloustein School of Planning and Public Policy
Rutgers, The State University of New Jersey

Orin T. Puniello Office of Predictive Analytics
Ketchum Global Research & Analytics

Paul C. Siracusa Edward J. Bloustein School of Planning and Public Policy
Rutgers, The State University of New Jersey

Jocelyn Elise Crowley Edward J. Bloustein School of Planning and Public Policy
Rutgers, The State University of New Jersey

Abstract

We present a cost efficient methodology for using social media to recruit a low-incidence, hard-to-reach population. For qualitative research investigating the policy implications of divorce later in life, we were challenged to recruit a pool of qualified respondents to secure 80 in-depth interview subjects of geographic diversity divided evenly by sex. Although “gray divorce” (i.e., divorce at or over 50 years old) is increasingly common – approximately one in four of all divorces in the United States now occurs within this age group – the incidence rate of individuals in the general population meeting these criteria renders traditional recruitment methods cost-prohibitive. With limited resources and the need for microtargeting, we turned to Facebook: Over four waves totaling 13 days, we recruited 178 qualified respondents from which we successfully selected 40 male and 40 female interviewees while preserving geographic diversity. We specified the characteristics of Facebook users who would be exposed to our solicitation; overall, we generated the respondent pool at a mean cost of $1.18 per respondent, for a total recruitment cost of $210.04. Compared to other recruitment approaches, the speed and cost-efficiency of this methodology was overwhelmingly superior.

Introduction

Social networking facilitates new ways of contacting hard-to-reach populations. Facebook – now ground zero for social media – provides previously unimaginable cost-effective approaches to participant recruitment. Indeed, aware that “social scientists have sought to utilize it for participant recruitment and data collection” (Rife et al. 2016), we turned to Facebook to solve our puzzle-at-hand: Given limited resources and the need for microtargeting, how could we efficiently recruit in-depth interviewees from a hard-to-reach population?

More specifically, for qualitative research investigating the policy implications of gray divorce (divorce at or over 50 years old), we were challenged to recruit a pool of qualified respondents from which to secure 80 interview subjects of geographic diversity divided evenly by sex. Although gray divorce is increasingly common – approximately one in four of all domestic divorces now occurs within this age group – the incidence rate of individuals in the general population meeting these criteria renders traditional recruitment methods cost-prohibitive and/or too slow; indeed, the common theme behind recruitment through advertising in magazines, posted notices, or telephone or mail contacting, is that they are all either too expensive and/or too time-consuming.

With limited resources and the need for microtargeting, we turned to Facebook, where, within 13 days spread out over four waves, we recruited 178 qualified respondents from which we successfully selected 80 interviewees while preserving geographic diversity. We specified the characteristics of Facebook users who would be exposed to our solicitation: geography (United States), age group (50–65+), gender, placement of the advertisement on a mobile device (news feed) and placement on a desktop (news feed or right column of the screen).

Once a respondent clicked on our Facebook ad, she or he was redirected to an online screener we administered directly. Overall, we enrolled 178 qualified respondents at a mean cost of $1.18 per respondent for total recruitment costs of $210.04. Compared to other approaches, the cost-efficiency is exceptional. This article details this fast, cost-efficient methodology for using social media to recruit a low-incidence, hard-to-reach population for qualitative in-depth interviews.

Literature

On Hard-to-Reach Populations

In the late 1940s, Lundberg and Larsen (1949) investigated whether the personal characteristics and views of survey nonrespondents differed from those of respondents. In so doing, they concluded that the problem of nonresponse is widespread, since some individuals are difficult or impossible to reach. Similarly, and beyond probability sampling, in using respondent-drive sampling (RDS) to recruit prostitutes in Manchester, England for interviews about drug use, Faugier and Sargeant (1997) highlighted the benefits of a nonrandom sampling approach when working with hard-to-reach populations. Such a method, they advised, works in the absence of an adequate sampling frame or when respondents are prone to hide their involvement because of a topic’s sensitivity.

Managing cost-effective data collection or recruitment from low-incidence, hard-to-reach populations is a challenge for which instructive lessons can be drawn from literature on other sampling techniques. Discussing RDS under conditions reasonably similar to ours, Teitler et al. (2003) stressed the need to allocate data collection resources efficiently when searching for the hard-to-reach; Link and Burks (2003) demonstrated that tailoring data collection designs when contacting the hard-to-reach can help “increase participation and reduce costs – or at least allow for more efficient use of a fixed pool of financial resources.”

On Recruiting Through Social Networking

With the rise of social networking, researchers are realizing new ways of contacting the hard-to-reach. Facebook’s potential as a medium to locate the hard-to-reach was demonstrated in a 2015 Survey Practice article, “Facebook as a tool for respondent tracing” (Schneider et al. 2015). Remarkably, the authors noted that at the time of their article, over “160 million people in the United States ha[d] Facebook accounts,” and unlike a person’s home address, telephone number or email, those profiles are substantially stable over time (Schneider et al. 2015).

Using Facebook’s huge membership pool, those authors detailed the process of contacting former foster care youth who had been randomly selected for a longitudinal evaluation of a prior intervention. A major challenge of the survey was locating the intended respondent, especially those who had “run away” from home as their 18th birthday approached. After combing public records failed, the researchers searched through Facebook by examining “individual profiles for clues to the Facebook users’ identities,” such as birth date, high school, and physical location. Eventually, the researchers were able to locate and interview 20 of 33 (slightly over 60%) of the respondents searched through Facebook (Schneider et al. 2015).

From simple follow-up field interviews to respondent-driven sampling and differential incentives, methods of contacting the hard-to-reach have evolved and become more efficient. Today, given its enormous following, Facebook offers researchers further opportunities to continue and expand this positive trend by providing previously unimaginable and cost-effective ways of connecting with hard-to-reach populations, whether for direct data collection or, as in our case, recruitment of in-depth interviewees.

Recruitment Protocol and Outcomes

For our qualitative project, we needed to locate and recruit 80 interview respondents, divided evenly between men and women with nationwide geographic dispersion, who experienced divorce at or over the age of 50. Our main concern was that while currently approximately 25% of all divorces in America are gray, the incidence rate is still nonetheless so low in the general national population as to stymie traditional in-depth interviewing recruitment efforts.

Our qualifying criteria for interviewees were as follows:

  1. 50 years of age or older;
  2. divorced, with the divorce (whether a first, second, or later divorce) having taken place at or after the age of 50 or older; and
  3. English-speaking.

These restrictions meant that our targeted respondents were hard-to-identify and hard-to-sample. Two standard methods of reaching this type of population include screening surveys and recruitment advertising. The first method involves using telephone or mail surveys to locate potential respondents. Screener questionnaires are then employed to identify persons hidden within the general population, but who qualify for the study (Teitler et al. 2003; Tourangeau 2014). As previously noted, due to the low incidence of this group in the general population, surveys of any type were simply cost-prohibitive.

We then investigated the possibility of advertising, for example, in The AARP Magazine, but this option was both extremely expensive and would limit exposure to those organizational members. We also researched using public posters to recruit participants at locations across the country; that approach, however, was rendered unmanageable under cost-constrained circumstances by the fairly extensive coordination efforts required to properly locate and maintain those notices. Once again, these measures proved impractical.

We therefore sought out alternative forms of respondent recruitment, and more specifically, began to examine social networking sites as vehicles for identifying our target sample (Bhutta 2012; Rife et al. 2016; Schneider et al. 2015). Facebook, the obvious first choice, appeared promising in that we could display our recruitment notice with relative efficiency to those users in our desired geographic regions who met the age requirement for having experienced a gray divorce.

Indeed, the outcome well-exceeded our expectations: Over four waves that together totaled 13 days, we were contacted by 178 presumptively qualified respondents at an average cost of $1.18 per respondent, for a total recruitment cost of $210.04. Broken down in further detail, this total cost reflects the number of times the study advertisement is “clicked” multiplied by the “cost per click.” The “cost per click” varies based on the selection criteria we used to locate our sample. In this study, it ranged from $0.21 per click for the first wave of data collection from both sexes, to $0.09 per click for the third wave of men only. Summary statistics indicate that over the course of this study, our advertisement was presented to 25,183 Facebook users who clicked a total of 1,325 times at a mean cost of $0.15852 per click (resulting in the total cost of $210.04).

Apparently in order to maintain a particular aesthetic, Facebook places strict guidelines on the use of images in all advertisements (Figure 1). The key restriction is that any image may only be 20% text, and Facebook provides a grid tool to test potential advertisements for the text-image proportions. For academic institutions or well-known products, this can be a limitation because of the brand equity of the institutional or commercial logo. We therefore developed two alternatives, one that is a “tight shot” of the informal Rutgers University seal (Figure 2), and another that is a picture of a Rutgers University monument sign on the flagship New Brunswick Campus (Figure 3). In both instances, these two images allowed us to represent Rutgers’ identity without violating Facebook’s rules.

Figure 1  Facebook text rules.


SP-Vol-10_Weiner_fig1.jpg


Figure 2  Recruitment advertisement: informal university seal.


SP-Vol-10_Weiner_fig2.jpg


Figure 3  Recruitment advertisement: university monument sign.


SP-Vol-10_Weiner_fig3.jpg


The institutional seal-as-visual study advertisement was simple; the title indicated the principal investigator’s institutional affiliation (Edward J. Bloustein School of Planning and Public Policy, Rutgers University), and showed the Bloustein School logo and date. The ad displayed the Rutgers University shield as the central visual element reading, above the visual, “Rutgers Professor seeks volunteers for study on divorce experienced at age 50 or older” and below the visual, “If you had a divorce at or after 50 years old, and would like to participate in this interesting academic study, please click the link to learn more about the project” (Figure 2). Similarly, the monument sign study advertisement contained the exact same narrative, only with a classic Rutgers University monument sign substituted as the central visual element (Figure 3).

If a potential respondent clicked through the advertisement, she or he saw the following text on the first page of the online screener questionnaire:

  • My name is Jocelyn Elise Crowley, and I am a Professor of Public Policy at the Edward J. Bloustein School of Planning and Public Policy at Rutgers, The State University of New Jersey. Thank you for your interest in participating in this study. I am studying the experiences of women and men who divorced at or after the age of 50. To do so, I am asking for volunteers, who divorced at or after the age of 50, to participate with me in a one-hour telephone interview. This project is fully confidential, and has been approved by the Rutgers University Institutional Review Board. Confidential means you will not be identified in any resulting publications or reports that come from my research and that no one will ever know that you participated. However, to schedule your interview at a convenient time for you, I will need to contact you. So that I can do so, kindly provide the following information.

To screen for study qualification, potential respondents were then asked, “Did you divorce when you were 50 years of age or older?” Those who answered negatively were thanked for their interest, advised they did not meet the criteria for participation, and exited from the process. From those answering affirmatively, we collected their self-reported first and last names, sex, preferred email address, telephone numbers, and preferences regarding time and day of the week for the one-hour interview. They were then advised that the researcher would be contacting them over the next few weeks to schedule an interview.

After performing a second, confirmatory screening review, and taking into account dropouts and/or other failures to qualify, 167 of the original 178 presumptively qualified respondents constituted the final recruited sample. We then interviewed 80 of these qualified respondents.

Between July 23, 2014, and February 9, 2015, we collected four waves of data; we used waves for two primary reasons: First, we could constrain the number of “fresh” contacts at any given time so that the primary investigator could contact and interview respondents in a timely fashion. Second, we initially believed that we only needed 60 interviews, 30 with men and 30 with women. However, after that number of interviews, we had not reached data saturation, i.e., we were still learning new content from our respondents about their divorce experiences (Hennink et al. 2011). We therefore conducted two more waves of data collection, of which wave 3 was men only, and wave 4 women only, bringing us to a total of 80 interviews (40 of each sex).

Table 1 describes the disposition of each recruitment wave. Moreover, since Facebook reaches all across the United States (if not the globe), we were able to reach qualifying respondents across a wide variety of locations (Figure 4).

Table 1 Outcomes of Facebook recruiting effort.

Wave 1
7/23 to
7/28/14
Wave 2
8/19 to
8/21/14
Wave 3
1/5 to
1/6/15
Wave 4
2/6 to
2/7/15
Totals
Male Female Male Female Male only Female only Male Female
Names collected 21 38 6 34 46 22 73 94
Interviews completed 12 21 5 9 23 10 40 40
Incomplete: not qualified 2 1 2 0 5
Incomplete: no response 20 8 13 4 45
Incomplete: no show 1 0 1 1 3
Incomplete: declined 3 0 0 1 4
Incomplete: not available 0 0 1 0 1
Incomplete: bad contact info 0 0 1 0 1
Incomplete: excess, unused 0 17 females 5 males 6 females 5 23

Figure 4  Geographic distribution of qualified Facebook recruited respondents.


SP-Vol-10_Weiner_fig4.jpg


Limitations and Extensions

This protocol is highly adaptable to a wide variety of recruitment circumstances beyond in-depth interviewees, including focus group recruitment, message and/or question pretesting, and nonprobability surveys. While the implications for researchers seeking hard-to-reach/hard-to-contact populations are profound, the approach is not without limitations.

One potential limitation is that Facebook allows respondents to engage with our advertisement content by sharing the recruitment notice with “Facebook friends” and connections online. While this may be favorable, resulting a second-level de facto form of respondent-driven sampling, it may also be undesirable in that some intra-Facebook-user communications may unfavorable to, or may undermine the purpose of, the advertisement. While it does not appear that this limitation affected our effort, it is worth noting.

Another, more specific limitation is raised by the “basic question about the effect of recruitment process on … representativeness” (Belson 1960). While our qualitative study made no probability-sample-backed effort to generalize to the national population of gray divorcees, we were concerned with the potential for recruitment bias. Our simple-but-effective effort to address recruitment bias centered on gathering more than twice the required number of interviews and, within the larger set of recruited potential respondents, balancing the interviewed sample, as best as possible, on our selection criteria. Still, and with Belson’s caution in mind, recruitment bias is the most prominent limitation of this approach, taking two forms. The first is akin to sampling population undercoverage, i.e., that non-Facebook users cannot be contacted through this method; the second, endemic in virtually all sampling approaches, is self-selection bias: to be sure, “self-selection bias is possible even if the research has used random sampling techniques” (Costigan and Cox 2001).

To illustrate the undercoverage issue, out of the general U.S. population as of January 2017 there were 32.3 million Facebook users in the 45–54 year old age range; 14.1 in the 55–64 range, and 20.2 in the 65 and over range, aggregating to 66.6 million Facebook users in the 45 and over group (Statista 2017). The Census Bureau projected that group, at July 2017, to include 142.7 million persons1 (U.S. Census 2014); the implication is that approximately 53% of that age group could not have been reached through Facebook-based recruiting alone.

Another potential limitation is self-selection bias, which is typically a function of a relationship between the subject matter of a study (as reflected in the content of the recruitment approach) and the interests and motivations of a potential respondent. It is a core principle of sampling and bias analysis that if “selection is correlated with respondent characteristics in variables important to the study, it can undermine the representative nature and bias the sample on which estimates are based” (Shapiro-Luft and Capella 2013). Moreover, “the sample selection process under nonprobability sampling is subjective, which does not guarantee the elimination of selection biases” (Lee 2009).

Given the research design and core questions for our study, we were not concerned about this potential bias, largely because the subject matter was not related to the propensity to be a Facebook user (Dennis 2010). However, other qualitative studies, particularly those involving Internet and social media use, sociability, communication propensity, and the like, are advised to explore the possibility of self-selection bias at the time they design their recruitment protocol.

One suggestion to reduce Facebook sampling self-selection bias, which would not have been practical in our hard-to-reach and hard-to-contact circumstances, is to omit the selection criteria from the recruitment notice (Springer et al. 2016). Still, we note that prior research studying divorcing couples, albeit not based on Internet recruiting, found that men who participate “are more invested in their fathering role” (Costigan and Cox (2001) citing Boyd (1985); for a quantitative perspective, see “Assessing and compensating for self-selection bias (non-representativeness) of the family research sample,” Braver and Bay (1992)). Applying this observation to our study suggests that those with more extreme experiences – both positive and negative – may have had a greater propensity to respond to the recruitment notice. This concern was addressed in our project’s substantive analysis by balancing the use of interviews from respondents with different gray divorce experiences (Crowley forthcoming, 2018).

This recruitment protocol need not be a replacement alternative to traditional sampling approaches but could be deployed as part of a dual frame to reduce the costs of traditional methods, such as telephone recruitment; moreover, traditional methods may still be needed to ensure a representative sample on the variables of interest.2 Such efforts may take many forms, including combining recruitment with outreach to community groups, with print advertising in media of interest to the hard-to-reach/hard-to-contact population, as well as with telephone recruitment based on zip codes in which the target hard-to-reach/hard-to-contact population is concentrated (Willis 2015).

Qualitative researchers would be well advised, particularly in nonprobability sampling, to study the impact of the limitations we have noted. Useful additional research could include comparisons of probability-based to nonprobability sampling methods within the same research protocol, coding assessment of substantive outcomes based on different recruitment methods for the same research question, and similar comparative techniques that vary recruitment approaches while controlling for other aspects of the study.

Conclusion

We have detailed an easy and cost-efficient approach to recruiting for qualitative in-depth interviewees from a hard-to-reach and hard-to-contact population, which easily permits stratification by geography and gender. While this recruitment method is not immune to the potential limitations inherent in all sampling approaches, social media recruiting in general – and Facebook in particular – is applicable to a wide variety of research contexts and represents nothing less than a change in ease, speed, and economic efficiency in this practice area.

Acknowledgements

This work and the underlying qualitative research project were supported by Rutgers University’s Edward J. Bloustein School of Planning and Public Policy’s Michael J. and Susan Angelides Public Policy Research Fund. M.B. Crowley, M. Patrick Simon, and K. T. G. Weiner-Simon provided ongoing support and encouragement.

References

Belson 1960
Belson, W.A. 1960. Volunteer bias in test-room groups. Public Opinion Quarterly 24(1): 115–126.
Bhutta 2012
Bhutta, C.B. 2012. Not by the book: Facebook as a sampling frame. Sociological Methods and Research 41(1): 57–88.
Boyd 1985
Boyd, S.T. 1985. Study of the father. American Behavioral Scientist 29(1): 112–128.
Braver and Bay 1992
Braver, S.L. and R.C. Bay. 1992. Assessing and compensating for self-selection bias (non-representativeness) of the family research sample. Journal of Marriage and Family 54(4): 925–939.
Costigan and Cox 2001
Costigan, C.L. and M.J. Cox. 2001. Fathers’ participation in family research: is there a self-selection bias? Journal of Family Psychology 15(4): 706–720.
Crowley forthcoming 2018
Crowley, J.E. forthcoming, 2018. Gray divorce: what we lose and gain from mid-life splits. Berkeley, CA, University of California Press.
Dennis 2010
Dennis, J.M. 2010. KnowledgePanel: processes & procedures contributing to sample representativeness & tests for self-selection bias. Knowledge Networks Research Note. Available at http://www.knowledgenetworks.com/ganp/docs/KnowledgePanelR-Statistical-Methods-Note.pdf.
Faugier and Sargeant 1997
Faugier, J. and M. Sargeant. 1997. Sampling hard to reach populations. Journal of Advanced Nursing 26(4): 790–797.
Hennink et al. 2011
Hennink, M., I. Hutter and A. Bailey. 2011. Qualitative research methods. Sage, Thousand Oaks, CA.
Lee 2009
Lee, S. 2009. Understanding respondent driven sampling from a total survey error perspective. Survey Practice 2(6): 1–6.
Link and Burks 2013
Link, M.W. and A.T. Burks. 2013. Leveraging auxiliary data, differential incentives, and survey mode to target hard-to-reach groups in an address-based sample design. Public Opinion Quarterly 77(3): 696–713.
Lundberg and Larsen 1949
Lundberg, G.A. and O. Larsen. 1949. Characteristics of hard-to-reach individuals in field surveys. Public Opinion Quarterly 13(3): 487–494.
Rife et al. 2016
Rife, S.C., K.L. Cate, M. Kosinski and D. Stillwell. 2016. Participant recruitment and data collection through Facebook: the role of personality factors. International Journal of Social Research Methodology 19(1): 69–83.
Schneider et al. 2015
Schneider, S., A. Burke-Garcia and G. Thomas. 2015. Facebook as a tool for respondent tracing. Survey Practice 8(1): 1–6.
Shapiro-Luft and Cappella 2013
Shapiro-Luft, D. and J.N. Cappella. 2013. Video content in web surveys: effects on selection bias and validity. Public Opinion Quarterly 77(4):
936–961.
Springer et al. 2016
Springer, V.A., P.J. Martini, S.C. Lindsey and I.S. Vezich. 2016. Practice-based considerations for using multi-stage survey design to reach special populations on Amazon’s Mechanical Turk. Survey Practice 9(5): 1–8.
Statista: The Statistics Portal 2017
Statista: The Statistics Portal. 2017. Facebook users by age in the U.S. 2017. Available at https://www.statista.com/statistics/398136/us-facebook-user-age-groups/.
Teitler et al. 2003
Teitler, J.O., N.E. Reichman and S. Sprachman. 2003. Costs and benefits of improving response rates for a hard-to-reach population. Public Opinion Quarterly 67(1): 126–138.
Tourangeau 2014
Tourangeau, R. 2014. Defining hard-to-survey populations. In: (R. Tourangeau, B. Edwards, T.P. Johnson, K.M. Wolter and N. Bates, eds.) Hard-to-survey populations. Cambridge University Press, Cambridge, UK.
U.S. Census Bureau, Population Division 2014
U.S. Census Bureau, Population Division. 2014. U.S. Census: Table 3. Projections of the population by sex and selected age groups for the United States: 2015 to 2060” (NP2014-T3).
Willis 2015
Willis, G.B. 2015. Research synthesis: the practice of cross-cultural cognitive interviewing. Public Opinion Quarterly 79(S1): 359–395.
Footnotes
1 The July 1, 2017 data point is estimated as the mean of the resident ­populations for both sexes for July 1, 2015 and July 1, 2020.
2 We are grateful to an anonymous reviewer for these points.


About Survey Practice Our Global Partners Disclaimer
The Survey Practice content may not be distributed, used, adapted, reproduced, translated or copied for any commercial purpose in any form without prior permission of the publisher. Any use of this e-journal in whole or in part, must include the customary bibliographic citation and its URL.