The Performance of Vignettes in Focus Groups and Cognitive Interviews in a Cross-Cultural Context

Introduction

Research has shown that hypothetical vignette situations need to be culturally appropriate and that vignettes may be an unfamiliar task for speakers of languages other than English, including Spanish, Chinese, Korean, and Vietnamese (Goerman and Clifton 2011; Sha 2016). A written vignette can clarify sensitive questions because respondents may more openly express opinions and can be useful when survey questions pertain to populations that are difficult to recruit (Brondani et al. 2008; Willis 2015). Therefore, vignettes may be a cost-saving way to test survey questions for rare situations, particularly for multilingual studies as recruiting non-English speakers tends to be more challenging. However, researchers need to consider whether vignettes work equally well in different research settings. We extend this literature by systematically examining results from the same vignette testing a homeownership question in seven different languages. We also examine the performance of this vignette in different research contexts, comparing when vignettes are embedded within cognitive interviews versus focus groups.

In this study, we address the following broad questions:

• Do we uncover the same problems respondents have with a survey question with vignettes administered in focus groups compared to cognitive interviews across multiple languages?
• Is administering vignettes in cognitive interviews and/or focus groups effective for uncovering problems with a survey question?

Although both cognitive interviews and focus groups are used to uncover issues with and improve survey questions, these methods have different strengths. Focus groups are better suited to revealing how a group thinks about concepts collectively, with the opinions and ideas of one respondent building on those of another to form a larger picture. Cognitive interviews are used to understand individual respondents’ reactions to a survey question. Focus groups may be more appealing to researchers because they can be accomplished more quickly and cost effectively than cognitive interviews (see Table 1). However, respondents’ reactions and responses to vignettes may be contingent on the language and the methodological setting in which vignettes are administered.

In this study, we examine both methodology and language simultaneously by testing a problematic survey question about homeownership with a vignette rather than by recruiting respondents that represent a variety of homeownership statuses, some of which may be uncommon or hard to identify. We hypothesize that vignettes administered during cognitive interviews will be more effective at uncovering problems with a survey question compared to vignettes administered during a focus group for two reasons. First, cognitive interviews typically produce more detail from each respondent, and second, social desirability may play a greater role in focus groups. Finally, although this may be true in English to a certain extent, we hypothesize that this effect would be more pronounced in other languages due to linguistic and cultural norms regarding disagreement (Hall 1976).

Methods

Data

Data for this study are drawn from a research study conducted by the U.S. Census Bureau to pretest the decennial Census questionnaire and materials in seven languages: Arabic, Chinese, English, Korean, Russian, Spanish, and Vietnamese. The Census questionnaire was designed to be administered at the household level and included questions regarding demographic characteristics and occupancy status. The vignette was developed in English first and then translated using a team-based approach into the target languages and was designed to test a survey question that previous cognitive testing in Spanish had shown to be problematic.

Cognitive interviews were conducted with a total of 100 respondents distributed across the seven languages (see Table 2). Interviewers first administered the entire Census questionnaire to respondents, then retrospectively probed about questions and administered the vignette. Additionally, 30 focus groups were conducted with 274 other respondents. As shown in Table 2, there were four focus groups in most languages and six in Spanish, with 7–12 respondents per group. The focus groups included discussion of decennial Census materials (e.g., a multilingual brochure) and interviewer introductory messages. Except Russian and Spanish focus groups, most focus group respondents only discussed the Census questionnaire during the vignette. The vignettes were administered at the end of these focus groups in order to learn more about how the results compared to similar testing in the cognitive interviews. The focus groups and cognitive interviews had respondents representing diverse demographic characteristics.

The cognitive interviewers and focus group moderators wrote summary reports that provided detailed descriptions and direct quotes, which the research team used as the data source.

One limitation of this study is that the sample sizes varied by language and were relatively small. There was also variation in the level of detail found in cognitive interview and focus group summaries across interviewers despite using a shared summary template. Finally, cognitive interview respondents answered a survey that contained the same question found in the vignette, while focus group respondents saw the survey question for the first time during the vignette exercise.

Survey Question and Vignette

The vignette scenario stated, “Mrs. Smith is responding to the census questionnaire for her household. She lives with her son, who pays the mortgage, but she doesn’t pay anything. How should she respond?” The survey question following the scenario had a dual goal of collecting information on the ownership status and payment status of the housing unit (see Figure 1).

Figure 1  Vignette survey question.

Prior testing has shown that respondents tend to be confused about whether the focal point of the question is an individual person or the housing unit, and may not understand terms like “occupied without payment of rent.” To test this question, respondents who do not pay anything for their housing but who should not choose the “occupied without payment of rent” category (e.g., those living with a householder who pays) would need to be interviewed. Using a vignette allowed us to test respondents’ understanding of the scenario without recruiting people who met various homeownership/payment criteria.

During cognitive interviews, respondents were read the vignette aloud and asked to answer it and provide their rationale for the answer given. In focus groups, respondents were given a handout with the vignette written on it and were asked to select a response option in writing. The group then discussed the reasoning for their responses.

Coding Scheme

Two researchers coded all interview summaries and focus group summaries. For cognitive interview summaries, the coders captured the response option that was chosen and the verbatim answer with the reason for choosing that response option. For focus group summaries, the coders captured the number of respondents who chose each answer option, as well as the rationale that was reported for choosing each option. This information was based on the responses that the respondents verbally reported during the discussion as recorded in the summary. If respondents gave no answer during discussion or reported being unable to pick between two or more answers, the response was counted as “other.”

Intercoder Reliability

The research team conducted a test of intercoder reliability by randomly selecting one cognitive interview summary and one focus group summary from each language (N=14), with coders independently coding the summaries. Agreement on a code was scored as 1, disagreement as 0. The intercoder agreement rate was calculated for six codes for cognitive interview summaries and eight codes for focus group summaries. Agreement occurred for 98% of all codes. The dataset was analyzed using descriptive statistics, complemented by a qualitative analysis of responses.

Results

Figure 2 compares the responses given by respondents in all languages to the vignette in cognitive interviews and focus groups.

Figure 2  Comparing cognitive interview (CI) and focus group (FG) responses in all languages.

Since Mrs. Smith lives with her son who pays the mortgage, the correct response was “owned with a mortgage,” which was given more often by respondents in focus groups than in cognitive interviews (67% and 61% of responses, respectively). Qualitative evidence indicates that some respondents in focus groups changed their answers to “owned with a mortgage” after listening to opinion leaders in the group discussions, but quantifying the prevalence of changed answers was not possible using the data available to us.

“Occupied without payment of rent” was the most common incorrect answer chosen by respondents, and it was endorsed by respondents more often in cognitive interviews than in focus groups (28% and 22% of responses, respectively).

The proportion of respondents who chose the correct answer (“owned with a mortgage”) varied by language, as shown in Figure 3.

Figure 3  Response of “owned with a mortgage” by language.

Spanish- and Arabic-speakers showed a wide disparity between the results of administering the vignette in cognitive interviews and focus groups. Spanish-speaking respondents selected “owned with a mortgage” about three times more often during the focus groups (67%) than the cognitive interviews (20%). Among Arabic-speaking respondents this disparity is present, although less pronounced compared to Spanish-speakers. During the focus groups, 86% of Arabic-speaking respondents selected the option “owned with a mortgage” compared to 53% during cognitive interviews. In contrast, English-speakers were more likely to select the correct answer in cognitive interviews (93%) than in focus groups (79%). In the rest of the languages, the percentage selecting “owned with a mortgage” in a cognitive interview compared to a focus group only varied between 1% and 7%.

Vietnamese-speakers selected the correct answer relatively less often than speakers of other languages regardless of the methodology. Qualitative findings indicate that the subject matter of the survey question was more controversial for Vietnamese respondents, and some objected to the personal nature of asking about homeownership and payment, as demonstrated in the following excerpt from a cognitive interview summary:

• [The respondent] thought [the ‘occupied without payment of rent’] option is too personal. Pay[ing] or not paying is an individual and very private decision. Census should not care about who pay[s] or not. [Vietnamese, CI]

Respondents chose the second answer option, “owned free and clear,” relatively less often, as shown in Figure 4.

Figure 4  Response of owned free and clear by language.

In contrast to the other six languages, Spanish-speakers participating in cognitive interviews chose this answer option with relative frequency. Our qualitative findings for Spanish-speakers in particular indicate that many Spanish-speakers were unfamiliar with the concept of a mortgage, which may account for this discrepancy. Although mortgages exist in other countries, the “culture of credit” is not as predominant as in the United States and people are used to paying off their homes. Further, Hispanic immigrants in particular own homes at relatively lower rates than other immigrant groups (Jensen 2001) and other ethnic groups (Bureau 2017). This pattern would lead one to expect less familiarity with housing concepts, as illustrated in the following quote:

• [‘Mortgage’ means] if they already finished paying for the house. But I am not familiar [with the concept of ‘mortgage’], because in my country, we pay for the house and that’s it. But I don’t know about the system here. [Spanish, CI]

As a result, respondents may have assumed that the house must be paid off despite the mention of a mortgage in the vignette. Alternatively, respondents may have been unfamiliar with the term “free and clear.” Notably, the tendency to assume the housing unit was paid off due to a lack of familiarity with mortgages or with the phrase “free and clear” was a problem that emerged much more frequently in the cognitive interviews than in the focus groups among Spanish-speakers (40% and 5% of responses, respectively).

After “owned with a mortgage,” the most frequent answer endorsed by respondents across languages was “occupied without payment of rent” (see Figure 5).

Figure 5  Response of occupied without payment of rent by language.

In most languages, this incorrect answer was given by respondents more in the one-on-one cognitive interview setting than in the focus groups. Of note are the Chinese- and Arabic-speakers, who demonstrated the greatest gap in the percentage of reporting this answer (14% gap for Chinese-speakers and 30% gap for Arabic-speakers). Choosing this answer option might be the result of misunderstanding the term “occupied without payment of rent,” or of misunderstanding the focal point of the survey question (i.e., the housing unit rather than the respondent). Again, our findings indicate that testing this question with a vignette only in focus groups would mask the misinterpretation of this response category for Chinese- and Arabic-speakers.

Qualitative analysis shows that some respondents engaged in role-playing as either the mother or son when answering, although we did not include instructions for respondents to embody a role (Cohen, Manion, and Morrison 2011). For example, one respondent said,

• If I were in the situation in my age, I would choose [‘owned with a mortgage’]. But I should pretend that I am this grandmother? A grandmother will simply think about the fact [that] she doesn’t pay. So, I think she will choose that [‘occupied without payment’ option]. [Korean, FG]

In this case, a respondent engaging in role-playing gave the incorrect response to the vignette, despite understanding the survey question. This role-playing highlights how respondents perceived different possible answers depending on who fills out the survey, without noticing that the question focuses on the housing unit. In addition, role-playing, if not identified by the researcher, can muddy the results of testing a survey question.

Discussion

In this study, we find that administering a vignette related to homeownership in cognitive interviews resulted in respondents reporting more wrong answers than in focus groups, and that this effect differs by language. We theorize that opinion leaders led to more respondents choosing the correct answer in focus group discussion in some languages, but future research is needed. This finding suggests that vignettes in focus groups may be less well-suited for uncovering problems with survey questions compared to cognitive interviews, particularly for Arabic- and Spanish-speakers. However, this finding is contingent on language: in English and Korean, cognitive interviews were more effective than focus groups at producing discussion of incorrect answers indicative of misunderstanding the survey question. Taken together, these results indicate that researchers must consider target language when translating vignettes and choosing a research setting in which to administer vignettes.

Additionally, both the demographic characteristics of particular immigrant populations and the complexity of translating a survey concept into another language may drive misunderstandings of technical terms such as “mortgage.” Researchers should consider the role of language in survey item interpretation when selecting a pretesting methodology and which respondents to include in pretesting (Pérez 2016).

Finally, vignettes offer a means to test survey questions without recruiting rare or hard-to-reach populations, such as multilingual populations representing various homeownership statuses. However, researchers must be aware of the efficacy of using vignettes in different research methodology contexts and language settings, and further research should investigate under what conditions vignettes can be used in place of recruiting specific populations.

Disclaimer

Any views expressed are those of the author(s) and not necessarily those of the U.S. Census Bureau.

Acknowledgement

The authors acknowledge RTI International for conducting cognitive interviews and focus groups in Spanish, Arabic, Chinese, Korean, and Vietnamese. We also acknowledge Research Support Services for conducting Russian and Spanish cognitive interviews and focus groups, and both organizations for English focus groups. We also acknowledge colleagues in the Center for Survey Measurement who conducted cognitive interviews in English and Spanish.