The Use of Vignettes in Evaluating Asian Language Questionnaire Items

Introduction

Cognitive interviews (CI) are often used to evaluate questionnaires. As part of the cognitive interviewing methodology, vignettes can be presented to CI participants to review specific questionnaire items. Vignettes are hypothetical situations and can be a story, a scenario, or a description, and are usually as short as a few sentences or a paragraph (Willis 2005). They can also present an advantageous approach when it may be too demanding to have all participants review every question item or cost prohibitive to recruit participants with characteristics that correspond to all skip patterns. Past studies have used vignettes to investigate how respondents of the English-language questionnaire interpret information provided to them (Bates and DeMaio 1989; Gerber, Wellens, and Keeley 1996) and whether the information matches survey definitions (Martin et al. 2004; Martin and Polivka 1995). Vignettes have also been used to identify participant comprehension issues with Spanish translations; however, written vignettes need to be adapted to include drawings and verbal descriptions to accommodate Spanish-speaking participants of lower educational levels (Goerman and Clifton 2009). In a cross-cultural setting, Goerman and Clifton (2011) also successfully used vignettes in English and Spanish cognitive testing.

We are not aware of prior literature that examined the application of vignettes to Asian languages or their effectiveness in evaluating Asian-language questionnaires. In this study, we attempted to fill in this knowledge gap. We examined whether vignettes were effective in evaluating two 2010 Census questionnaire items that were translated in Chinese, Korean, and Vietnamese. Effectiveness referred to whether vignettes could generate the information to satisfy the research objectives of the evaluation. The objectives were to examine the accuracy and naturalness of the translated questionnaires. It also aimed to address issues related to sociocultural context that affected participant comprehension. Ultimately, problematic translated items were identified and corrected. To examine the effectiveness of the vignettes, we investigated: (1) whether participants were able to complete the vignette task by reviewing the hypothetical vignette situations and forming a reaction or response to the question, (2) how a participant reached an answer and whether that answer is intended by the survey (elicited through follow-up probes), and (3) how the vignette task and its follow-up probes helped with identifying comprehension issues with the translation and any alternative translation terms that participants might have found easier to understand.

Methods

We used CI data from a U.S. Census Bureau study that was designed to pretest multilingual translations of the 2010 paper self-administered Census form. A total of 69 in-depth CIs were conducted in Chinese Korean, and Vietnamese over two rounds of CIs in Illinois, North Carolina, and the Greater DC area (for each language, 12 CIs in the first round and 11 CIs in the second round). Research participants were native Chinese, Korean, and Vietnamese speakers who spoke little or no English. Participant recruitment focused on including a diverse range of age, sex, educational attainment, and year the participant came to stay in the United States. The participants were first asked to fill out the Census form by themselves as if they had received it at home via mail, and then they participated in the cognitive debriefing, during which the vignettes were administered. More details can be found in Pan et al. (2009). Below, we describe the questionnaire items that were selected for the vignette task, the reason for the selection, and the follow-up probes.

Vignette for the Relationship Question

The relationship question asks about relationship between household members and Person 1 (the householder who owns or rents the home). As shown in Figure 1, a total of 14 relationship terms were provided as answer categories. They included biological, adopted, and in-law relationships, as well as nonrelatives.

Figure 1   Relationship question and answer categories.

In the Asian culture, adopted, step, and nonrelative relationships are not as common in cohabiting situations. In this question item, six categories described these relationships: adopted son or daughter, stepson or stepdaughter, roomer or boarder, housemate or roommate, and unmarried partner, and other nonrelative. This posed two challenges: (1) recruiting for a large enough group of participants for each of these specific households might not be realistic, and (2) it was not always feasible for the researcher to verify that the respondents had selected the intended relationship answer category for their living situation. Therefore, we created four vignettes to investigate participant comprehension of these relationship terms. Each of the four vignettes described a fictitious household. Although the names of the residents were first developed in English, we changed them to culturally appropriate names in Chinese, Korean, and Vietnamese.

The first vignette (A) was designed to test the term “housemate or roommate,” which was intended for those who shared a dwelling and the expenses. It said:

Mary lives in an apartment with her best friend Nancy. They share all their housing expenses equally. How is Mary related to Nancy?

The second vignette (B) was designed to test the term “unmarried partner,” which was intended for those who lived together in a marriage-like relationship but were not legally married. In the first round of cognitive testing, the vignette described a couple living together for 8 years. Because 8 years is a long time period, the description was modified to say 2 years in the second round of testing to avoid the perception of a common-law marriage. This was the only vignette that changed between rounds. The vignettes said:

(Round 1) David and Kimberly have lived together as a couple for 8 years but have never been legally married. How is David related to Kimberly?> (Round 2) David and Kimberly have lived together as a couple for 2 years but have never been legally married. How is David related to Kimberly?

The third vignette (C) was designed to test the term “roomer or boarder,” which was intended to describe roomers or boarders who paid for occupying a room or receiving meals. It said:

Ana rents a room from Mrs. Doe. Ana also pays for two meals a day that Mrs. Doe cooks. How is Ana related to Mrs. Doe?

The final vignette (D) was designed to test two terms: “stepson or stepdaughter” and “adopted son or daughter.” The vignette had two parts: the first part described a child of the spouse from a previous marriage, and the second part described a child who was adopted through legal proceedings. They were:

[Person A] and [Person B] are married and have three children. The oldest child is Joe. Brad’s ex-wife Jenny had given birth to Joe. How is Joe related to [Person A]? The next child is Max. [Person A] adopted Max 5 years ago. How is Max related to [Person A]?

We started with providing the participants with an example vignette so that the participants can practice performing this cognitive task. Then, they were directed to read the first vignette. After studying the description of a household in a vignette situation, participants had to identify how one resident was related to another by choosing from the list of 14 available relationship categories. Follow-up probes were asked to understand how participants reached their answers and what alternative translation might be more appropriate.

For each vignette, interviewers administered up to four follow-up probes, as shown below. Because Round 2 interviews incorporated findings about alternative translations from the first round, the Round 1 probe was revised to ask participants to compare two or more translations to describe the relationship term.

1. (Vignettes A, B, C, and D) You chose ___________. How did you choose that answer?
2. (Vignettes A, B, C, and D) What do you think they mean by [relationship term]?
3. (Vignettes A and C) Does [term 1] mean the same thing as [term 2] or are they different things to you?
4. (Vignettes A, B, and C)
   1. (Round 1) Is there another term you would use to describe this type of relationship?
   2. (Round 2) Which of the following terms would you use to describe this type of relationship?

Vignette for Baby Instruction

As shown in Figure 2, an instruction appeared in italics immediately following the age question text: Please report babies as age 0 when the child is less than 1 year old. This “baby instruction” aimed to prompt participants to put “0” as the age for a child who has not reached 1 year old.

Figure 2   Baby instruction.

We wanted to study comprehension of the baby instruction. Because not all participants had a baby in the household or they might not notice the instruction while filling out the form, we used a vignette situation and a follow-up probe, as follows:

The instruction in Question 7 says: “Please report babies as age 0 when the child is less than 1 year old.” If a person has a 4-month old baby girl, what age should she write here?

Results and Discussion

Table 1 shows two types of issues encountered by participants at each vignette and by language: (1) difficulty completing the vignette task and (2) difficulty choosing the intended answer. The vignette task required participants to consider the hypothetical vignette situation and then form a reaction or a response. Failure to do so automatically led to not choosing an intended answer.

As shown in Table 1, not choosing the intended answer occurred in the “housemate or roommate” vignette in all three languages. Both Chinese and Vietnamese interviews demonstrated difficulty with the “baby instruction” and “roomer or boarder” vignette, but Korean interviews did not. The “stepson/adopted son” vignette was difficult in the Korean and Vietnamese interviews, but no issues were detected in the Chinese interviews. In addition, Chinese and Korean interviews did not show major problems when completing the vignette task; however, the Vietnamese interviews demonstrated difficulty in three of the five vignettes. The “unmarried partner” vignette did not cause major difficulties in the Chinese and Korean interviews, but some discomfort was expressed in the Vietnamese interviews about this type of relationship.

When follow-up probes were administered, we learned how participants arrived at their answers, which subsequently informed the revisions in the translation to aid comprehension. For example, we learned that the concept of “housemate or roommate” and “roomer or boarder” were not clear in the Chinese language. Participants could not distinguish a “housemate or roommate” from a “roomer or boarder,” “unmarried partner,” or “other nonrelative.” The solution was to adapt the translation to use a description that stated the co-renting and cost-sharing aspect of a housemate or roommate and to indicate the room and board aspect of being a boarder. We also learned that while adoption involved a legal process in the United States, a child could be considered “adopted” in an informal arrangement in some Asian countries. Specifying the legal aspect in the translation for “adopted son or daughter” was particularly important in the Vietnamese translation.

The Asian tradition of “plus 1” age-counting convention was discussed more frequently in the Korean interviews than the Chinese and Vietnamese interviews. This tradition adds one to a person’s age at the advent of the Lunar New Year. In other words, a person’s age would always be “plus 1” (1 year older than the actual age since birth). Thus, babies are considered 1 year old at birth. This tradition is still practiced in modern Korea (i.e., age is reported as “plus 1” unless the words . While Chinese and Vietnamese interviews demonstrated difficulties with the baby instruction because the participants wanted to write the months rather than “0”, it was not as problematic in the Korean interviews. For example, a Korean participant thought that this baby instruction was placed especially for immigrants, so the participant did not write the baby’s age as plus 1. Although this instruction was intended for all Census form takers, it provided special assistance to this Korean participant. This is an example of where cultural norms can potentially affect response pattern and the vignette exercise revealed this issue.

In addition, vignette design and writing affected the effectiveness of the vignette method. First, vignettes were not as effective when they included too many details for participants to think through. For example, although the vignette description for “stepson or stepdaughter” (D) was short, it involved four characters: a husband, a wife, an ex-wife, and the husband’s biological son with the ex-wife. It aimed to ask how a child was related to one of the parents; however, the description started with both parents as if they were the focal point of the description. CI participants had to sort out the relationship between each character before they could choose an answer about how the son related to the wife. As a result, some participants were unable to complete the vignette task and asked the interviewer for help. Other participants changed their answers, chose multiple relationship answer categories, or chose an answer based on how one parent is related to another. The difficulties exhibited by one of the Vietnamese participants were noted in interviewer notes, as follows:

The participant took quite some time to give an answer. At first, his response was that no answer category was listed for the relationship. He then said it should be “vợ bé”(second wife) but soon changed his mind to “con nuôi”(adopted son or daughter). After reading the vignette again, he considered “con riêng” (stepchild or stepdaughter) but still was not quite sure. Then after a while he said it could be “mẹ gh_ẻ” (stepmother). Because stepmother was not among one of the listed categories, he finally decided that no answer categories would fit._

Second, there was some evidence of order effect. For instance, vignette C for “roomer or boarder” was administered right after vignette B for “unmarried partner.” After hearing the “roomer or boarder” vignette, several respondents volunteered that the vignette did not describe a marriage between the residents and deliberated whether it was an “unmarried partner” relationship. It was unclear whether they would have thought of the possibility should the vignettes were administered in a different order.

In summary, this study showed that vignettes could be practicably used to evaluate Asian language questionnaire items. In addition to the operational benefit of allowing more participants to give feedback, vignettes are effective in examining comprehension issues related to translation and the sociocultural context surrounding it. When there was difficulty completing the vignette task or providing intended answer, follow-up probes helped to understand whether translation or sociocultural influences were the reason for the difficulty. In addition, vignettes led participants to consider a term or a phrase in a hypothetical, less personal situation. For example, the only vignette without difficulty in the Vietnamese interviews was about “unmarried partner.” Even though there was some discomfort about the concept of living together without being married, Vietnamese participants were able to choose the intended answer of “unmarried partner.” Furthermore, vignettes were more effective when they were developed to minimize order effect and complicated descriptions.

Because the primary purpose of the project was to detect potential translation issues, the vignette situations and follow-up probes were geared toward eliciting only comprehension issues regarding key terms and phrases. Future research should examine the effectiveness of vignettes in soliciting additional information among Asian research participants and in comparison with a control group using English.

Acknowledgement

The author acknowledges Dr. Yuling Pan, formerly a Sociolinguist at the Census Bureau, for her contribution to a conference presentation that shared information in this article.

Disclaimer

This paper is released to inform interested parties of research and to encourage discussion of work in progress. Any views expressed on (statistical, methodological, technical, or operational) issues are those of the author and not necessarily those of the U.S. Census Bureau.