Interviewers in Computer Assisted Telephone Interviews: A Standardization Controversy

Introduction

One of the primary aims of standardizing the interview process is to control interviewer influence, that is, to make interviewer behaviors independent of situational factors (i.e., conditions in which the interview is conducted, respondent’s characteristics) (Fowler and Mangione 1990). However, standardization may lead to communication problems between the interviewer and the respondent, which can be characterized by the respondent’s lack of knowledge of terminology used in the question or the inability of the interviewer to engage in a conversation initiated by the respondent. As a remedy to these difficulties, an alternative approach has been proposed – conversational/flexible interviewing – to assure that all respondents understand survey questions as intended (Beatty 1995; Suchman and Jordan 1990). Here, interviewers should deviate from the standardized script and clarify the question concept so that a respondent interprets questions consistently and correctly. A series of studies (Conrad and Schober 2000; Schober and Conrad 1997) has shown that conversational interviewing improves response accuracy when respondents’ circumstances are not typical. However, the drawback of this approach is that it requires additional time to clarify concepts and to train the interviewers in these concepts. As a result, data collection might last longer and cost more money.

Despite its disadvantages, standardized interviewing is practiced in all sectors of the survey industry, and conversational interviewing can be considered only as an option occasionally implemented in the survey practice. This universal use of standardized interviewing motivated us to investigate the interviewers’ experiences associated with conducting interviews. In particular, telephone interviewers were asked how often and in what context they were likely to violate the standardization rule. Moreover, computer-assisted telephone interviewing (CATI) studio managers were invited to comment on the remarks delivered by the interviewers.

This paper aims to emphasize the importance of proper design of CATI scripts and of the official instructions telephone interviewers are provided with. Moreover, the paper shows how inflexibility on the part of the clients ordering research services can affect fieldwork activities and – as a consequence – can lead to survey data of poor quality.

Methodology

Within this paper, we will outline selected results of the study carried out between 2009 and 2010 among 12 major Polish commercial survey organizations^[1]. Each of these companies has CATI facilities, and each carries out telephone interviews on a regular basis. Organizations selected to participate were required to have a certificate in the CATI category issued by the Interviewing Quality Control Program^[2]. In 2009, a total of 18 firms were certified; however, six of them did not participate in the study; either they refused or their studio had been closed down.

The research was based on three methods of data collection: a standardized self-administered questionnaire for CATI interviewers, a standardized self-administered questionnaire for CATI studio managers, and in-depth interview (IDI) with well-experienced telephone interviewers. Additionally, follow-up research was carried out in 2013. During IDIs, CATI studio managers were asked to comment on the results obtained in the main study. The detailed information concerning fieldwork is presented in Table 1. The paper focuses on the outcomes of the qualitative part of the research^[3]. We will describe the problems the interviewers encountered while conducting standardized telephone interviews as well as remarks from fieldwork managers concerning these difficulties.

Results

The interviewers claim that that adhering to the standardized interview protocol is troublesome, because they often encounter respondents who do not understand the questions as initially phrased. Surprisingly, in such cases, the majority of interviewers deviate from interviewing rules (i.e., the rule of asking questions in the form written in the script or the rule prohibiting explanation of terms used in the questions). Usually, interviewers reformulate the questions using language that is more easily understood by the respondents, or they explain any terms which may be unclear. Although all interviewers are aware of the importance of standardizing the interview protocol, there seems to be unspoken consent in most research firms to deviate from the rules when the respondents appear to experience cognitive difficulties. This consent is usually unspoken; however, in isolated cases, there are supervisors who instruct the interviewers on how to reformulate the questions.

We often have to simplify the questions. If people don’t understand the questions, they are irritated and want to hang up. Of course, it’s a silent consent. No one is instructed to behave in such way, but no one finds fault with it as well. When I’m talking to a dimwit who doesn’t understand most words, I have two possibilities: either finish the interview or adapt to the respondent. (CATI interviewer)

It is notable that almost all interviewers indicated that questions used in the CATI scripts are often formulated using complicated vocabulary and syntax. The questions have not been adapted to suit the cognitive skills of the average respondent. In the interviewers’ opinions, if the researchers placed greater significance on the design of research tools by listening to the interviews and talking to CATI interviewers, there would be no reason for destandardizing the procedure. It seems that researchers working with general population surveys should design questions that are as simple as possible. These preliminary measures would minimize the risk of respondents not comprehending the questions.

Sometimes the questions are so horrible that even I don’t know what the matter is. So how can those poor respondents with elementary education understand such questions? […] No researcher, no supervisor, that hadn’t worked as a CATI interviewer knows the things that we the interviewers know. (CATI interviewer)

Sometimes questions are so long that when I finish reading them, I don’t remember what the beginning is. The respondent – without written text – all the more. (CATI interviewer)

Based on interviewers’ opinions, it seems there are two types of questions that are particularly troublesome, and dealing with such questions often causes interviewers to deviate from the standardization procedure. First, there are questions with semantic scales containing a lot of response categories. In face-to-face interviews, asking such questions can be supported by visual materials presented to the respondent. In CATI, in most cases, it is not possible. Therefore, implementing these questions in CATI scripts should be avoided and a specialized technique – known as a split question (Dillman 1978), unfolding (Groves 1979) or branching (Malhotra, Krosnick, and Thomas 2009) – should be utilized. In this technique, two questions, instead of one, are asked. The first one refers to the direction of respondents’ feelings (e.g., are they, or are they not, satisfied with something). The second question asks them the degree/intensity of their feelings (e.g., are they very satisfied or quite satisfied). The interviewers, while destandardizing the protocol, seem to unconsciously use this technique and they break the questions into two parts as they correctly feel it reduces the potential for respondent confusion.

A similar practice is utilized when it comes to another type of question: the check-all-that-apply question. Here again, as the use of visual material is usually not possible, respondents find it difficult to comprehend and memorize all response categories that are read to them by the interviewer. In order to overcome these difficulties, the interviewer often changes the structure of the question and treats each response category as a single item; they read it and ask whether the respondent chooses this category or not. Then they repeat this procedure with all remaining items.

It seems that the interviewers do what should be done by researchers wording the questions and preparing the CATI scripts. They adjust research tools to the conditions typical of a telephone interview; they reword the questions in a way which, according to the methodological literature (de Leeuw, E.D. 2008), is appropriate as far as channels of communication in CATI are concerned.

This issue was also the subject of the IDIs with CATI studio managers. The managers tend to share the interviewers’ opinions, and they consider the inadequate design of CATI scripts to be one of the major factors causing difficulties in conducting the interviews in a standardized way. The managers point out two main reasons for this situation.

First, it is a common practice in large international research projects for the local agency to be responsible only for fieldwork activities and have no influence on the design of the research tools. Some clients sympathize with the agency/interviewers and apologize for the inconveniences caused by the design of survey questions. However, the clients claim that nothing can be done to improve the situation because no changes in the methodology are accepted. There are also clients who are not interested in the fieldwork difficulties and threaten to change the vendor if the agency keeps reporting problems with the research tools.

Second, according to CATI studio managers, there are researchers who are responsible for the interviewing difficulties. Some of the researchers have no prior fieldwork experience and their knowledge about this process is rather limited.

Many researchers have no experience in the field – not only those in commercial research companies, but also people from academia. For example, academics want us to ask people over the telephone whether […] there are more ethnic or cultural minorities. In such cases, we clutch our heads in astonishment. Sometimes it is not possible to persuade them [clients] not to do it and we go to the field with such stuff. What is the value of data collected in that way? […] Why are the researchers resistant to feedback? Because they don’t have time. Because any change during fieldwork causes significant changes in the dataset. Because, every single change in the questionnaire has to be discussed with the client, and the client has no time as they are on holidays. And so on and so forth. (CATI studio manager)

Imagine a survey in which we have 39 items with a 11-point scale and the interviewers must read them for four companies. It is a disaster. […] We have also problems with the forced-choice questions, with no “hard to say” option. How to standardize the interview when you have a respondent who keeps saying that they do not know how to answer, and 50 percent of your questions are forced-choice questions? The client insists on not adding “hard to say” category. They want to have all substantial answers. I can’t change it, because they won’t accept the dataset. (CATI studio manager)

Although the interviewers claim that consent for destandardizing the protocol is usually unspoken, almost all CATI studio managers say that the supervisors in their companies are rather active in helping the interviewers cope with communication difficulties. The main strategy implemented in such cases is filling the CATI script with the instructions on how to behave in particular situations. Sometimes, such instructions are transmitted orally.

Sometimes, following the questionnaire in a strict way would lead to an interview break-off. In such a case, there is consent for breaking the standardization rule. We know which questions might be troublesome and we place proper instructions in the script. We try to “standardize” these deviations from the procedure. (CATI studio manager)

It very often happens that when I go through the questionnaire with the interviewers, I say “This question is not entirely well formulated. If the respondent doesn’t know how to answer, explain them it in that way.” This is “soft standardization.” The interviewers should know that they can do, only if it comes to certain questions. I express it in a very clear way. However, it is a bit of an awkward issue. (CATI studio manager)

Discussion

A popular strategy used by the interviewers while conducting the interview with respondents having difficulty in understanding the questions is deviating from the prescribed protocol to ensure the respondents complete the survey. It can be said that the interviewers use the elements of conversational interviewing. However, they have no official instructions regulating this practice and these deviations are made based on interviewers’ decisions or are the result of supervisors’ direct advice.

Although deviations made by the interviewer can have specific interactional functions (in that they improve cohesion and coherence in the survey) (Haan, Ongena, and Huiskes 2013), in situations described in our research, such deviations were not a simple consequence of the lack of structure in a natural conversation in the standardized interview (Houtkoop-Steenstra 2000). These deviations were mainly the result of the inadequate design of CATI scripts and, even more, the lack of proper and official instructions that the interviewers should be provided with.

Undoubtedly, more attention should be paid to the design process in creating questionnaires and to the preparation of rules regulating interviewers’ behavior when problems arise during interviews. Moreover, as we see it, interviewers’ opinions are valuable sources of information about the interview process (see Loosveldt 1997), and these perspectives should be taken into consideration while preparing survey research tools (Gwartney 2007). As well, silent monitoring of interviewers’ activities (particularly by clients) might be useful in both assessing the bias introduced by uncontrolled conversational interviewing (performed without any official instructions) and in demonstrating to the clients the outcome of poor survey design and of their own inflexibility.