Introduction
A successful research study provides high quality data to answer the research questions. Its success can also be quantified by whether the study was completed on time and within budget. To do this, the project manager must execute, control, and monitor the plan that manages risks, communications, human resources, and additional project management processes. Having a systematic approach to manage these processes is often a necessity on large and complex quantitative surveys (Piskurich 2012; Weeks 2011). However, the use of a project management approach is not well-documented on qualitative survey research projects. This is possibly because qualitative research projects often do not involve a large sample and vary in the degree of structure. For example, Willis (2005) describes the logistics of cognitive testing to involve 9–15 respondents to be repeated for at least two rounds, recruited using quota sampling rather than achieving a representative sample, and would only involve “several interviewers.”
While less common than large quantitative studies, large and complex qualitative research studies are commissioned to provide in-depth, ethnographic picture of topics that are not well understood by only looking at the “numbers.” Managing scope, time, and cost of a large-scale qualitative study can be as challenging as managing large-scale quantitative studies. Therefore, applying a project management approach may be advantageous. Our efforts to apply and integrate a project management approach on a large qualitative survey research project offer an opportunity to estimate the impact of this approach on the study’s success and quality, which is measured by whether the requirements were fulfilled and to what degree. We also provide recommendations for future qualitative research studies.
Methods
In the time surrounding the 2010 Census, the Census Bureau investigated why people could have been duplicated (counted twice) in the decennial census, using a total of 276 qualitative interviews. The Census Bureau will use the results to inform methods to minimize duplication in the 2020 Census.
During the active recruitment period, the team of eight recruiters contacted over 3000 households by recontacting a list of suspected duplicates from the 2010 Census. Privacy and confidentiality were of utmost importance in this project, and as such, recruiters could not reveal to potential respondents that they, or someone in their household, may have been duplicated. Through two rounds of interviewing over 6 months, interviewers conducted 226 cognitive interviews using a very detailed 129-page paper-and-pencil interviewer-administered survey [see Childs, Sorokin, and Jurgenson (2011) for more details]. Interviewers also conducted 50 semi-structured, ethnographic-style interviews with people in these living situations. Interviewers completed interviews in five metropolitan areas around the country. For details about the study design and findings, see Peytcheva et al. (2013).
Managing a Large Team of Interviewers and Recruiters
We assembled a team of 13 interviewers to complete the large number of interviews. More than half were credentialed and published authors, had an average of 14 years of experience (ranging from 3 to 27 years), and five were experts on the topic of study, census coverage issues. Because recruitment determines the success of the study, we had seven recruiters who had expertise in cold calling. Both recruiters and interviewers received project-specific trainings so that their performance could be measured against the protocol.
Managing Documentation
The interview data were recorded on tape and in the form of typed interview summary reports that were stricken of personally-identifiable information. Although one official summary exists for each of the 276 interviews, there were multiple draft versions and we estimate at least 600 files at up to 30 pages each were exchanged among team members by the end of the project. At the onset of the project, the sponsor (Census Bureau) and the RTI project manager agreed on ways to organize this large amount of data, including using consistent file naming convention and a version control system to facilitate quick identification of the correct files.
Results
Overall, we believe applying a project management approach led to a successful study and a quality product. The sponsor verified and accepted all deliverables. More importantly, the project produced what it was assigned to produce for the Census Bureau – a qualitative picture of the issue of duplication. We were able to complete 276 interviews across 27 living situations on time and within budget, while at the same time, diversifying the interview findings by including other complex and less studied situations. There was no breach of confidentiality, privacy, and no negative public reactions.
Risk Mitigation
Preventing risks from being realized and identifying and correcting known issues quickly minimizes rework.
Managing Complex Respondent Recruitment Criteria, Process, and Sensitivity
Unlike the purposive or convenience sampling often used for qualitative studies, this project focused recruiting from a list sample composed of 2010 Census households where at least one person was identified as a potential duplicate by a computer algorithm from the Census Bureau. Because the interview involved re-contacting respondents many months to over a year after the census, these recruiting phone calls were much like cold contacts. Our risk analysis showed that respondent concerns regarding legitimacy of the study could be a high impact risk. To increase the study’s credibility, the Census Bureau sent out advance letters informing the sample that they may be contacted by a third party, under contract with the Census Bureau for research. In addition, we implemented stringent standards to protect the confidentiality of the list, which resided on encrypted laptops. For any given case, there were up to 74 columns in the sample list that provided information crucial for recruitment. However, this list and any personally identifying information could not be printed or emailed, and information could only be shared by telephone or written and mailed via traceable mail. To control cost, we organized the list by distance to the interviewers (sorted by zip code) and worked the closest cases first. We also created an algorithm to prioritize the cases to be called to diversify the types of respondents that would be covered, while fulfilling the 27 necessary living situations that Census wanted to study (for details, see Peytcheva et al. 2013). On a daily basis, we monitored the permutations of desired demographics in the duplicated persons list and selected recruitment samples that were more likely to result in living situations that met Census Bureau’s research agenda.
Managing Communications
Among the seven recruiters, the recruitment manager, and 13 interviewers, there were up to 231 lines of communication (calculated using this formula: N (N–1)/2 where N=the number of team members). Because few of them were co-located, managing communications helped to monitor the tasks through to completion. For example, we provided recruiters and interviewers with a protocol about scheduling and rescheduling “no-shows,” respondents who did not show up for scheduled appointments. The protocol covered the contact history and at what point the recruiter or the interviewer should follow up. No-shows and rescheduling translates into an increased workload for recruiters, thus requiring additional time and cost that could have been avoided. Interviewers were required to alert the recruitment managers immediately when no-shows happened. When the number of no-shows and interview rescheduling increased, we identified the root causes and monitored the factors that were within our control, such as making appointment reminder calls.
During the first round of data collection, communications between recruiters and interviewers became a problem and several corrective and preventive actions were taken. Because of the geographic diversity of the list sample, recruiters usually scheduled appointments in the field at locations preferred by the respondents. This was a convenience for the respondents, but not the interviewers who were more familiar with the cognitive laboratory setting. Sometimes interviewers also had limitations with their willingness to travel to some locations, distances or during certain (high traffic) times of day. On occasion, this caused interviews to be lost, causing rework. We mitigated this issue and prevented future risks by giving more work to interviewers who did not mind more extended travel, rematching recruiters who were more sensitive to interviewer preference, and retraining interviewers on scheduling protocol. We would not have been able to complete all 276 interviews by the project end date without swiftly implementing these changes. When concern about location, distance, and timing became a source of tension between interviewers and recruiters and threatened interviewing progress, the management addressed the issue right away. By doing this, we were able to minimize the negative impacts of lost interviews and rework for both the recruiters and interviewers.
A communication hierarchy also existed among the project management team and the sponsor, who monitored the technical performance of data collection by having a weekly telephone conference call. At least 48 hours prior to the meeting, the project manager sent an agenda that included a status report and a list of open issues. During the meeting, status updates were kept to a minimum (because they were in the status report already reviewed by the sponsor) and participants focused most of the time on closing open issues and approving or rejecting changes to the project scope. All decisions were documented in written meeting minutes that were approved by the sponsor.
Managing the Relationship with the Sponsor
During the course of the project, whenever significant discoveries were made, the project manager worked with the sponsor to re-baseline the project management plan. Examples of approved changes that contributed to the success and quality of the study were as follows: diversifying situations to be interviewed by screening respondents even more thoroughly; redirecting resources from recruitment cells that had insufficient numbers of listed sample; and allowing recruiters to tailor recruitment introductions. We recommend future qualitative research studies to make continuous improvement and keep stakeholders engaged by assisting them to make fact-based decisions and changes to the baseline as needed.
Assessing Quality of Qualitative Products
The level of detail in the interview summary reports is an indicator of the quality of both the summary product and the interview itself. By the final draft, these were all accepted by the sponsor and allowed the sponsor to meet their goal of making informed decisions to resolve the duplication. Census Bureau staff requested clarifications or feedback from the interviewers for less than ten percent of the draft interview summary reports, but did not require any interview to be reworked due to inappropriate level of details.
Demographics
Another quality indicator for this study was the diversity of the demographics of the subjects of the interviews (the duplicate persons). Figure 1 shows the age distribution for all of the duplicates covered in the completed interviews, which was as expected given the list sample.
Geography
During the first round of data collection, the majority of duplicate addresses for a case were “local duplicates” defined as being 12 miles or less apart. In the second round, at the request of the sponsor, only 10 percent of the completed interviews were local—the majority of cases had much greater distances between the two identified addresses. As more distant duplicates were more interesting to the Census Bureau, the improvement from first round recruitment can be attributed to implementing the preventive action of making more strategic assignments by carefully analyzing the list sample and making sample selections.
Our strategic sample selection strategies also resulted in additional living situations being represented in the data, beyond those required by the study plan (for example, two sides of a duplicate that were geographically diverse in mileage). These situations were identified as interesting and worth pursuing by the sponsor in weekly conference calls.
More quality measurements can be found in Sha et al. (2012).
Assessing Productivity: The Human Component
One source of complication in project management is the human component. Although all recruiters and interviewers were highly-skilled and experienced, they were not equally productive, as measured by the number of cases recruited or completed. We ranked the recruiters and interviewers by their productivity and compared their identified strengths. Among the three most productive recruiters who recruited 80 percent of the completed interviews, Recruiter B possessed all the identified strengths but was not as productive as Recruiter A who demonstrated only some of the “strengths” (see Table 1). In fact, Recruiter A recruited almost a fourth of all completed interviews, including members of the more difficult-to-reach populations. Future qualitative research studies should use experienced recruiters but understand that many factors contribute to productivity and tailor case assignments according to recruiters’ strengths. There may also be yet unidentified strengths that could be more tied to performance than those that were identified.
Discussion and Conclusion
At the end of this project, our approach managing this research process differed from where we started in terms of the continuous improvements we made. That was the key to the success of the project management approach. Using regular recruiter and interviewer briefings, results from audits, and feedback from the project sponsor, the improvements focused on preventing risks from being realized, implementing corrections, and minimizing rework. To assure quality, all studies should monitor and continuously improve the process of respondent recruitment, interviewing, and communications so that the product meets the requirements of the study.
Although this paper discusses applying a project management approach on a large-scale qualitative research project, the methods and measures are also applicable to small-scale qualitative efforts. In the absence of a project management approach, corrective and preventive actions are unlikely to be implemented in a timely and systematic fashion that are necessary for controlling risks and issues from jeopardizing quality and minimizing rework that would multiply the cost. As federal budget uncertainty continues, future large and small-scale qualitative research studies should use a project management approach to balance the constraints of cost, time, and scope, while achieving the desired quality
Acknowledgements
Disclaimer: This paper is released to inform interested parties of research and to encourage discussion of work in progress. Any views expressed on (statistical, methodological, technical, or operational) issues are those of the author and not necessarily those of the Census Bureau.
Mandy Sha is a certified Project Management Professional (PMP)®. The authors thank Katherine Kenward of Research Support Services (RSS), Julie Feldman of RTI International, and Sarah Heimel of the Census Bureau for their contribution to a conference presentation that shared information in this paper. The authors also acknowledge the RTI-RSS team and Ryan King, who made this study a success