Photo of Kyley McGeeney

As the first “Interview with an Expert” article of 2018, we are pleased to bring you an interview recently conducted with American Association for Public Opinion Research (AAPOR) member (and Associate Chair of the AAPOR Education Committee) Kyley McGeeney. Kyley is Senior Director of Survey Methods at PSB (Penn Schoen Berland) where she serves as a methodology consultant to researchers company-wide, works closely with the firm’s Census Bureau team, and will be working to design PSB’s 2018 likely voter screens and models. Kyley’s expertise includes overall research design, pre-election polling, sampling, questionnaire design, data collection protocol, weighting, analysis, and new survey technologies. Her past work includes the AAPOR 2016 Election Polling report as well as studies for the Air Force, Navy, Department of Labor, Department of State, and Sallie Mae. She is co-author of numerous publications, studying online panels, comparing web to telephone surveys, examining the use of technology for surveys (mobile, texting, apps), as well as reviewing more traditional telephone survey methods. Kyley is a graduate of The New School, and she earned her Master of Professional Studies degree in Applied Statistics at Cornell University.

As a methodologist at PSB, as well as nearly a decade of experience as a methodologist at Pew Research Center and at Gallup, we were interested in hearing her expert opinion regarding the role of a methodologist. Specifically, we wanted to learn about the particular skills and responsibilities required of a methodologist, the rewards and challenges Kyley has faced in her long and exceptional career as a methodologist, and her advice for early-career and seasoned researchers who aspire to shift into the realm of “methodologist.”

1. Thank you, Kyley, for agreeing to this interview. Let’s start with learning something about your background. How did you become a methodologist?

Most people don’t grow up saying they want to be a methodologist and I’m no exception. I went to school for statistics and initially took a job at Gallup so I could work in management consulting. However, once there I was really drawn to the methodology side of the survey research projects and soon found myself working as a methodologist.

I then had to supplement my statistical training with that in survey research methodology. I did so by taking courses at the Joint Program in Survey Methodology (JPSM) at the University of Maryland; I attended conferences, short courses, and webinars; I voraciously read books and journal articles; and eventually I conducted, presented, and published my own methodological research.

2. From your perspective, what is a methodologist? What would you say are the top skills or attributes required of a methodologist?

A methodologist serves a very important role. A methodologist designs all the steps in conducting a survey to make sure that the data collected are accurate and tell the right story. Typically, someone comes to me with a research question that they want to answer using a survey. It’s my job as a methodologist to identify the population of interest and figure out how we will sample them; identify a sample frame with contact info; decide what mode we’re going to use to contact and survey them; decide on other elements of the data collection protocol such as how many reminders we’ll send or calls we’ll attempt; design the questionnaire in terms of what we’ll ask and how; and, finally, weight and analyze the resulting data.

As you can tell from these activities, a methodologist needs strong statistical and data analysis skills. What’s equally important is for a methodologist to be able to weigh trade-offs. There’s typically no “right” answer when it comes to survey research methodology. There are also usually conflicting priorities such as cost, timing, and accuracy. For each step above, I have to weigh the pros and cons of various alternatives, make choices, be able to support and defend my choices, and then convince others of these choices. To do so, it is important to be well-versed in the literature, understand the priorities of the survey sponsor or client, be willing to take a stand for things I think are important but also understand where it’s necessary to compromise.

To follow up on that question, I wonder if you can talk about whether your definition of a methodologist includes knowledge of qualitative in addition to quantitative methods. For instance, you mention how you have to “weigh the pros and cons of various alternatives.” When considering alternatives, are there times when the solution may point to using qualitative methods (such as focus groups or in-depth interviews) rather than a quantitative design? If so, does a methodologist need to be versed in both quantitative and qualitative research methods?

I am personally much more well-versed in quantitative methods than qualitative. I think people who are proficient in either would qualify as a methodologist, and I don’t think you need to be an expert in both to count. That being said, what I think you do need to know, even as a quantitative methodologist, is when certain qualitative methods would be appropriate or even better serve the project goals than quantitative. Along those lines there may even be situations in which collecting survey, qualitative, or experimental data (“designed data”) at all might not be the answer if a secondary/big data/administrative record (“organic data”) source exists. In fact the AAPOR Best Practices for Survey Research (http://www.aapor.org/Standards-Ethics/Best-Practices.aspx) includes “consider alternative data beyond a survey” as the second step.

3. Talk to us about how you have utilized some of your skills as a methodologist at PSB or your previous employers. What examples can you give from your own work experiences that illustrate what it means to be a methodologist?

I’ve typically utilized my skills as a methodologist in three ways. In all of these cases I’m primarily using the same set of skills just in different contexts or with different outputs. In each case I’m helping to ensure high quality data are collected just with slightly different “customers.”

The first is as a methodologist on various project teams. For instance, at PSB I’m leading all the quantitative research underlying the 2020 Census communications campaign. Beyond that I’m a resource to all the teams at PSB who ask me to advise them on various methodological aspects of the research they’re conducting for their clients. At Pew Research Center, I was an internal resource to the eight substantive project areas such as the Internet and Tech project. Finally, at Gallup I worked on a number of project teams researching everything from employee engagement to sexual assault in the Air Force. When there are questions about how to ask something on a web survey or how to reach a particular hard-to-reach group, they come to me.

The second way I have utilized my skills as a methodologist is as part of a methodology team for large-scale internal data collection efforts. These data ultimately became public facing, but they crossed projects and clients. At Pew Research Center this meant managing the American Trends Panel, their probability-based web panel. At Gallup this meant being part of the methodology teams for the Gallup Poll, The Gallup World Poll, and The Gallup Panel. In these roles, I helped: decide which sample frames we used and how we sampled from them; identify the elements of the data collection protocol such as content, mode, and timing of survey invitations; and design and execute the weighting protocol.

The third way I have used my skills is by adding to best practices through methodological experimentation. At all three firms where I have worked, part of my job included designing, conducting, analyzing, presenting, and publishing the results of methodological experiments. This ranges from testing various telephone sample frames to experimenting with smartphone apps for web surveys. This work results in changes to the methodology for the projects I’ve discussed above and, more generally, adding to the survey research literature.

4a. With your varied and extensive work as a methodologist, there must be times when you found your job particularly rewarding. Can you give us examples from your career when you felt particularly satisfied or proud of your contributions as a methodological expert?

A time I felt particularly proud of my contribution as a methodological expert was during my time at Gallup working on the 2010 Prevalence/Incidence Survey of Sexual Assault in the Air Force. We conducted this large-scale study to understand not just prevalence and incidence of sexual assault in the Air Force but also what victims did after the event occurred, if they told anyone, and if not, why they didn’t do so.

Given the topic, this survey would have been important to me no matter what. However, what really makes it stand out in my memory is how accurate the results were: our survey estimate of reported assaults was within 20 cases of the actual number of reported assaults. Since this number could be verified, it lent credibility to every other finding from the survey that didn’t have a true value against which to compare. A striking example was just how many assaults go unreported: 76% of women and 87% of men who were assaulted did not formally report the incident. Because our methodology led to accurate estimates for the verifiable metrics, it made it much harder for people to disregard other findings in the survey like this rate of underreporting.

4b. And what about challenges? What are some of the greatest challenges or hurdles you have faced in your career as a methodologist?

One of the greatest challenges is knowing when to compromise and when to stand my ground when it comes to methodological recommendations. Earlier I talked about how a methodologist needs to weigh trade-offs: there are usually multiple ways to conduct a survey, no right answer, and always a tension between cost, speed, and accuracy. However, with each survey you not only have a different research question and options for how to answer it, the real challenge is that you also have different criteria for evaluating the options as well. Every company, client, partner, and project has different tolerances and different priorities when it comes to these factors. Something that is of the utmost importance to one client might be of little consequence to another. Knowing when to push back and when to compromise in these various situations can be tricky. As the methodologist, it is up to me to figure out what each client is comfortable with and what they feel is important, and balance that against what’s necessary to collect high quality data.

5a. Thinking back on your career, and any lessons you may have learned along the way, what advice would you give researchers – researchers new to survey research as well as seasoned researchers – who, as you were, find themselves “drawn to the methodology side” of their survey research projects and want to become a methodologist?

The greatest lesson I’ve learned is that the research landscape and technology are always changing and we as methodologists have to keep up. In my career, that has meant experimenting with mobile browser, smartphone app, and text message surveys. It’s meant exploring different ways to passively collect data, to collect and analyze nonprobability data, and to append commercially-available data. In the future, this is going to become even more important.

In 10 years, I think the field of survey research is going to look pretty different than it does today. Rising costs and declining response rates will likely mean that when we do conduct surveys, they’ll involve nonprobability sampling. More and more, we’ll also be using non-survey data to answer our research questions. It will be increasingly important for methodologists to understand those data sources (both what exists and their limitations), as well as data science techniques to harvest and analyze them.

My advice to methodologists is to start studying all of this now: learn R, take a data science course, experiment with big data. That way when the time comes, we’ll be ready.

5b. In addition to staying up-to-date with the latest (and forward-looking) methods and techniques, what advice would you give researchers on how to make the shift from management consulting (as in your case), or whatever their current position is, to the position of “methodologist”? Who should they network with? How do they do that? Where do they find a “methodologist” job? Is there an approach they might use in their current job to help them move into a methodologist position?

Let’s assume you are working in some other area related to surveys or research. You now want to make the move to methodologist. Here are some steps to help you along!

  1. Learn all you can about survey research methodology. For me, I used a combination of formal and informal training to do so. I took a few classes at the University of Maryland’s Joint Program in Survey Methodology (JPSM). I also took JPSM short courses, which are typically two to three days and a great introduction to the basics of a number of methodology topics. Through AAPOR I attended webinars over the year as well as short courses at the annual conference. I keep current with the literature by reading journal articles on survey research methodology as well. Also, if you don’t already know statistics this will be as important to learn during this time as survey methods-specific topics.

  2. Become a member of professional organizations. These organizations provide great networking opportunities including mentoring and jobs. They offer journal subscriptions, learning opportunities (webinars, short courses), and often put on annual or bi-annual conferences. Their listservs and websites can be a great resource when you have questions and there are often lively debates about current issues in survey research.

    I am a member of AAPOR, DC-AAPOR, European Survey Research Association (ESRA), World Association for Public Opinion Research (WAPOR), Council of Professional Associations on Federal Statistics (COPAFS, Affiliate Organization Representative), European Society for Opinion and Market Research (ESOMAR), American Statistical Association (ASA), Insights Association, and Washington Statistical Society (WSS). That’s obviously a lot and you could definitely choose just a few but it’s important to be involved in some way.

  3. Attend the survey research conferences of your professional organizations. Most of the organizations above have a conference every year or every other year. This is where researchers come together to present their work and learn about what others are doing. This is a great way to stay very current on survey research topics. There are also typically other educational opportunities, such as the AAPOR conference short courses I mentioned above. These conferences are a great way to network with people in the field from across the country and even the world.

    I have also found that you get the most out of membership when you volunteer for your professional organizations. I am the Associate Education Chair for AAPOR as well as a member of the Standards Committee. Anyone who is interested in volunteering for AAPOR can submit their information here.

  4. Conduct your own research. Even if you’re only doing client work, there may be ways to embed experiments to test issues you’ve been grappling with in your projects. If you do conduct your own research, look for ways to disseminate your results. This may mean presenting them at a conference, submitting them for publication at a journal, or publishing them on your company’s website. I first presented my methodological work at the AAPOR Annual Conference. Since then I’ve used all the outlets above, including publication in Survey Practice!

  5. Find a survey research mentor. I wouldn’t be where I am today if it hadn’t been for great mentors along the way. They answered questions, showed me the ropes, and helped me network in the field. Darby Steiger at Gallup was my first mentor: we co-authored my first AAPOR presentation and I attended my first AAPOR conference with her. At Pew, Scott Keeter and Courtney Kennedy were my go-to folks. I would tag along to their speaking engagements and sit in on their interviews so that when it was my turn I’d know what to do. Mentors are great because they’re not just someone you go to for questions or advice, they also sort of become advocates on your behalf, loop you in on opportunities, and generally help you navigate the professional world.