Mobile phone surveys are increasingly common in low- and middle-income countries, but the methodology for these surveys is still emerging. This brief addresses a critical question: How should researchers handle nonworking phone numbers in response rate calculations?

A common formula for response rates (American Association for Public Opinion Research 2016) is:

AAPOR Response Rate1=CompleteComplete+Parital+Refusal+Noncontact+Other+Unknown Eligibility

Nonworking numbers are not assigned to any person; they are “not eligible” (AAPOR 2016: 18) and should be excluded from the calculation.

Identifying nonworking numbers in lower income countries can be challenging. There are internationally standardized call outcome codes (see examples here) for voice modes (e.g., interactive voice response). However, mobile network operators oftentimes do not assign codes correctly. For example, a mobile network operator we worked with assigned integrated services digital network (ISDN) code 19 (no answer) to nonworking numbers instead of ISDN code 1 (unallocated or unassigned). As a result, we could not distinguish numbers that were working and nonworking. Text-based surveys (e.g., short message service or SMS) are even more challenging because there are not standardized codes, as there are for voice. Furthermore, information about call outcomes varies by mobile network operator and country.

If nonworking numbers cannot be identified, researchers may tend to lump nonworking and working numbers in an “unknown eligibility” category, which is in the denominator in response rate calculations. As a result, response rates are understated — i.e., lower than they should be. If nonworking numbers could be excluded from the denominator, the response rate would increase.

We recommend the following:

  1. Researchers conducting mobile phone surveys should be transparent about their identification and treatment of nonworking numbers. When in doubt, include numbers in the denominator to provide a conservative response rate.

  2. Test error codes empirically — by attempting to contact obviously nonworking numbers such as “999999999999”) and to review the error codes that operators return.

  3. Be wary of third parties that claim to “validate” phone numbers: In our experience in Ghana and other sub-Saharan Africa countries, these third parties have a high rate of false negatives (i.e., they return impossibly high working number rates).

  4. If resources allow, conduct a special methods study using manual telephone calls to estimate the proportion of “unknown eligibility” numbers that are working and nonworking (Kennedy, Keeter, and Dimock 2008). In a survey in Nigeria, the first author (Lau) found that humans could code numbers as nonworking better than the computer-assisted telephone interviewing (CATI) system. That information was then used to adjust the response rate using AAPOR Response Rate 3.

The telecommunications landscape in low- and middle-income countries is changing rapidly. We hope that in the future, better methods will exist to identify nonworking numbers. Until then, we recommend transparency and conservative approaches outlined here when calculating response rates.