Research in Times of the Pandemic
COVID-19 outbreak started at the beginning of 2020 in Wuhan, China, from the start seemed something distant and far away for many. The alienation from ongoing epidemics made people perceive it as some post-apocalyptic movie or role-playing game you can play on our video game console. I have to admit I was one of those people who thought COVID-19 would never reach a dramatic scale. Sitting in the Istanbul mammoth airport lobby in January 2020 and reading Plagues and Peoples by William H. McNeill. I was throwing my “expert knowledge” how the infectious disease spread and how novel coronavirus can be managed effectively. Unfortunately, I was not alone. As it was explored earlier by Kahneman and Tversky, people tend to err and they tend to err more when they are overconfident in their judgments. In a couple of months, pictures from TV screens became reality for many globally.
The public healthcare challenges called for extraordinary measures that limited face-to-face interactions. However, even without governmental restrictions the public mood and attitudes dramatically changed ways of interpersonal communication. For the social researchers, that meant drastic changes not only in thin their professional life but also in their day to day activities. While reflecting on those issues, I would like to share my personal experience, reflections and lessons learned from the last 5 months of COVID-19 outbreak.
The first case in Georgia was confirmed on the 26th of February. The first several weeks everything in Georgia, including day-to-day activities in research, did not see any substantial changes. However, with an increase in the number of infected people in Georgia and around the globe, the organization I work for suspended all ongoing face-to-face fieldwork and soon started working remotely. Reasons for the suspension of face-to-face activities were influenced by several reasons. The first one is clearly related to public healthcare issues. The average national represented face-to-face survey means employment of around 30 enumerators/interviewers and 8 fieldwork supervisors spread through the country. Depending on the research design number of completed interviews and potential contact vary, but on average during one study interviewers in total contact 3000 households and interview around 1500 people. Taking into consideration the nature of COVID-19 and how rapidly it can spread, the average face-to-face fieldwork could be the disaster from the public healthcare standpoint. Another reason was the potential decrease in the response rate. Though the epidemiological situation in Georgia was not so worrying at the beginning of March, psychological factors could naturally influence the respondents’ willingness to participate in the survey. The last, but not the least factor is related to the changed priorities and goals of the research institution, donors, and other relevant stakeholders. The COVID-19 created a situation when already planned projects and activities lost their relevance. Drastic changes in our societies created the need for other projects, the application of alternative research methods, and the rearrangement of the logic of work.
Since face-to-face polling became impossible, both because of public healthcare and legal constraints, the research institutions globally halted their customary activities. For instance, Pew Research Center, one of the biggest and leading institutions operating globally, in late April announced that it suspended its global operations. Though face-to-face activities halted, data still should be collected. The most widespread substitute for the majority of pollsters was Computer-assisted telephone interviewing (CATI) and web-based panels. The data collection was transferred in the remote and virtual space not only in the quantitative data collection context but in the case of qualitative research methods too. In-depth interviews and focus group discussions in time of pandemics were conducted using different videotelephony and online chat applications. The challenges related to the application of qualitative methods using online modes of data collection are inexhaustible and it deserves a separate piece to be written. To be more concise, in this blog I will focus on the quantitative modes of data collection.
Though alternatives were diverse, in the Georgian context CATI was the primary mean of data collection in the quantitative research methods. The fact that the absolute majority of Georgians own personal mobile phones, provides a unique possibility to reach out to the respondents without physically contacting them. While using the random digit dialing (RDD) technique, it is possible to generate an effective sampling frame and meet the requirements of representatives. Also, using that method eliminates the possible challenges of the ethical issues related to obtaining the mobile phone numbers that are unfortunately often violated when institutions are obtaining phone numbers from a number of semi-legal or illegal sources. The CATI survey is proficient when it comes to collecting the data during the severe epidemiologic context, for example, CATI was used in Sierra Leone after the Ebola crisis to collect data on the Socio-Economic Impacts of Ebola 2014-2015. There are several considerations regarding designing an effective CATI survey. From my personal experience, they can be summarized in the following way:
Keep the overall length of questionnaire short
Keep the wording of the questions short
Avoid scales that are more than 5 points
Scales should be fully labelled
Dummy/dichotomous answer options are the best option
Try to randomize the sequence of matrix-type questions, everywhere it is possible
Long vignette text usually does not work
Include at least two or three questions from already conducted face-to-face research
The last point relates to the importance of cross-checking and validation of CATI results. After conducting the CATI mode it is also utterly important to compare the frequency and distribution of the variables to other, usually face-to-face modes. Though it is not methodologically justified to directly compare the results from different modes of data collection, still this possibility makes it possible to have a general idea about the potential discrepancies and similarities.
To increase the response rate, several ideas have been circulated in the polling industry. One of the suggestions was related to sending an SMS message in advance of calling and specifying the agency running the survey. Often providing the financial incentives are also perceived as an important means to encourage respondents, but a CATI survey experiment conducted in Georgia challenges this idea. From my personal standpoint, keeping it short is the best way to increase the overall response rate.
CATI being effective, but is not the only mean of data collection. The online surveys, like web-based panels and similar modes, can solve the shortcoming of the CATI mode, like limitations regarding the length of the questionnaire, wording, and application of a large fully-labelled scales in the answer options. Also, in online survey researchers can freely use different audio-visual materials to enrich their survey instrument. However, in the Georgian context application of online surveys is restricted in several ways. The first circumstance is related to the non-existence of the sufficient pool of respondents and the sampling frame to run the representative surveys. Though Andrew Gelman and colleagues have demonstrated that non-representative polls, online polls can be also used for extrapolation using multilevel regression and poststratification. However, this needs the existence of good baseline data from previously collected data, also higher numbers of observations compared to the other types of surveys. During the COVID-19 lockdown, CRRC-Georgia conducted an experimental online survey, when questioners were distributed using the Facebook platform. As was expected, the socio-demographic characteristics were skewed toward young, urbanite, and educated individuals. Therefore, while conducting and planning the online surveys in the Georgian context, one should take into consideration the following caveats. The first one is to carefully determine the aim of the survey: either need for extrapolation or need for explaining the certain causal and/or correlational mechanism between variables. Also, it is important to understand, how to use and interpret the collected data.
The COVID-19 provided a number of challenges to social researchers, but it also gave us a number of possibilities to expand and enhance existing data collection methods. Researchers globally, as well as in our region have already started sharing their experience regarding their approaches and best practices in terms of data collection in the current extreme circumstances. In this context, the openness and transparency of applied methods, research instrument, and collected data are crucial to ensure not only the methodological issues like replicability and verifiability but also to increase the public trust toward the social science in general and public opinion polling in particular.
For the future read:
Kahneman, D. and Tversky, A., 1977. Intuitive prediction: Biases and corrective procedures. Decisions and Designs Inc Mclean Va.
McNeill, W. H. (1998). Plagues and peoples. 1976. Garden City, NY: Anchor P.
Tversky, A. and Kahneman, D., 1983. Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment. Psychological review, 90(4), p.293.
Wang, W., Rothschild, D., Goel, S., & Gelman, A. (2015). Forecasting elections with non-representative polls. International Journal of Forecasting, 31(3), 980-991.
Author: Rati Shubladze, Researcher