The Dangers of Unscientific Surveys in the Arab World

posted on: Oct 28, 2017

SOURCE: THE WASHINGTON POST

BY: JUSTIN GENGLER

Hardly a day passes without a new survey promising insight into pressing issues facing the Arab world. Yet, owing to the unscientific and sometimes politicized nature of many of these polls, the result is too often confusion rather than clarity, to the detriment of the many researchers and practitioners working to carry out high-quality, nonpartisan surveys in the region.

Scientific survey research in the Arab world has made remarkable strides in recent years. Global interest in Arab public opinion has been spurred by dramatic events and trends, including the terrorist attacks of 9/11, the ensuing war in Iraq and more recently the Arab Spring uprisings. These and other events have highlighted the importance of how ordinary Arab citizens think and act, even if they may often have a limited influence on formal decision-making in government.

Arab leaders themselves, including the region’s many autocratic rulers, have also come to better appreciate survey research. Whereas a decade ago Arab governments tended to view opinion surveys as posing needlessly provocative questions to citizens, particularly since the Arab uprisings of 2011 officials have been more inclined to permit surveys even on sensitive topics, partly to remain informed about shifting public opinion.

And so, even as political and methodological barriers to opinion research in the Middle East persist, the number, scope and sophistication of scientific surveys being conducted has burgeoned. Some sense of this can be gleaned from Arab nations’ participation in prominent cross-national survey projects. The widely utilized World Values Survey, for instance, which has surveyed citizens in almost 100 countries since 1981, included more Arab states (12) in its most recent wave, conducted during 2010-2014, than in all previous five waves combined.

New regionally focused projects have also been initiated. The Arab Barometer survey has studied the attitudes of almost 50,000 men and women in 15 Arab countries since 2006, using questions developed by a team of experts based mostly in the region. A spinoff from the Arab Barometer, the Arab Opinion Index conducted by the Arab Center for Research and Policy Studies in Qatar, interviewed over 18,000 Arab citizens in 12 countries in its most recent 2016 survey alone.

In these and many other social scientific surveys, researchers work hard to use best practices in survey research. Researchers also work to ensure that surveys interview large and nationally representative samples, usually at respondents’ residences. They take care to recognize and correct for the possible biasing effects of a wide range of factors, including survey mode (face-to-face vs. telephone vs. online), the characteristics of field interviewers (gender, ethnicity or signals of religiosity such as the female headscarf), and the presence of third parties (a woman’s husband, for instance) during the interview.

Researchers must also design questions that do not lead respondents to a certain answer and are likely to elicit truthful responses. This has led to the recent proliferation of survey experiments, which study sensitive topics indirectly rather than through traditional straightforward questions.

Fortunately, one problem that survey researchers working in Arab countries do not have to face is public resistance to participation in surveys. For the past two years, I have led a collaboration studying attitudes toward surveys among residents of the Gulf state of Qatar. Our surveys point to a consistent conclusion: Gulf and other Arabs are no less likely to enjoy or value surveys than any other cultural group. This is reflected in response rates from the region, which commonly exceed 50 percent, while single-digit rates are common in Western contexts.

This enthusiasm for survey research in the Middle East makes it even more important to pay close attention to methodology. Some widely reported surveys still are not transparent about their methods, making it impossible to judge the validity of the results. Still other polls report methodological details that directly undermine their claims of being representative and thus reliable.

One example is the Arab Youth Survey commissioned by the United Arab Emirates-based public relations firm ASDA’A Burson-Marsteller, which is described as measuring the “hopes, fears, and aspirations of Arab youth.” Conducted since 2008, the survey is perhaps the largest and most widely promoted of its type, surveying Arabs aged 18 to 24 from 16 countries. Yet its annual report draws conclusions and even makes cross-national comparisons based on individual surveys of as few as 150 respondents per country, and it offers little information about how respondents are chosen and interviews are conducted.

Moreover, there is a potential conflict of interest here: A PR firm is being hired by the UAE government to then conduct a survey that is meant to independently assess popular views of that government, among other things. This raises questions about the survey’s finding that the UAE is very popular in the Arab world. The 2017 report contains an entire section highlighting how the “the UAE sprints ahead of the pack as the country in which most young Arabs would like to live and want their countries to emulate.”

Another example is the separate surveys in Qatar, Saudi Arabia, the UAEand Egypt reported earlier this month by the Washington Institute for Near East Policy. Analysis of these surveys centered on the common negative view of Iran among all four samples of citizens; their agreement with the confrontational approach of President Trump toward Iran; and their willingness to make concessions in the ongoing Gulf crisis involving Qatar and the three other surveyed nations, again for the purpose of countering Iran. Despite the many scientific surveys conducted in these countries, including on the same topic, these studies are introduced as being “unprecedented.”

Most concerning is that only limited methodological details about the surveys have been released. Although the surveys are said to include “random, geographic probability” samples, only the Qatar survey reports actual sample demographics. But these demographics diverge from known population statistics on key dimensions, including education level and especially geographical location. Only 1 in 5 Qatar survey respondents had a college education, but about a third of Qatari citizens have university degrees. Similarly, half the Qatar survey sample came from the commercial capital Doha, but most citizens live in residential districts in outlying municipalities.

This underrepresentation of educated citizens and those outside of Qatar’s urban core, combined with the survey’s vague mention of “face-to-face interviews” conducted by a market research firm, suggests that interviews were conducted in public spaces rather than at respondents’ residences. This practice, which many polling companies lacking household access unfortunately resort to, violates the most fundamental assumption of survey research: that all members of a population have an equal likelihood of being selected for participation in a survey. Of course, all surveys violate this assumption to some extent. But this survey raises particular concerns. Customers in a shopping mall or worshipers outside a mosque may indeed be approached at random, but they are unlikely to be representative of a national population.

Ultimately, one cannot know to what extent sample distortions in these and other surveys produce similarly distorted substantive conclusions, but that’s exactly the problem: There is simply no way of knowing what the results of a survey would have been had it been conducted scientifically.

Problems with survey methodology aren’t just an academic concern. When flawed surveys are used to offer authoritative conclusions about public opinion and even policy advice, they may alter rather than simply measure opinions if individuals change their opinions based on how they believe others think. This is why almost 40 countries around the world restrict the publication of pre-election surveys and exit polls.

Moreover, flawed surveys hurt the ability of researchers to conduct rigorous public opinion research in the Arab world. In our study in Qatar, we found that how individuals view the integrity of surveys is associated with their willingness to take part. Politicized results from unscientific surveys may also make states warier of permitting surveys that ask about important issues. Finally, these surveys sustain the companies that carry out unscientific polls, making it more difficult to find data collection partners willing and able to carry out proper surveys.

All of this threatens to limit and even reverse the substantial progress made in measuring public opinion in the Arab world.

Justin Gengler is assistant research professor and head of the policy department at the Social and Economic Survey Research Institute at Qatar University.