Surveys And Fraudulent Data

Many Surveys, about One in Five, May Contain Fraudulent Data

John Bohannon | February 24, 2016

How often do people conducting surveys simply fabricate some or all of the data? Several high-profile cases of fraud over the past few years have shone a spotlight on that question, but the full scope of the problem has remained unknown.  Yesterday, at a meeting in Washington, D.C., a pair of well-known researchers, Michael Robbins and Noble Kuriakose, presented a statistical test for detecting fabricated data in survey answers. When they applied it to more than 1000 public data sets from international surveys, a worrying picture emerged: About one in five of the surveys failed, indicating a high likelihood of fabricated data.

The problems were much more serious in non-OECD countries. OECD is the so-called rich countries club, so what Kuriakose and Robbins found was that data problems were more common in surveys in poorer countries. They found 5 percent of surveys in rich countries to have duplication problems, compared to 26 percent in the other countries.

Where is the duplication coming from? Kuriakose and Robbins suggest it is people hired to do data collection who make up responses because it’s less effort than actually gathering the data. And of course this happens in rich countries too, as in the notorious survey fabricated by Michael LaCour.

One thing I don’t quite understand in this new paper is why the authors don’t list the surveys where they suspect fraud. That would be good to know, right? What they should really do is post all their raw data, but perhaps they don’t have permission from the individual surveys to do this. But they could still post all their code and give their results on a survey-by-survey basis. Especially when fraud is involved, it makes sense for us to be able to see exactly what analysis was done here.

Source: https://www.washingtonpost.com/news/monkey-cage/wp/2016/02/27/can-you-trust-international-surveys/

<more at http://www.sciencemag.org/news/2016/02/survey-says-many-surveys-about-one-five-may-contain-fraudulent-data; related articles and links: https://www.washingtonpost.com/news/monkey-cage/wp/2016/02/27/can-you-trust-international-surveys/ (Can you trust international surveys? February 27, 2016) and http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2580502 (Don't Get Duped: Fraud through Duplication in Public Opinion Surveys. Noble Kuriakos and Michael Robbins. December 12, 2015. Statistical Journal of the IAOS. [Abstract: Fraud in survey research can take many forms, but a common form is through duplication of valid interviews. Duplication of a valid interview has a number of advantages: expected relationships between the variables will hold across the data set and, if done across a number of interviews, this approach can evade many standard techniques to detect fraud such as straight-lining analysis and the application of Benford's law. In this paper, we consider the likelihood of encountering near duplicates in survey data, suggest methods to fingerprint suspicious observations, report on our analysis of over 1,000 publicly available survey datasets and argue that nearly one in five widely used country-year surveys surveys from major international data sets have exact or near duplicates in excess of 5% of observations.])>