An updated post from July 2015
One of the themes I return to with regularity is the way that the media rarely concerns itself with the quality of scientific sources. There is a huge difference between a Cochrane survey of all available research, or a large scale properly controlled trial, and the type of 'study' where you choose 12 people who only ever buy Volkswagen Golfs and ask them what's the best family car. Yet the media just churn it all out with equal weight, telling us that 'a study has found...' or 'research shows...' They may give us a hint of a source, but that rarely gives enough information to be sure of the quality.As a demonstration of this, I did a bit of a butterfly-on-a-wheel analysis of a story in today's papers. It tells us what the top ten things are that parents do to embarrass their children - things like dancing and trying to use yoof-speak. And according to my favourite newspaper (the i), this is the result of research.
So let's dig a little deeper. What's the source? According to the paper it is that highly respected research establishment Thorpe Park, which is a theme park in Surrey. Now, in principle, it is possible to obtain reasonable quality data at Thorpe Park if the the park undertook the survey properly and professionally, although immediately we have a problem because the type of people who attend Thorpe Park are self-selecting and will not be typical of the population at large.
So how did Thorpe Park undertake the 'research'? Well, they didn't. It wasn't done at Thorpe Park at all - it was a poll taken for Thorpe Park by an online polling company. In a way this is a better situation, because online polling companies can address a wider section of the population, though of course they can only ever interact with people who have access to the internet - a much larger, but still self-selecting group. However, being professionals, pollsters can use statistics to at least try to correct for this.
That assumes, of course, that this was the right kind of poll. There are some online polling companies like YouGov that are pretty hot on getting their sample right and other good things, though even they have their limitations. But there are others who are essentially marketing organizations who are less worried about sampling and data quality and more interested in delivering useful messages for companies to use in their PR. Let's be clear - I am not saying these marketing-oriented pollsters make things up. They definitely don't. They pay out good money to people to take part in their polls. But in the end, the aim of their polls is to gain publicity.
Not entirely surprisingly, this poll was undertaken by a marketing-oriented pollster. According to its website Our research enables brands to create unique data-led content – content that can be published and shared across multiple channels with a view to grabbing attention in a busy media landscape. What we don't know is what data the pollsters provided, because someone somewhere is certainly misinterpreting it. The newspaper says that the list of ways it gives for parents to embarrass their children is the 'top ten things' that parents do. This could only be discovered if respondents were given a simple text box and asked to type in the embarrassing things with no prompting. Actually, what the poll did was to already decide what the top ten things were and got their respondents to order them by marking those they considered the worst.
So was this 'research' in the scientific sense? Obviously not. And I don't want to make a mountain out of a molehill. It was just a fun story, put together so Thorpe Park had a reason for getting its name into the news. But I really think that newspapers should use better language. Don't call something like this 'research' or a 'study'. Call it a fun poll or a straw poll or an entertaining guess that makes it clear that this is not in any sense scientific. Otherwise, when they carry a story about 'research' on climate change or particle physics the public might be inclined to be equally dismissive.
Something I'll add for the 2025 update. In my recent book Brainjacking I suggest a quick three-part checklist for anything like this. What's the source? What's the evidence? What's the quality? You might call it a bullshit detector. I think this example demonstrates its value well.
These articles will always be free - but if you'd like to support my online work, consider buying a virtual coffee or taking out a membership:
Comments
Post a Comment