As a demonstration of this I did a bit of a butterfly on a wheel analysis of a story in today's papers. It tells us what the top ten things are that parents do to embarrass their children - things like dancing and trying to use yoof-speak. And according to my favourite newspaper, this is the result of 'research'.
So let's dig a little deeper. What's the source? According to the paper it is that highly respected research establishment Thorpe Park, the theme park in Surrey. Now in principle it is possible to obtain reasonable quality data at Thorpe Park if the the park undertook the survey properly, although immediately we have a problem because the group of people who attend Thorpe Park are self-selecting and may well not be typical of the population at large.
So how did Thorpe Park undertake the 'research'? Well, they didn't. It wasn't done at Thorpe Park at all - it was a poll taken for Thorpe Park by an online polling company. In a way this is a better situation, because online polling companies can address a wider section of the population, though of course they can only ever interact with people who have access to the internet - a much larger, but still self-selecting group. However, being professionals, pollsters can use statistics to at least try to correct for this.
That assumes, of course, that this was the right kind of poll. There are some online polling companies like YouGov that are pretty hot on getting their sample right and other good things, though even they have their limitations. But there are others who are essentially marketing organizations who are less worried about sampling and data quality and more interested in delivering useful messages for companies to use in their PR. Let's be clear - I am not saying these marketing-oriented pollsters make things up. They definitely don't. They pay out good money to people to take part in their polls. But in the end the aim of their polls is to gain publicity.
Not entirely surprisingly this poll was undertaken by a marketing-oriented pollster. According to its website "Our research enables brands to create unique data-led content – content that can be published and shared across multiple channels with a view to grabbing attention in a busy media landscape." What we don't know is what data the pollsters provided, because someone somewhere is certainly misinterpreting it. The newspaper says that the list of ways it gives for parents to embarrass their children is the 'top ten things' that parents do. This could only be discovered if respondents were given a simple text box and asked to type in the embarrassing things with no prompting. Actually what the poll did was to already decide what the top ten things were and got their respondents to mark those they considered the worst.
So was this 'research' in the scientific sense? Of course not. And I don't want to make a mountain out of a molehill. It was just a fun story, put together so Thorpe Park had a reason for getting a press release into the news. But I really think that newspapers should use better language. Don't call something like this 'research' or a 'study'. Call it a fun poll or a straw poll or similar language that makes it clear that this is not real in any sense scientific. Otherwise, when they carry a story about 'research' on climate change or particle physics we might be inclined to be equally dismissive.
Comments
Post a Comment