Skip to main content

What is a representative audience sample?

Poll of polls from BBC website
One of the reasons I wrote Dice World is that I love probability and statistics, so it was fun to see a stats row in the news.

Ukip has been kicking up a fuss over the makeup of the audience in the opposition leaders' debate last week. They say that the BBC (or, to be precise, ICM, who assembled the audience for the BBC) were biassed in favour of left-wing parties, producing the clearly overwhelming anti-Farage sentiment in the audience.

Here is what I've seen reported as the makeup ICM used: about 58 Conservative/ Ukip, 102 for Labour, the Lib Dems, SNP or Plaid Cymru, all arguably parties of the left. And 40 undecided. (This was from a fairly dodgy source, so if anyone can confirm, or has better numbers, please let me know.)

So if we ignore the undecided, that's 36 per cent who have said they will vote in a way that might make them relatively positive to Farage.

So the question is, how can you be representative? There are two significantly different interpretations of what 'representative' means in this context. One is to take the last election, the only true nationwide poll we have, as a starting point, and the other is to take a sample poll as organizations like ICM generally do.

If they had gone for the 2010 election, the Conservatives had 47 per cent of the seats (Ukip, of course, had none) - which sounds a lot more that their representation here, but that just reflects the oddities of the first past the post voting system. If you go on the only relevant figure, the percentage of votes cast, they had 36 per cent of the vote - which means that the proportion was perfect.

So how about asking people now? Based on the latest poll of polls (see above), the Conservatives have around 34 per cent of the vote, which might again make the numbers seem reasonable, were it not for the rise of Ukip. They currently stand at 12 per cent in the poll of polls, so the combined Conservative/Ukip percentage on this basis should have been 46 per cent: on this measure they were under-represented.

There is inevitably some room for subjective choice. Personally speaking, I think the votes from the last generation election (i.e. 36 per cent) is the best starting point. This is because we know general election polling is often well adrift of sampled polls, so these numbers provide the only truly reliable poll, but we do need to bear in mind that it is five years old - and that means it show the position before the rise of Ukip.

However, it would seem odd for ICM to use these figures, as the BBC wouldn't need to bring them in to use the popular vote from the last election. They could do what I did and look up the numbers. So ICM must have (and did) use a poll to decide the proportions, and in those circumstances, it does seem that Ukip has a reasonable claim that the makeup of the audience was non reflective of the UK at large. Here's ICM's explanation of what they did:
A total of 30 small geographical areas (Super Output Areas, as defined by the Office for National Statistics) were selected within a 20-mile radius of the venue. A minimum of 8 people were recruited within each area, in line with both demographic quota variables that reflected the composition of the UK population by gender, age, ethnicity, and social grade, and political protocols that reflected the balance as agreed between the broadcasters and the political parties. One fifth of the total number recruited was on the basis of being a self-defined 'undecided voter'. Separately, a small number of SNP and Plaid Cymru supporters were recruited in Scotland and Wales, using alternative recruitment strategies, reasonably decided upon by ICM. [my italics]
So, in fact, the audience was not representative of the country at all, but just of the location the debate took place, meaning that all bets were off.

Lies, damned lies and statistics, eh?


  1. I have a real beef with election pollsters. I reckon that the samples they choose are just too small to give meaningful results, especially when the results are divided up so finely, and the sample covers people of very varied ages and backgrounds. They tend to ask about 1000 people, who will to some extent be self-selected, given that some people asked will decline to be involved. You can tell the lack of meaning as the pollsters themselves say that the figures may vary by three per cent either way. And I disagree with the contention that a poll of polls is better - meta-analyses like this have their own problems. I'd only attach credibility to a poll of upwards of 10,000 people. And the actual election, of course.


Post a Comment

Popular posts from this blog

Why I hate opera

If I'm honest, the title of this post is an exaggeration to make a point. I don't really hate opera. There are a couple of operas - notably Monteverdi's Incoranazione di Poppea and Purcell's Dido & Aeneas - that I quite like. But what I do find truly sickening is the reverence with which opera is treated, as if it were some particularly great art form. Nowhere was this more obvious than in ITV's recent gut-wrenchingly awful series Pop Star to Opera Star , where the likes of Alan Tichmarsh treated the real opera singers as if they were fragile pieces on Antiques Roadshow, and the music as if it were a gift of the gods. In my opinion - and I know not everyone agrees - opera is: Mediocre music Melodramatic plots Amateurishly hammy acting A forced and unpleasant singing style Ridiculously over-supported by public funds I won't even bother to go into any detail on the plots and the acting - this is just self-evident. But the other aspects need some ex

Is 5x3 the same as 3x5?

The Internet has gone mildly bonkers over a child in America who was marked down in a test because when asked to work out 5x3 by repeated addition he/she used 5+5+5 instead of 3+3+3+3+3. Those who support the teacher say that 5x3 means 'five lots of 3' where the complainants say that 'times' is commutative (reversible) so the distinction is meaningless as 5x3 and 3x5 are indistinguishable. It's certainly true that not all mathematical operations are commutative. I think we are all comfortable that 5-3 is not the same as 3-5.  However. This not true of multiplication (of numbers). And so if there is to be any distinction, it has to be in the use of English to interpret the 'x' sign. Unfortunately, even here there is no logical way of coming up with a definitive answer. I suspect most primary school teachers would expands 'times' as 'lots of' as mentioned above. So we get 5 x 3 as '5 lots of 3'. Unfortunately that only wor

Best writing advice

I saw on Twitter the other day (via someone I know answering it), the question 'What's the best writing advice you would give to someone who wants to become a writer?' My knee-jerk response was 'Don't do it, because you aren't one.' What I mean by this is that - at least in my personal experience - you don't become a writer. Either you are one, or you aren't. There's plenty of advice to be had on how to become a better writer, or how to become a published writer... but certainly my case I always was one - certainly as soon as I started reading books.  While I was at school, I made comics. I wrote stories.  My first novel was written in my teens (thankfully now lost). I had a first career that wasn't about being a writer, but I still wrote in my spare time, sending articles off to magazines and writing a handful of novels. And eventually writing took over entirely. If you are a writer, you can't help yourself. You just do it. I'm writ