Skip to main content

What is a representative audience sample?

Poll of polls from BBC website
One of the reasons I wrote Dice World is that I love probability and statistics, so it was fun to see a stats row in the news.

Ukip has been kicking up a fuss over the makeup of the audience in the opposition leaders' debate last week. They say that the BBC (or, to be precise, ICM, who assembled the audience for the BBC) were biassed in favour of left-wing parties, producing the clearly overwhelming anti-Farage sentiment in the audience.

Here is what I've seen reported as the makeup ICM used: about 58 Conservative/ Ukip, 102 for Labour, the Lib Dems, SNP or Plaid Cymru, all arguably parties of the left. And 40 undecided. (This was from a fairly dodgy source, so if anyone can confirm, or has better numbers, please let me know.)

So if we ignore the undecided, that's 36 per cent who have said they will vote in a way that might make them relatively positive to Farage.

So the question is, how can you be representative? There are two significantly different interpretations of what 'representative' means in this context. One is to take the last election, the only true nationwide poll we have, as a starting point, and the other is to take a sample poll as organizations like ICM generally do.

If they had gone for the 2010 election, the Conservatives had 47 per cent of the seats (Ukip, of course, had none) - which sounds a lot more that their representation here, but that just reflects the oddities of the first past the post voting system. If you go on the only relevant figure, the percentage of votes cast, they had 36 per cent of the vote - which means that the proportion was perfect.

So how about asking people now? Based on the latest poll of polls (see above), the Conservatives have around 34 per cent of the vote, which might again make the numbers seem reasonable, were it not for the rise of Ukip. They currently stand at 12 per cent in the poll of polls, so the combined Conservative/Ukip percentage on this basis should have been 46 per cent: on this measure they were under-represented.

There is inevitably some room for subjective choice. Personally speaking, I think the votes from the last generation election (i.e. 36 per cent) is the best starting point. This is because we know general election polling is often well adrift of sampled polls, so these numbers provide the only truly reliable poll, but we do need to bear in mind that it is five years old - and that means it show the position before the rise of Ukip.

However, it would seem odd for ICM to use these figures, as the BBC wouldn't need to bring them in to use the popular vote from the last election. They could do what I did and look up the numbers. So ICM must have (and did) use a poll to decide the proportions, and in those circumstances, it does seem that Ukip has a reasonable claim that the makeup of the audience was non reflective of the UK at large. Here's ICM's explanation of what they did:
A total of 30 small geographical areas (Super Output Areas, as defined by the Office for National Statistics) were selected within a 20-mile radius of the venue. A minimum of 8 people were recruited within each area, in line with both demographic quota variables that reflected the composition of the UK population by gender, age, ethnicity, and social grade, and political protocols that reflected the balance as agreed between the broadcasters and the political parties. One fifth of the total number recruited was on the basis of being a self-defined 'undecided voter'. Separately, a small number of SNP and Plaid Cymru supporters were recruited in Scotland and Wales, using alternative recruitment strategies, reasonably decided upon by ICM. [my italics]
So, in fact, the audience was not representative of the country at all, but just of the location the debate took place, meaning that all bets were off.

Lies, damned lies and statistics, eh?


  1. I have a real beef with election pollsters. I reckon that the samples they choose are just too small to give meaningful results, especially when the results are divided up so finely, and the sample covers people of very varied ages and backgrounds. They tend to ask about 1000 people, who will to some extent be self-selected, given that some people asked will decline to be involved. You can tell the lack of meaning as the pollsters themselves say that the figures may vary by three per cent either way. And I disagree with the contention that a poll of polls is better - meta-analyses like this have their own problems. I'd only attach credibility to a poll of upwards of 10,000 people. And the actual election, of course.


Post a Comment

Popular posts from this blog

Is 5x3 the same as 3x5?

The Internet has gone mildly bonkers over a child in America who was marked down in a test because when asked to work out 5x3 by repeated addition he/she used 5+5+5 instead of 3+3+3+3+3. Those who support the teacher say that 5x3 means 'five lots of 3' where the complainants say that 'times' is commutative (reversible) so the distinction is meaningless as 5x3 and 3x5 are indistinguishable. It's certainly true that not all mathematical operations are commutative. I think we are all comfortable that 5-3 is not the same as 3-5.  However. This not true of multiplication (of numbers). And so if there is to be any distinction, it has to be in the use of English to interpret the 'x' sign. Unfortunately, even here there is no logical way of coming up with a definitive answer. I suspect most primary school teachers would expands 'times' as 'lots of' as mentioned above. So we get 5 x 3 as '5 lots of 3'. Unfortunately that only wor

Why I hate opera

If I'm honest, the title of this post is an exaggeration to make a point. I don't really hate opera. There are a couple of operas - notably Monteverdi's Incoranazione di Poppea and Purcell's Dido & Aeneas - that I quite like. But what I do find truly sickening is the reverence with which opera is treated, as if it were some particularly great art form. Nowhere was this more obvious than in ITV's recent gut-wrenchingly awful series Pop Star to Opera Star , where the likes of Alan Tichmarsh treated the real opera singers as if they were fragile pieces on Antiques Roadshow, and the music as if it were a gift of the gods. In my opinion - and I know not everyone agrees - opera is: Mediocre music Melodramatic plots Amateurishly hammy acting A forced and unpleasant singing style Ridiculously over-supported by public funds I won't even bother to go into any detail on the plots and the acting - this is just self-evident. But the other aspects need some ex

Which idiot came up with percentage-based gradient signs

Rant warning: the contents of this post could sound like something produced by UKIP. I wish to make it clear that I do not in any way support or endorse that political party. In fact it gives me the creeps. Once upon a time, the signs for a steep hill on British roads displayed the gradient in a simple, easy-to-understand form. If the hill went up, say, one yard for every three yards forward it said '1 in 3'. Then some bureaucrat came along and decided that it would be a good idea to state the slope as a percentage. So now the sign for (say) a 1 in 10 slope says 10% (I think). That 'I think' is because the percentage-based slope is so unnatural. There are two ways we conventionally measure slopes. Either on X/Y coordiates (as in 1 in 4) or using degrees - say at a 15° angle. We don't measure them in percentages. It's easy to visualize a 1 in 3 slope, or a 30 degree angle. Much less obvious what a 33.333 recurring percent slope is. And what's a 100% slope