Skip to main content

Shall I compare thee to a summer's day - the problem with comparison reviews

I'm a sucker for consumer programmes, especially those that give us an expert opinion on everyday products, such as Channel 4's Tried and Tasted. It's cheap and cheerful stuff - but there's something highly entertaining about a panel including Michel Roux Jr and Jay Rayner blind tasting meat pies and ice cream (not simultaneously). However, this show presents us with an extreme example of the problem facing most comparative reviews - how do you choose the products to be in your sample?

Don't get me wrong, I'm not knocking the journalists who suffer their way through a comparative test. I once tested getting on for 70 laptops (over a year, not all at once) for a magazine, and it's an arduous job. But the fact remains that unless you try absolutely everything you can get your hands on, as I did with those laptops, there is a hidden selection process going on, which can strongly skew the results.

To take a recent example, I saw in a newspaper a comparison of supermarket croissants. Winner was the expensive Waitrose 1 Normandy butter croissants, while, for example, Asda's version which were long life and individually wrapped (and, frankly, not very nice) received a good dressing down as not coming close to anything that could really be said to resemble a croissant. But the thing is, both Asda and Waitrose sell more than one type of croissant. Asda also has a Normandy butter croissant in its 'Extra Special' range which is pretty much the real thing - certainly far better than its long life, straight and squidgy version. So how do they decide which croissants we see and which we don't?

This pre-selection is taken to the extreme on Tried and Tasted, where only four products are compared. I can see that a small sample is necessary for TV, and they try to mix it up by having, say, two supermarket products, one branded and one artisan - but it's still a very restricted set. So while we can enjoy the fact that M and S makes the best steak and ale pie from the four selected, or that Asda and Tesco's hummus offerings were rated far better than the much more expensive branded and artisan versions, it would be easy enough to produce a totally different ranking using the same divisions of supermarket/branded/artisan but with different choices.

The most important thing - which should, surely be obvious - is to compare like with like. If you're going to look at premium croissants, only look at premium croissants. No one would seriously consider a comparative car review putting a Vauxhall Corsa, and Aston Martin DB10 and a stretched limo against each other. Unless you divide your products into sensible categories, you will always have the danger of a very mixed and misleading comparison.


  1. The purpose of the show is to entertain, not to find the best croissant or whatever-it-is they are tasting. The fun lies in seeing the presumed expert taster preferring the market-leading Gordon's Gin to the much more expensive Bombay Sapphire. And in seeing how they disagree with each other.

    1. I’m sure you’re right, Sally - and that’s why I enjoy it, I suspect. But I was just using it as a hook to talk about comparative reviews in general.


Post a Comment

Popular posts from this blog

Is 5x3 the same as 3x5?

The Internet has gone mildly bonkers over a child in America who was marked down in a test because when asked to work out 5x3 by repeated addition he/she used 5+5+5 instead of 3+3+3+3+3. Those who support the teacher say that 5x3 means 'five lots of 3' where the complainants say that 'times' is commutative (reversible) so the distinction is meaningless as 5x3 and 3x5 are indistinguishable. It's certainly true that not all mathematical operations are commutative. I think we are all comfortable that 5-3 is not the same as 3-5.  However. This not true of multiplication (of numbers). And so if there is to be any distinction, it has to be in the use of English to interpret the 'x' sign. Unfortunately, even here there is no logical way of coming up with a definitive answer. I suspect most primary school teachers would expands 'times' as 'lots of' as mentioned above. So we get 5 x 3 as '5 lots of 3'. Unfortunately that only wor

Why I hate opera

If I'm honest, the title of this post is an exaggeration to make a point. I don't really hate opera. There are a couple of operas - notably Monteverdi's Incoranazione di Poppea and Purcell's Dido & Aeneas - that I quite like. But what I do find truly sickening is the reverence with which opera is treated, as if it were some particularly great art form. Nowhere was this more obvious than in ITV's recent gut-wrenchingly awful series Pop Star to Opera Star , where the likes of Alan Tichmarsh treated the real opera singers as if they were fragile pieces on Antiques Roadshow, and the music as if it were a gift of the gods. In my opinion - and I know not everyone agrees - opera is: Mediocre music Melodramatic plots Amateurishly hammy acting A forced and unpleasant singing style Ridiculously over-supported by public funds I won't even bother to go into any detail on the plots and the acting - this is just self-evident. But the other aspects need some ex

Which idiot came up with percentage-based gradient signs

Rant warning: the contents of this post could sound like something produced by UKIP. I wish to make it clear that I do not in any way support or endorse that political party. In fact it gives me the creeps. Once upon a time, the signs for a steep hill on British roads displayed the gradient in a simple, easy-to-understand form. If the hill went up, say, one yard for every three yards forward it said '1 in 3'. Then some bureaucrat came along and decided that it would be a good idea to state the slope as a percentage. So now the sign for (say) a 1 in 10 slope says 10% (I think). That 'I think' is because the percentage-based slope is so unnatural. There are two ways we conventionally measure slopes. Either on X/Y coordiates (as in 1 in 4) or using degrees - say at a 15° angle. We don't measure them in percentages. It's easy to visualize a 1 in 3 slope, or a 30 degree angle. Much less obvious what a 33.333 recurring percent slope is. And what's a 100% slope