Don't get me wrong, I'm not knocking the journalists who suffer their way through a comparative test. I once tested getting on for 70 laptops (over a year, not all at once) for a magazine, and it's an arduous job. But the fact remains that unless you try absolutely everything you can get your hands on, as I did with those laptops, there is a hidden selection process going on, which can strongly skew the results.
To take a recent example, I saw in a newspaper a comparison of supermarket croissants. Winner was the expensive Waitrose 1 Normandy butter croissants, while, for example, Asda's version which were long life and individually wrapped (and, frankly, not very nice) received a good dressing down as not coming close to anything that could really be said to resemble a croissant. But the thing is, both Asda and Waitrose sell more than one type of croissant. Asda also has a Normandy butter croissant in its 'Extra Special' range which is pretty much the real thing - certainly far better than its long life, straight and squidgy version. So how do they decide which croissants we see and which we don't?
This pre-selection is taken to the extreme on Tried and Tasted, where only four products are compared. I can see that a small sample is necessary for TV, and they try to mix it up by having, say, two supermarket products, one branded and one artisan - but it's still a very restricted set. So while we can enjoy the fact that M and S makes the best steak and ale pie from the four selected, or that Asda and Tesco's hummus offerings were rated far better than the much more expensive branded and artisan versions, it would be easy enough to produce a totally different ranking using the same divisions of supermarket/branded/artisan but with different choices.
The most important thing - which should, surely be obvious - is to compare like with like. If you're going to look at premium croissants, only look at premium croissants. No one would seriously consider a comparative car review putting a Vauxhall Corsa, and Aston Martin DB10 and a stretched limo against each other. Unless you divide your products into sensible categories, you will always have the danger of a very mixed and misleading comparison.
The purpose of the show is to entertain, not to find the best croissant or whatever-it-is they are tasting. The fun lies in seeing the presumed expert taster preferring the market-leading Gordon's Gin to the much more expensive Bombay Sapphire. And in seeing how they disagree with each other.
ReplyDeleteI’m sure you’re right, Sally - and that’s why I enjoy it, I suspect. But I was just using it as a hook to talk about comparative reviews in general.
Delete