Is the obsession with symmetry leading physicists astray?

Not my idea of symmetry
(Image by Gregory H. Revera from Wikipedia)
Physicists love symmetry. A huge amount of the physical theory developed in the last 60 years has been derived as a result of starting from mathematical symmetry structures and using them to fit to observed aspects of the universe. The whole Higgs business is the result of a need to explain why a symmetry that was assumed isn't actually observed. (I'm not saying the Higgs field idea is wrong, by the way - it does its job well - but that's how it came about.)

However, I do wonder how much this obsession with symmetry is based on the tools that are in vogue, and an over-dependence on mathematical 'beauty', rather than on a reflection of reality.

The thing that made me ponder this was re-reading the introduction to the book Symmetry and the Beautiful Universe by Leon Lederman and Christopher Hill. It's a good book, but it is a bit worrying that the foundation laid in the introduction is a crude approximation.

Let me give a flavour of it:
Symmetry is ubiquitous... We see the graceful symmetry of a flower's petals, of a radiating seashell, of an egg... We see the ideal symmetrical disks of the Moon and Sun and their motions in apparently perfect symmetrical circles...
Yet every one of those examples is only symmetrical-ish. They are sort of symmetrical, but not really. To consider, for instance, the Moon to be symmetrical is to return to the Aristotelian universe where everything in the heavens is made of perfect spheres. But Galileo discovered with his crude telescopes that the Moon was anything but perfect and symmetrical. It's all an approximation.

Now I'm sure physicists would respond that these concepts of symmetry are only models and almost inevitably the symmetry is broken at the detailed level. Which would be fine if these were just treated as useful ad-hoc models. A bit like the traditional physicist's line of 'Let's assume the cow is a sphere.' But when the assumption of symmetry, something we never truly observe in real world macro objects, becomes so central, so driving to the theories that underly physics, I can't help but wonder whether the whole thing is an elaborate fantasy.

Perhaps in our modern version of Plato's cave we are not watching the shadows of reality, but of a fiction. When the likes of Tolkien or Martin construct a complex fantasy, we say that they are world building. Could this be happening in physics too? Only time will tell - but the good thing about science is that though it can go down wrong paths for decades or even centuries, it eventually finds enough evidence to backtrack and start again. I'm not saying our current ideas are wrong, though almost certainly some are. And I always advocate going with the theories best supported by evidence right now. But we always need to remember that the scientific endeavour isn't a matter of fact and certainty, but our best attempt given what we currently know.