Take one simple example - prime numbers. Until the 1800s, the number 1 was mostly considered to be a prime number. After all it has no divisors other than itself and 1 - so it should be. But then it was decided that one was too unique (!) - its strange properties like 1x1=1 made it somehow not a real prime number. So now 1 is excluded from the list and prime numbers start at 2. That's fine, that's the rule - but it is painfully arbitrary. If they wanted to, someone could decide that 2 wasn't a prime number either, as every other prime number is odd. They haven't yet, but it could happen.
Science may be weird and wonderful. And it mostly depends on maths - so some maths I'm very happy with because it is tied into reality. But the work of some pure mathematicians gives me the willies.