What is good quality evidence?

I've recently had an interesting email discussion about UFOs, which has some lessons to learn for UFO fans (and for politicians, amongst others) on what constitutes good evidence. I had reviewed a book called The Compelling Scientific Evidence for UFOs and this resulted in some discussion with its author, Erol Faruk.

I'm what I'd call a hopeful sceptic when it comes to UFOs. I'd really like UFOs to exist, but I expect the evidence supporting a UFO sighting to be strong - let's fact it, this is a pretty remarkable claim, so we need strong evidence if we are to support it.

A big problem I have with many UFO sightings is that they are based on either pure witness testimony or very bad photographs that just show lights in the sky or fuzzy blobs. In other cases, UFO investigators make a huge leap from 'there's something strange' to 'because it's strange, it must be caused by an extraterrestrial ship.' I'd like to give two examples that cover these three issues.

Let's start with the fuzzy photos. There are all sorts of problems with these. One is the ease with which they can be faked. Like many before me, I've been tempted to have a go at this myself when I was at university, though just for fun.

Here's one of mine, showing a flying saucer over Aviemore, Scotland. It's actually a metal camping plate, thrown frisbee-like, and could have been better, but you get the idea. (These days it would be trivial to do with Photoshop, but back then it was not an option.)

I don't doubt that many of these fuzzy images we see aren't intentionally fake. But so many of them are pretty much fuzzy lights in the sky (try living near Heathrow as I once did), or could be ducks, reflections, all kinds of confusing things. Nowadays most people carry a good camera and routinely take clear, quick snaps, so we should be getting lots of good, clear photos. We aren't.

In my email discussion I brought up the particular issue of a sighting of a craft over Phoenix, Arizona in 1997, detailed in Faruk's book. According to this, ‘hundreds, possibly thousands of witnesses’ saw the Phoenix craft in 1997, but without producing any clear photos. By then many people had video cameras and many of the witnesses were out to see the comet Hale Bopp and would have had cameras on tripods etc. Where are the good, clear, unequivocal photos and videos? In the 1980s, I saw my own equivalent of a UFO (except it was identified). I looked out of a window at home and my jaw dropped to see a space shuttle on the back of a 747 flying past. I was not expecting it and had no camera prepared - but I still managed to get several clear enough photographs of it. What photos I’ve seen from Phoenix are just lights in the sky.
Unlikely to be a bird or Venus

The response I got to my doubt was 'For many UFO sightings there is "shock" element which usually leaves witnesses rooted to the spot as they're fixated on watching the unknown object with the result that they don't think of rushing indoors to locate a camera to get evidence of what they're seeing. Take a look at the first part of this video [linked] showing several witness testimonies of the Phoenix boomerang object from different vantage points.' But I don't want testimonies, I want good evidence. And I really
don’t accept the shock argument - I was shocked to see a space shuttle outside my window, but it was all the more reason to get a camera. And lots of these people out to see the comet would already have had cameras: they wouldn’t have needed to go and get them, so it should have been even easier for them than it was for me.

To finish off on photos I'd also mention the infamous 'saucers over the Capitol building' photo from Washington in 1952. This looks really impressive as often shown as a cropped shot of the Capitol building with an array of lights in the sky over it - far too regular to be anything other than a UFO formation. Only when you look at the full, uncropped photo, there is exactly the same pattern of lights on the steps in front of the building. The 'UFOs' are lens reflections from the street lights.

With the Phoenix example I was constantly referred to the witness statements. Erol commented 'No, witness testimony is rightfully not considered "good" scientific evidence, but if it comes from entirely credible witnesses - and multiple ones at that - they also cannot be summarily dismissed in my opinion. Once you have some spare time do take a look at that video and judge the credibility of the witnesses yourself. As far as I know none of them were - unfortunately - in a position to snatch a camera to take photos. Does that mean that their testimony is worthless?' Well, yes, pretty much if it is unsubstantiated by evidence. Human beings are terrible observers. And it's very easy to see something in the sky at night and misinterpret it. We need stronger evidence than testimony.

Which brings us onto the topic of Erol's book. This was another sighting in Delphos, Kansas. Three members of a family allegedly saw a craft which had landed on their farm take off and fly away. They  describe finding a glowing ring where the craft had landed, which they took a photo of. In following days there was a white ring on the ground and the soil where the craft had apparently been in contact seemed strange. Erol later analysed the soil and found it contained some interesting chemical constituents that made it water resistant and fluorescent. That is, indeed, interesting. The problem is that in and of itself, a strange material in the soil does not in any way indicate extraterrestrial involvement - that suggestion is solely based on the witness testimony. There was, of course, the photo of the 'glowing ring' - but this is indistinguishable from a photo of the white ring in the daytime. It has also been suggested elsewhere that there was a circular chicken feeder on the farm previously at this spot, which could have caused the markings.

And here we get the problem of separating good evidence from bad. Erol's response to my points was 'My analysis showed the presence in the ring soil of a highly water-soluble organic compound which I characterised in terms of its light absorption and fluorescence properties. I then used this information - and only this information - to propose a hypothesis for the sighting report. This hypothesis perfectly explains the vivid colours of the UFO [reported by the witnesses] and of the glow observed between the object and the ground during Ronald's observation. It also explains the glowing ring left behind after the UFO departed of which a photo was then taken . It perfectly explains the ring elongation of the ring towards the wind direction on the night of the event. It also explains the pitted craters on the ring felt when the witnesses touched its surface, as well as the moistness of the ring and the noted anaesthetic effects consistent with the dual hydrophobic/hydrophilic nature of the soil compound.'

Unfortunately, the only evidence that the ring was produced by a UFO was witness testimony from a small group of people (who allegedly were paid for providing their story, though I don’t know if this true). There was no strong evidence for either the presence of the UFO or the glowing ring - again, the only evidence this wasn't a flash photograph, or that it was taken at night, was from the witness statement. The only strong evidence was the chemical analysis - which showed there was something unusual in the soil, but did not provide anything suggestive of extraterrestrial origin. The weakness was the dependence on the family’s testimony to make the connection from ‘unusual chemical deposit’ to ‘UFO’.

I am not in any sense saying that this shows there aren’t UFOs. It doesn't. No evidence can prove this. But we always need to be aware of the quality of evidence at each stage of making a connection. And in this case, while one bit of evidence is good, it is nowhere near enough to make the leap to evidence that an extraterrestrial craft landed.


  1. Hello Brian – you are making fundamental logic errors in your analysis above. The Delphos case involves the reported sighting of a close proximity UFO by a family who also claim that the UFO left a glowing ring in the ground. The mother was sufficiently impressed with the brightness of the ring glow after the UFO left for her to rush back and fetch her camera to take a photograph of the same. They called the local sheriff the next day who took an additional photo of the ring in daylight. The ring clearly exists from the second photo and it also has the peculiar attribute of being elongated towards the wind direction on the night of the alleged event.

    Now, either their story is a very elaborate hoax - or it actually occurred as described. The notion that the ring might be the result of the legacy of a chicken coop would have to place the story in the category of a hoax, since the family would have known about that coop’s existence. That is why the chemical analysis of the ring soil is so very important! It revealed the presence of a highly unusual and air-sensitive water soluble organic compound that permeated uniformly throughout the ring soil material down to a level of 14 inches. It wasn’t a mish-mash of organic material – it was just one compound with varying degrees of oxidation – higher from surface material while less so if taken from beneath the surface.

    The analysis of this one compound was extremely difficult owing to its instability in air. Thin layer chromatographic analysis showed it to oxidise to a highly fluorescent (blue emission) product in a matter of seconds, such that its proper isolation and characterisation would have required working in a fume cupboard under an inert nitrogen atmosphere, which unfortunately I didn’t have access to in the lab that I was in.

    If the case was a hoax, where would the family have obtained such an unusual compound that defied proper analysis and had such unusual attributes of high water solubility while rapidly oxidising to produce a fluorescent derivative? The degree of hoax elaboration would have to be extraordinary in order to take into account the observed elongation of the ring towards the wind direction! The notion that the family received money for the story was a slur originally introduced by the arch debunker Phil Klass (who also initially proposed the chicken coop idea) to try and ridicule the story when it first came out.

    Now consider that these very unusual attributes of the soil compound are actually what would be required for a chemiluminescent reaction to take place! Although I wasn’t myself able to demonstrate chemiluminescence from ring soil material I do know that another U.S laboratory (the Battelle Memorial Institute) did observe light emission from the same material from their own investigations! Thus, the only logical position to take would be that the case is NOT a hoax and that the soil analysis provides overwhelmingly positive evidence for the actuality of the report as described by the family!

    Unfortunately, your default ‘logic’ position appears to be that you somehow ‘know’ that UFOs can’t possibly exist, and that therefore the Delphos report just HAS to be explained some other way! This is not a valid argument because no one – not you, nor Seth Shostak or any other scientist – can possibly foresee what kind of technology a super advanced alien technology ‘out there’ would be capable of. We have a measly 400 years of scientific heritage to be proud of – what will WE be achieving in another thousand years for example? The only possible course of action as far as investigating UFOs is concerned is to FOLLOW where the evidence points to! It is very clear where the Delphos case analysis is indicating - even if you yourself refuse to accept it!


Post a Comment