INFT 4017 CIS Research Methods

How to Lie with Statistics (and Probability)

In this lecture we look at the importance of understanding probability and statistics.

Benjamin Disraeli famously said "There are three kinds of lies: lies, damned lies, and statistics."

Probability and statistics are powerful tools for understanding data and making predictions, but they are prone to misunderstanding and sometimes even to deliberate misuse. We will look at some examples of both in this lecture, and ideally learn how to ask the right questions when faced with statements based on statistical analysis and probability-based predictions, and how to avoid making the same mistakes ourselves.

The title of this lecture, is of course, meant as humour. It is based on the title of a book, "How to Lie with Statistics" by Darrell Huff, which is a classic in the field. There are a number of other books which expose cases of misuse of statistics, for example "Bad Science" by Ben Goldacre. These, and many others, are useful general reading in support of this topic.

PROBABILITY

It is surprisingly easy to misunderstand how probability works and how you can (or can not) make predictions or statements based on probabilities. In this section we consider how probability is misunderstood due to people's perceptions, due to generalising a perception wrongly, and due to ignorance of the independence or otherwise of events.

PERCEPTION and risk analysis

Road versus train versus air travel

There is often a complete ignorance of relative risks, exemplified by the safety of different forms of national travel.

The community often reacts very badly to events such as rail and air crashes, but is surprisingly indifferent to road deaths.

Look at the coverage and reactions to two train crashes in the UK, in Hatfield and in Potter's Bar. These incidents triggered investigations and litigations, and were subject to ongoing media coverage. In the Potters Bar incident, 7 people died. There were calls for more stringent maintenance and testing of facilities, and fines for negligent transport suppliers.

Contrast this to the coverage given to road accidents. Most road accidents go unreported in the press, perhaps due to the sheer number of them. However in the UK, thousands of people die annually in traffic accidents but it does not trigger the same investigations as rail traffic.

Behaviour and illness

http://news.bbc.co.uk/2/hi/uk_news/magazine/7937382.stm

To some extent, the press and media are responsible for misunderstandings, frequently because the reporters do not themselves comprehend what the statistics mean, and besides, sometimes the clear truth doesn't make a good story.

For example, it was recently reported in the UK press that alcohol was implicated in various cancer risks for women. In response to this, a columnist criticised the way science was presented to the public and instead offered an alternative reading of exactly the same data in How to understand risk in 13 clicks.

SPECIFIC or generic probabilities

Often misunderstandings about probabilities arise from people mistakenly either personalising or generalising a principle. For example the birthday paradox can give estimates on the chances that
there are probably two people in this room who have the same birthday
Note that THIS IS NOT THE SAME AS
someone in this room probably has the same birthday as me
The chance of someone having the same birthday as a given person is significantly less than that of any two people from a group having the same birthday (see same birthday as you).

INDEPENDENCE of events or not

Something that many people fail to understand is whether probabilities are independent or not. The distinction is in some cases critically important for properly comprehending the science, and in fact misunderstanding it underlies some very widespread fallacies. People misunderstand the independence of events in both directions, sometimes falsely assuming events are influenced by each other (see the lotteries example below) and other times assuming that events are not related when in fact they are(see the examples from expert witnesses and ID-vs-evolution).

Chances of winning lotteries

For example, consider how people select numbers for lottery draws. Many choose the same numbers every week on the basis that eventually these numbers will come up. Now while that is true (that the numbers should eventually come up), it is no more likely for these to come up than it is likely for the exact same numbers as the previous weeks to come up again. This is because the chances of any given set of 6 numbers being drawn is *exactly the same* every week, and is not affected by how recently those numbers were drawn. The statistics on how often various numbers have been drawn over say the last 6 months or whatever are also completely irrelevant to the chances of whether they will be drawn again this week, yet many people completely misinterpret these stats and assume for example that certain numbers are "overdue" to be drawn, or that other numbers have "used up" their allowance for the present and are hence less likely to be drawn this week. These interpretations are completely wrong, and are based on an assumption that the events are not INDEPENDENT, when in fact they are. That is, drawing of a number is a completely indepedent event (with the minor exception that a number can only be drawn once in a week).

Other misconceptions are based on misunderstanding when events are not independent, as the next two examples show.

Intelligent Design or Evolution?

Let's look at an example, based on Richard Dawkins' "The Blind Watchmaker" chapter 2.

This fallacy we consider here is that the chance of complex objects, such as eyes or echolocation capabilities, occurring as the result of random mutation is vanishingly small, so much so that the argument is made that it could not have been the result of random mutation but must be due to an Intelligent Designer who must have overseen the creation of complex objects. (Note that the alternative hypothesis, namely that an Intelligent Designer could arise as the result of some natural process is nowhere investigated).

The main misunderstanding here is a failure to recognise the non-independence of the events that led to the complex object.

It is true that the chance of a complex object arising spontaneously as a result of random mutations is extremely low@. However this is not how such complex objects arise - they are an outcome (still incomplete) of a process which consists of a stream of non-independent events. Those events themselves are partly influenced by random mutations, however they are also influenced by completely non-random forces, in this case the filtering force of natural selection.

@ Note however that the chance is not actually zero, and that given enough time (an enormous quantity well over the lifetime of the universe so far) it becomes reasonably likely to occur - rather like the monkeys with typewriters reinventing Shakespeare. However it is significantly less probable than successfully guessing someone's decryption key in your own lifetime.

To explain it differently, it is reasonably probable, over a given length of time, that mutations occur (apparently mutations of various molecules occurs at a reasonably predictable interval). If those mutations were not favourable to the life form's survival, then the mutated offspring generally died without reproducing themselves (and hence without reproducing ther mutated genes). On the other hand, the mutations are sometimes favourable to the life form, and hence make it more competitive and able to survive and reproduce better. Evolution postulates that complex objects arose from slightly-less complex objects through the filtering process of survival or otherwise, over the mutated offpsring. Each favourable mutation preserved becomes the basis for later (sometimes much later) mutations, which themselves may be favourable or not. However the important feature is that all life today is the result of a stream of favourable mutations, because any unfavourable mutations were weeded out before the life form was able to reproduce.

So the complexity of an eye could easily arise, over an extremely long period of time, but a period of time well within the existence of the Earth. It would arise because initially some creature developed a light sensitivity which helped it find food or gave some other improved survival trait. Then this light sensitive species over time mutated again and again, sometimes the mutations failing to improve the creature's survivability, but at other times, improving it, and in some cases, improving its light sensitivity.

From a purely mathematical point of view, the difference between spontaneously arriving at an eye and developing an eye gradually over aeons is similar to moving through a search space. There are a number of what might be called pathways through the space of all possible living things, and certain points in that space are not reachable via evolution because there are no intermediate steps by which to arrive there (taking into account context), as there are no survivable intermediates.

However for every successful intermediate life form, the probability of a mutation is relatively high (certainly compared to that of spontaneously arriving at the same point) and the probability of that mutation being beneficial is likewise relatively high. It may take more time than many humans can comprehend but it takes significantly less time than the postulated age of the Earth and of life itself. In short, the stepwise refinement of evolution's processes requires no external agent to explain complexity.

It is also worth looking at the Wikipedia entry on irreducible complexity to read further about this and about falsifiability of hypotheses.

"Expert" witnesses

An example is when expert witnesses misunderstand probabilities and hence offer "expert" testimony that harms the case.

One well-known example is where an expert in sudden infant death syndrome ("cot death") gave evidence against mothers who had lost more than one child to cot death. His evidence was important in obtaining conviction against a number of mothers who were subsequently cleared of the conviction on appeal. The expert witness, Professor Sir Roy Meadow, is apparently known for his belief that:

one sudden infant death is a tragedy, two is suspicious and three is murder, until proved otherwise
In the case of one mother, Sally Clarke, who had been convicted of the murder of her two infant sons, the expert witness has said that the probability of two children in the same family suffering cot death was 73 million to one. In a later case, his testimony that "cot death did not run in families" was given against Trupti Patel who had lost three children to cot death, while her own mother had lost five children in early childhood (see the Guardian report).

The expert witness apparently failed to recognise that there are inheritable conditions that increase the chance of babies dying of cot death. His assumption was that these were essentially independent events and that while loss of one child to cot death was unfortunate, loss of more than one implied something more sinister. He based his assessment that the chances of two babies dying naturally was 73 million to 1 on merely squaring the probability of two such deaths being independent, namely he squared the probability of one such death in a non-smoking affluent family (around 8,500:1). (he may have been confused with Oscar Wilde's remark that "To lose one parent, Mr. Worthing, may be regarded as a misfortune; to lose both looks like carelessness").

Let's look more closely at the explanation of why the statistics were controversial in this and other cases.

The problems with the statistics of this case were threefold, namely that the probability of the alternate hypothesis (i.e. that the mother was a double murderer of her children) was not considered, that the probability of such deaths in the population at large was the same of that within families (hence ignoring the possibility of congenital conditions giving rise to a far greater probability) and finally that the deaths were independent events (again, the possibility of heritable conditions will render this assumption false).

These cases show the need for expert evidence following one of the basic rules of evidence which is to not give evidence outside one's area of expertise. The expert witness in the case above clearly did not have the necessary expertise to calculate probabilities that the court needed to gain a true understanding of the case.

STATISTICS

Now let's turn to statistics.

The scientific method

A core precept to using statistics is to get it the right way around:
look at the data first *then* form your hypotheses, NOT form the hypothesis and then select the data to support
Francis Bacon first came up with what we call the scientific method which is essentially that we must first collect data, inspect it for phenomena and draw hypotheses which we then subject to further testing.

What we do not do is to draw hypotheses prior to inspecting the data, then decide to keep only the data that supports the case being presented. This is called being selective with data.

Selectivity with data

A good example of being selective with data is done with "clinical testing" or "clinical trials" of various products. Of course, when something is said to be "clinically tested", that is completely meaningless unless the actual results are given. However, even when the results are given, they are sometimes not representative of all of the data generated in the trials. For example, suppose a company claims that their toothpaste will give "whiter teeth in just two weeks". Now it is easy to find data to support this claim - one gives the product to, say, 20 groups of 20 users and asks them to use it for two weeks then report back (or else be assessed somehow) for the whiteness of their teeth.

Firstly, they can be selective with the data and only report on the group(s) whose results were positive. The fact that say 18 out of the 20 groups showed no improvement would not be reported.

In fact, if improvement in tooth whiteness was essentially random, then with enough trials there would be a result that confirmed the product's effect, even if the majority of other results denied it.

Secondly the trial itself may give results due to atypical user behaviour. The users might be motivated to brush their teeth more thoroughly or more often if they were aware their teeth were going to closely inspected.

Thirdly, if the users were asked for their own perception about the improved whiteness of their teeth, their perceptions may not be accurate and could be motivated by a desire to please the assessors, or embarrassment at having no improvement to report. Also, it is known that people often observe what they expect to observe, i.e. that their perceptions are affected by their expectations.

Falsifiability of hypotheses

When drawing hypotheses for testing, it is an essential part of the scientific method that these hypotheses be testable, and in particular that they be falsifiable, namely that one can think up a test or procedure that can demonstrate that the hypothesis is false (if not necessarily that it is true). Some hypotheses are not easily provable (for example Fermat's last theorem) but would be very easily proved false (if for example someone found integers a, b and c so that a^n + b^n = c^n for n>2, though this won't happen now as the theorem has now been mathematically proved true).

Bertrand Russell, a mathematician, is often quoted in the context of falsifiability, having criticised religion for making itself unfalsifiable:

If I were to suggest that between the Earth and Mars there is a china teapot revolving about the sun in an elliptical orbit, nobody would be able to disprove my assertion provided I were careful to add that the teapot is too small to be revealed even by our most powerful telescopes. But if I were to go on to say that, since my assertion cannot be disproved, it is an intolerable presumption on the part of human reason to doubt it, I should rightly be thought to be talking nonsense. If, however, the existence of such a teapot were affirmed in ancient books, taught as the sacred truth every Sunday, and instilled into the minds of children at school, hesitation to believe in its existence would become a mark of eccentricity and entitle the doubter to the attentions of the psychiatrist in an enlightened age or of the Inquisitor in an earlier time.
(see Wikipedia entry on Russell's teapot).

Artifacts in results

Now let's consider artifacts, which are observed characteristics in data that are due to some unexpected variable. A good example is given in "Outliers" by Malcolm Gladwell (see chapter 1, "the Matthew Effect") where the observation was made that most professional football players were born in the early months of the calendar year. This characteristic turned out to be an artifact of the selection procedure since the selection process grouped children together according to (for example) their year of birth. Clearly those children born earlier in the year are going to be bigger and more developed than those later in the year, and hence are more likely to be selected for teams that get better and longer training. This effect is not restricted to sports, but also occurs in education where children study in year groups, the older ones in the group generally outperforming the younger ones.

Correlation and Causality

Another issues is that correlation does not imply causality. Some interesting examples are that there is a correlation between shark attacks and the consumption of icecreams, or that there is a correlation between global average temperatures and numbers of pirates. In some cases there is an indirect association between characteristics, for example the increase in shark attacks could be attributable to more people swimming during hot weather, and likewise more icecream is consumed during hot weather.

Summary

In this lecture, we have briefly looked at the many potential errors of understanding caused by misapplication of probability and statistics. Next week, Gavin will show indepth an example of how to develop experimental analyses trying to avoid these errors.

Further Reading

There are many books and webpages on this theme. Try some of the following:

The BBC web pages by columnist Michael Blaxland

Books


Last update hla 2009-06-04