Lately, I’ve been reading Thinking Fast and Slow by Daniel Kahneman, whose work on judgment and decision-making almost cannot be over-stated in its importance (hey, he won a Nobel Prize for it, and there isn’t even a Nobel for psychology!).
In the book, Kahneman discusses early conversations with his long-time collaborator Amos Tversky and how he came to the realization that even people with years of statistical training and practice can fail in their statistical intuitions.
Here is Kahneman on intuitive statistics:
We had concluded in the seminar that our own intuitions were deficient. In spite of years of teaching and using statistics, we had not developed an intuitive sense of the reliability of statistical results observed in small samples. Our subjective judgments were biased: we were far too willing to believe research findings based on inadequate evidence and prone to collect too few observations in our own research.
And later:
Like most research psychologists, I had routinely chosen samples that were too small and had often obtained results that made no sense. Now I knew why: the odd results were actually artifacts of my research methods. My mistake was particularly embarrassing because I taught statistics and knew how to compute the sample size that would reduce the risk of failure to an acceptable level. But I had never chosen a sample size by computation. Like my colleagues, I had trusted tradition and my intuition in planning my experiments and had never thought seriously about the issue. When Amos visited the seminar, I had already reached the conclusion that my intuitions were deficient…
These confessions of past errors, coming from such an eminent scientist, are powerful reminders to the rest of us to question our intuitive assumptions, use larger samples, and admit to our own faults.