I think it's important to realise from the outset that statistics is not maths.
Here's a very good post on the topic.
Statistics Is Not Math
“Here is a column of a couple of dozen numbers. From them, calculate the mean and standard deviation. When you are finished—it should take you a good fifteen to twenty minutes—report back to me.”
So goes the instruction in many, if not most or even all, undergraduate statistics courses across the land. Part of the reason this is so is nostalgia. Professors learn statistics in a certain way; they naturally teach it in that same way. Running through endless examples of plugging numbers into calculators and pressing certain buttons are what they did while growing up and, by God, what was good enough for them, is good enough for students. So what if the students forget why they’ve done it?
This inertia is a quirk of human nature and is common in any field of instruction: its limitations are overcome easily by all serious students. Far more restraining, however, are the pernicious effects of the belief that statistics is a branch of mathematics.
Statistics is not math; neither is probability. It is true that math has proven unreasonably effective in understanding statistics, but it is not, as Wigner suggested for the relationship between physics and mathematics, the best, or at least not the sole, language to describe its workings.
That language is philosophical. Just think: statistics self-named purpose is to compile evidence to use in quantifying uncertainty in (self-selected) hypotheses. How this evidence bears on the hypotheses may be best described mathematically, but why it does so cannot be. It also cannot be that because statistics uses so much math that it is math. This would be equivalent to saying that accounting is a branch of mathematics because it too rests on multitudinous calculations.
Statistics rightly belongs to epistemology, the philosophy of how we know what we know. Probability and statistics can even be called quantitative epistemology. Our axioms concern themselves with what probability means; that is, of the interpretation of uncertainty. But we abandon those axioms too quickly, choosing instead to follow the path of equations, nearly always skimping on what those equations actually mean.
To master probability and statistics requires mastering a great chunk of math. But we begin to go wrong when we mindlessly apply equations in inappropriate situations because of the allure of quantification. Worse, we routinely reify the mathematics; for example, p-values positively wriggle with life: to most, they are mysterious magic numbers. Equations become a scapegoat: when what was supposed to have been true or likely because of statistical calculation turns out to be false and even ridiculous, the culprits who touted the falsity point the finger of blame at the math.
Philosophy sharpens the mind. It teaches us to recognize and eliminate sloppy thinking and writing, two elements rife in our field. If people spent more time thinking about what they are saying and doing, much error would be reduced or eliminated.
I’ll give just one example. Ask any statistician for the definition of a confidence interval. Chances are overwhelming that he’ll tell you something false. But he’ll believe the falsity, and because of that, he’ll go on using confidence intervals, interpreting them wrongly, and he’ll justify their use because, well, because they are being used. The reason this behavior persists is sloppy writing on the part of textbook writers: flaws which could have been largely eliminated had the authors had some philosophical training.
What is a confidence interval? It is an equation, of course, that will provide you an interval for your data. It is meant to provide a measure of the uncertainty of a parameter estimate. Now, strictly according to frequentist theory—which we can even assume is true—the only thing you can say about the CI you have in hand is that the true value of the parameter lies within it or that it does not. This is a tautology, therefore it is always true. Thus, the CI provides no measure of uncertainty at all: in fact, it is a useless exercise to compute one.
But ask your neighborhood statistician and you will hear words about “95% confidence”, about “long runs”, about “other experiments”, etc., etc. These poorly chosen phrases are a bar to clear thinking. They make the utterer forget that all he can say is some tautological, and therefore trivial, truth. He has concentrated on the math, making sure to divide by n minus one in the appropriate place, etc., and has not given any time to consider why the calculation exists.
Much nonsense in the last century has been promulgated because of sloppy thinking in statistics. It is time to stop thinking about the mathematics and more on the meaning.
Statistics rightly belongs to epistemology, not mathematics.
Topic: Book recommendations & reviews