Last time I mentioned some papers on the historical role of statistics in medicine. Here they are, by Donald Mainland:
- “Statistics in Clinical Research: Some General Principles” (1950) [journal, pdf]
- “The Rise of Experimental Statistics and the Problems of a Medical Statistician” (1954) [journal, pdf]
I’ve just re-read them and they are excellent. What is the heart of statistical thinking? What are the most critical parts of (applied) statistical education? At just 8-9 pages each, they are valuable reading, especially as a gentle rejoinder in this age of shifting fashions around Data Science, concerns about the replicability crisis, and misplaced hopes that Big Data will fix everything painlessly.
Some of Mainland’s key points, with which I strongly agree:
- The heart of statistical thinking concerns data design, even more so than data analysis. How should we design the study (sampling, randomization, power, etc.) in order to gather strong evidence and to avoid fooling ourselves?
…the methods of investigating variation are statistical methods. Investigating variation means far more than applying statistical tests to data already obtained. … Statistical ideas, to be effective, must enter at the very beginning, i.e., in the planning of an investigation.
- Whenever possible, a well-designed experiment is highly preferred over poorly-designed experimental or observational data. It’s stronger evidence… and, as industry has long recognized, it cuts costs.
In all the applied sciences, inefficient or wrong methods of research or production cause loss of money. Therefore, sound experimentation was profitable; and so applied chemistry and physics adopted modern biological statistics while academic chemists, physicists, and even biologists were disregarding the revolution or resisting it, largely through ignorance.
- Yes, of course you can apply statistical methods to “found” data. Sometimes you have no alternative (macroeconomics; data journalism); sometimes it’s just substantially cheaper (Big Data). But if you gather haphazard data and merely run statistical tests after the fact, you’re missing the point.
These unplanned observations may be the only information available as a basis for action, and they may form a useful basis for planned experiments; but we should never forget their inferior status.
…a significance test has no useful meaning unless an experiment has been properly designed.
- Statistical education for non-statisticians spends too little time on good data design, and too much on a slew of cookbook formulas and tests.
…the increase in the incidence of tests—statistical arithmetic—has continued, and so also, very commonly, has the disregard of the more important contribution of statistics, the principles and methods of sound, economical experimentation and valid inference… Another obvious cause is the common human tendency to use gadgets instead of thought. Here the gadgets are the arithmetical techniques, and the statistical “cookbooks” that have presented these techniques most lucidly, without primary emphasis on experimentation and logic, have undoubtedly done much harm.
- Statistical education for actual applied statisticians also spends too little time on good data design, and too much on mathematics.
The most important single element in the training (and continuous education) of any statistician is practical experience—experience of investigations for which he himself is responsible, with all their difficulties and disappointments.
…even if a mathematician specializes in the statistical branch of mathematics, he is not thereby fitted to give guidance in the application of the methods.
- As an investigator, you must understand statistical reasoning yourself. You can (and should!) hire an applied statistician to help with the details of study design and data analysis, but you must understand their viewpoint to benefit from their help.
If, however, he is acquainted with the requirements for valid proof, he will often see that what looked like evidence is not evidence at all…
Of course study design is not all of statistics. But it’s a hugely important component that seems underappreciated in modern statistics curricula (at least in my experience). Even if it’s not the sexiest area of current research, I’m surprised my PhD program at CMU completely omits it from our education. (The BS and MS programs here do offer one course each. But I was offered much deeper courses in my MS at Portland State, covering design of experiments and also of survey samples.)
As a bonus, Mainland also offers advice on starting and running a statistical consulting unit. It’s aimed at medical scientists but useful more broadly.
I would quote more, but you should really just read the whole thing. Then comment to tell me why I’m wrong 🙂