Where Fisher, Neyman and Pearson went astray: On the logic (plus some history and philosophy) of Statistical Tests
DOI:
https://doi.org/10.14738/assrj.58.4867Abstract
Every scientific endeavour consists of (at least) two components: A hypothesis on the one hand and data on the other. There is always a more or less abstract level - some theory, a set of concepts, certain relations of ideas - and a concrete level, i.e., empirical evidence, experiments or some observations which constitute matters of fact. The focus of this contribution is on elementary models connecting both levels that have been very popular in the social sciences - statistical tests. Going from simple to complex we will examine four paradigms of statistical testing (Fisher, Likelihood, Bayes, Neyman & Pearson) and an elegant contemporary treatment. In a nutshell, testing is an easy problem that has a straightforward mathematical solution. However, it is rather surprising that the statistical mainstream has pursued a different line of argument. The application of the latter theory in psychology and other fields has brought some progress but has also impaired scientific thinking.
References
Ackermann, R. (1989). The New Experimentalism. Brit. J.Phil. Sci. 40, 185-190.
Aldrich, J. (1997). R. A. Fisher and the making of maximum likelihood. Stat. Sci. 12(3), 162-176.
Arbuthnot, J. (1710). An Argument for Divine Providence, taken from the Constant Regularity Observ’d in the Births of Both Sexes. Phil. Trans. R. Soc. 27, 186-190.
Bakan, D. (1970). The Test of Significance in Psychological Research. Chapter 25 in Morrision, D.E. and Henkel, R.E. (eds.). The Significance Test Controversy. Aldine Publishing Company, Chicago, IL, 231-251.
Barnett, V. (1999). Comparative Statistical Inference. (3rd ed.) Wiley, New York.
Berkson, J. (1942). Tests of Significance Considered as Evidence. JASA 37, 325-335.
Berthold, M.R.; and D.J. Hand (eds., 2003). Intelligent Data Analysis. (2nd ed.) Springer, Berlin.
Bookstein, F.L. (2014). Measuring and reasoning. Numerical inference in the sciences. Cambridge Univ. Press, New York.
Burnham, K.P.; and D.R. Anderson (2002). Model Selection and Multimodel-Inference. A Practical Information-Theoretic Approach. (2. ed.) Springer, New York.
Chow, Y. S.; and Teicher, H. (1997). Probability Theory. Independence, Interchangeability, Martingales. Springer, New York: Springer Texts in Statistics.
Cohen, J. (1988). Statistical Power Analysis for the Social Sciences. (2. ed.) Erlbaum, Hillsdale, NJ.
Cohen J. (1992): A power primer. Quant. Meth. Psychol. 112, 155–159.
Cornfield, J. (1966). Sequential Trials, Sequential Analysis and the Likelihood Principle. American Statistician 20(2), 18-23.
Cover, T.M. und J.A. Thomas (2006). Elements of Information Theory. Wiley, New York, 2nd ed.
Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25, 7-29.
Dienes, Z. (2011). Bayesian versus orthodox statistics: Which side are you on? Perspectives on Psychological Science, 6, 274-290.
Edwards, A. W. F. (1922). Likelihood. Johns Hopkins University Press, Baltimore, MD., 2nd ed.
Edwards, W.; Lindman, H.; and J. Savage (1963). Bayesian statistical inference for psychological research. Psychological Review 70, 193-242.
Efron, B.; and Gous, A. (2001). Scales of Evidence for Model Selection: Fisher versus Jeffreys. In: Lahiri, P. (ed.) IMS Lecture Notes 38 on Model Selection, 210-256.
Ellis, P.D. (2010). The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. Cambridge University Press, New York.
Fetzer, J.H. (ed., 2001). The Philosophy of Carl G. Hempel. Studies in Science, Explanation, and Rationality. Oxford University Press, New York.
Fisher, R.A. (1929). The Statistical Method in Psychical Research. Proceedings of the Society for Psychical Research 39, 189-192.
Saint-Mont, U. (2018). Where Fisher, Neyman and Pearson went astray: On the logic (plus some history and philosophy) of Fisher, R.A. (1935/1966). The Design of Experiments. Cited according to the 8th ed., Hafner Publishing Company, New York.
Fisher, R.A. (1935). The Logic of Inductive Inference. J. of the Royal Stat. Soc. 98, 39-54.
Fisher, R. A. (1955). Statistical Methods and Scientific Induction. J. of the Royal Stat. Soc., Ser. B17(1), 69-78.
Fisher, R.A. (1956/73). Statistical Methods and Scientific Inference. Hafner Publishing Company, New York. Citet according to the 3rd edition, 1973.
Fisher, R.A. (2003). Statistical methods, experimental design, and scientific inference: a re-issue of statistical methods for research workers, the design of experiments, and statistical methods and scientific inference. Oxford Univ. Press, Oxford.
Gelman, A; Carlin, J.B.; Stern, H.S.; Dunson, D.; Vehtari, A.; and Rubin, D.B. (2013). Bayesian Data Analysis (revised ed.) Chapman & Hall/CRC, New York.
Ghosh, J.K. (ed., 1988). Statistical Information and Likelihood. A Collection of Critical Essays by Dr. D. Basu. Springer, New York: Lecture Notes in Statistics.
Gigerenzer, G. (2004). Mindless Statistics. The J. of Socio-Economics 33, 587-606.
Gigerenzer, G.; Krauss, S.; and Vitouch, O. (2004). The Null Ritual. What you always wanted to know about significance testing but were afraid to ask. Chapter 21 in: Kaplan (ed.) The Sage Handbook of Quantitative Methodology for the Social Sciences. Sage, Thoudsand Oaks,
Good, I.J. (1988). The Interface between Statistics and Philosophy of Science. Statistical Science 3(4), 386-412.
Goodman, S.N.; and Royall, R. (1988). Evidence and Scientific Research. American J. of Public Health 78(12), 1568-1574.
Greenland, S.; Senn, S.J.; Rothman, K.J.; Carlin, J.B.; Poole, C.; Goodman, S.N.; and D.G. Altman (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur. J. Epidemiol. 31, 337–350.
Guttman,L.(1985).TheIllogicofStatisticalInferenceforCumulativeScience.Applied Stochastic Models and Data Analysis 1, 3-9.
Haig, B.D. (2016). Tests of statistical significance made sound. Educational and Psychological Measurement 35, 1-18.
Heckman, J.J. (2005). The Scientific Model of Causality. Socio. Methodology 35, 1-162.
Hedges, L.V.; and Olkin, I. (1985). Statistical Methods for Meta-Analysis. Academic Press, Orlando.
Hodges, J. S. (1990). Can / May Bayesians Do Pure Tests of Significance? In: Geisser, S.; Hodges, and Press, J. (eds.) Bayesian and Likelihood Methods in Statistics and Econometrics (vol. 7). Essays in Honor of George A. Barnard. North-Holland Publishing Company, Amsterdam: Studies in Bayesian Econometrics and Statistics, 75-90.
Hoeffding, W. (1965). Asymptotically optimal tests for multinomial distributions. Annals Math. Statistics 36, 369-408.
Howson, C. (2003). Hume’s problem: Induction and the justification of belief. Clarendon Press, Oxford.
Howson, C.; and Urbach, P. (2006). Scientific Reasoning. The Bayesian Approach. (3rd ed.) Open Court, Chicago and La Salle, IL.
Hubbard,R.(2004).Alphabetsoup.Blurringthedistinctionsbetweenp’sandα’sinpsychological research Theory & Psychology 14(3), 295-327.
Hubbard, R.; and Lindsay, R.M. (2008). Why P Values are Not a Useful Measure of Evidence in Statistical Significance Testing. Theory & Psychology 18(1), 69-88.
Hubbard, R.; and Ryan, P.A. (2000). The Historical Growth of Statistical Testing in Psychology - and its Future Prospects. Edu. & Psych. Measurement 60(5), 661-681.
Hurlbert, S.H., and C.M. Lombardi (2009). Final collapse of the Neyman-Pearson decision theoretic framework and rise of the neoFisherian. Annales Zoologici Fennici, 46, 311-349.
Jaynes, E.T. (2003). Probability Theory. The Logic of Science. Edited by Bretthorst, G. L. Cambridge University Press, Cambridge.
Jeffreys, H. (1939). Theory of Probability. Clarendon Press, Oxford.
Johnstone, D. J. (1986). Tests of Significance in Theory and Practice. The Statistician 35(5), Jones, L.V. (ed., 1986). The collected works of J.W. Tukey, Vol. III & IV: “Philosophy and Principles of Data Analysis.” Chapman & Hall, London.
Keiding, N. (1995). Test 4(2), 241-242. Commentary on Cox, D.R. (1995). The Relation between Theory and Application in Statistics. (with discussion) Test 4(2), 207-261.
Kotz, S.; and Johnson, N.L. (1993). Breakthroughs in Statistics. Vol. I: Foundations and Basic Theory. Spinger, New York.
Kullback, S.; and Leibler, R.A. (1951). On Information and Sufficiency. Annals of Mathematical Statistics 22(1), 79-86.
Lehman,E.L.(1993).IntroductiontoNeymanundPearson(1933).In:KotzandJohnson(1993),
Li, M.; and Vit´anyi, P. (2008). An Introcudtion to Kolmogorov Complexity and its Applications. (3rd ed.) Springer, New York.
Lindley, D.V. (1957). A statistical paradox. Biometrika 44, 187–192.
Lipton, P. (2004). Inference to the Best Explanation. (2nd ed.) Routledge, London.
Mayo,D.G.(1996).ErrorandtheGrowthofExperimentalKnowledge.The University of Chicago Press, Chicago, IL.
Saint-Mont, U. (2018). Where Fisher, Neyman and Pearson went astray: On the logic (plus some history and philosophy) of
McPherson, G. (1989). The Scientist’s View of Statistics - a Neglected Area. J. of the Royal Stat. Society 152, 221-240.
Meehl, P.E. (1967). Theory-Testing in Psychology and Physiscs: a Methodological Paradox. Philosophy of science 34, 103-115.
Meehl, P.E. (1978). Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology. J. of Consulting and Clin. Psych. 46, 806-834.
Meehl, P.E. (1990). Apraising and Amending Theories: The Strategy of Lakatosian Defence and Two Principles that Warrant it. Psychological Inquiry 1(2), 108-141.
Meehl, P.E. (1997). The Problem is Epistemology, not Statistics: Replace Confidence Intervals and Quantify Accuracy of Risky numerical Predictions. In: Harlow, L.L.; Mulaik, S.A.; and Steiger, J.H. (eds., 1997). What if there were no Significance Tests? Erlbaum, London, 393-425.
Naaman, M. (2016). Almost sure hypothesis testing and a resolution of the Jeffreys-Lindley paradox. Electronic Journal of Statistics 10, 1526–1550.
Neyman, J. (1955). The Problem of Inductive Inference. Communications on Pure and Applied Mathematics VIII, 13-46.
Neyman, J. (1961). The Silver Jubilee of My Dispute with Fisher, Journal of the Operations Research Society of Japan, 3, 145-154.
Neyman, J. (1977). Frequentist Probability and Frequentist Statistics. Synthese 36, 97-131.
Neyman, J.; and Pearson, E. S. (1933). On the Problem of the Most Efficient Tests of Statistical Hypotheses. Philosoph. Trans. Royal Soc. London A 231, 289-337. Cited according to Kotz and Johnson (1993), 73-108.
Oakes, M. (1986). Statistical Inference: A Commentary for the Social and Behavioral Sciences. Wiley, New York.
Ottenbacher, K.J. (1996). The Power of Replications and Replications of Power. The American
Pearl, J. (2009). Causality. Models, Reasoning and Inference. (2nd ed.) Cambridge Univ. Press.
Pearson, E. S. (1938). Student as Statistician. Biometrika 30, 210-250.
Pearson, E. S. (1955). Statistical Concepts and their Relation to Reality. J. of the Royal Statistical Society, Ser. B 17(2), 204-207.
Peto, R.; Pike, M. C.; Armitage, P.; Breslow, N. E.; Cox, D. R.; Howard, S. V; Mantel, N.; McPherson, K.; Peto, J.; and Smith, P. G. (1976). Design and Analysis of Randomized Clinical Trials requiring prolonged Oberservation of each Patient, I: Introduction and Design. British J. of Cancer 34, 585-612.
Popper, K.R. (1959). The Logic of Scientific Discovery. Martino Fine Books (Reprint, 2014).
Popper, K.R.; and Miller, D.W. (1983). A proof of the impossibility of inductive probability. Nature 302, 687-688.
Rissanen, J. (2007). Information and Complexity in Statistical Modelling. Springer, New York.
Robbins, H. (1970). Statistical methods related to the law of the iterated logarithm. The Annals of Mathematical Statistics 41, 1397-1409.
Robert, C.P. (2007). The Bayesian Choice. From Decision-Theoretic Foundations to Computational Implementation. (2nd ed.) Springer, Berlin.
Rosnow, R.L.; and Rosenthal, R. (1989). Statistical Procedures and the Justification of Knowledge in Psychological Science. American Psychologist 44, 1276-1284.
Royall, R. M. (1986). The Effect of Sample Size on the Meaning of Significance Tests. American
Royall, R.M. (1991). Ethics and Statistics in Randomized Clinical Trials. Stat. Sci. 6(1), 52-88.
Royall, R. M. (1997). Statistical Evidence. A Likelihood Paradigm. Chapman & Hall, London.
Royall, R. M. (2000). On the Probability of Observing Misleading Statistical Evidence (with discussion) J. of the American Statistical Association 95, 760-780.
Salmon, W.C. (1989). Four Decades of Scientific Explanation. University of Minnesota Press, Minnesota, MN.
Salsburg, D.S. (1985). The Religion of Statistics as practiced in Medical Journals. The American
Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative konowledge in psychology. American Psychologist 47(10), 1173-1181.
Schmidt, F. L. (1996). Statistical Significance Testing and Cumulative Knowledge in Psychology: Implications for Training of Researchers. Psych. Meth. 1(2), 115-129.
Spanos, A. (2014). Recurring controversies about P values and confidence intervals revisited. Ecology 95, 645-651.
Spielman, S. (1974). The Logic of Tests of Significance. Phil. of Science 41(3), 211-226.
Stigler, S.M. (1986). The History of Statistics. The Measurement of Uncertainty before 1900. The Belknap Press of Harvard University Press, Cambridge, MA.
Tukey, J.W. (1977). Exploratory Data Analysis. Addison-Wesley: Behavioral Science; Quantitative Methods.
Tukey, J.W. (1991). The Philosophy of Multiple Comparisons. Stat. Sci. 6(1), 100-116.
Tversky, A.; and Kahneman, D. (1971). Belief in the Law of Small Numbers. Psychological Bulletin 76, 105-110.
van Dyk, D.A. (2014). The role of statistics in the discovery of a Higgs Boson Annu. Rev. Stat. Appl. 1, 41-59.
Walker, S. (2003). On sufficient conditions for Bayesian consistency. Biometrika 90(2), 482-488.
Walker, S. (2004). New approaches to Bayesian consistency. Ann. Stat. 32(5), 2028-2043.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2018 Advances in Social Sciences Research Journal
This work is licensed under a Creative Commons Attribution 4.0 International License.
