Thorny Issues and Prickly Solutions: Publication Bias in Meta-Analytic Reviews in the Social Sciences
DOI:
https://doi.org/10.14738/assrj.24.1044Abstract
Two main issues that involve publication bias are the types of studies that get published and the lack of published studies reflecting null results (i.e., the file drawer problem). These issues are of central importance to researchers who conduct systematic (meta-analytic and narrative) reviews of a body of literature because they may result in misleading conclusions being drawn that, in turn, can adversely impact the decisions of policymakers and practitioners who rely on these reviews. Yet without a thorough unbiased critique, a body of literature may not be as evidence-based as the reviewers believe it to be from the extant research analyzed. Therefore, the purpose of this article is to consider issues involved in reporting and evaluating results of meta-analytic reviews, provide an overview of statistical techniques for addressing the file drawer problem (i.e., lack of journals publishing studies with null results), discuss the extent to which the file drawer is actually a “problem,” and offer some solutions.
References
Atkins, D., Best, D., Briss, P. A., Eccles, M., Falck_Ytter, Y., Flottorp, S., . . . GRADE Working Group. (2004). Grading quality of evidence and strength of recommendations. British Medical Journal, 328, 1490.
Banks, G. C., Kepes, S., & Banks, K. P. (2012). Publication bias: The antagonist of meta_analytic reviews and effective policymaking. Educational Evaluation and Policy Analysis, 34, 259_277.
Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088_1101.
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta_analysis. West Sussex, UK: Wiley.
Bradley, M. T., & Gupta, R. D. (1997). Estimating the effect of the file drawer problem in meta_analysis. Perceptional & Motor Skills, 85, 719_722.
Chan, A., Hrobjartsson, A., Haahr, M. T., Gotzsche, P. C., & Altman, D. G. (2004). Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols for published articles. Journal of the American Medical Association, 291, 2457_2465.
Charles, C. M. (2008). Building classroom discipline (9th ed.). Boston, MA: Pearson Education.
Cook, B. G. (2014). A call for examining replication and bias in special education research. Remedial and Special Education, 35, 233_246.
Cook, B. G., & Odom, S. (2013). Evidence_based practices and implementation science in special education. Exceptional Children, 79(2), 135_144.
Cooper, H., & Hedges, L. V. (2009). Potentials and limitations. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta_analysis (pp. 561_572). New York, NY: Russell Sage Foundation.
Cordray, D. S., & Morphy, P. (2009). Research synthesis and public policy. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta_analysis (pp. 473_497). New York, NY: Russell Sage Foundation.
Council for Exceptional Children. (2014). Council for Exceptional Children standards for evidence_based practices in special education. Retrieved from http://www.cec.sped.org/~/media/Files/Standards/Evidence%20based%20Practices%20and%20Practice/CECs%20EBP%20Standards.pdf
Dalton, D. R., Aguinis, H., Dalton, C. M., Bosco, F. A., & Pierce, C. A. (2012). Revisiting the file drawer problem in meta_analysis: An assessment of published and nonpublished correlation matrices. Personnel Psychology, 65, 221_249.
Dickersin, K. (2005). Publication bias: Recognizing the problem, understanding its origins and scope, and preventing harm. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta_analysis: Prevention, assessment and adjustments (pp. 11_34). Chichester, UK: Wiley.
Drotar, D. (2010). Editorial: A call for replications of research in pediatric psychology and guidance for authors. Journal of Pediatric Psychology, 35, 801_805.
Duval, S., & Tweedie, R. L. (2000). A nonparametric “Trim and Fill” method of accounting for publication bias in meta analysis. Journal of the American Statistical Association, 95, 89_98.
Egger, M., Davey Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta_analysis detected by a simple, graphical test. BMC Medical Research Methodology, 315, 629_634.
Elementary and Secondary Education Act [ESEA], P.L. 107_110 (2002).
Evangelou, E., Trikalinos, T. A., & Ioannidis, J. P. (2005). Unavailability of online supplementary scientific information from articles published in major journals. The FASEB Journal, 19, 1943_1944.
Ferguson, C. J., & Heene, M. (2012). A vast graveyard of undead theories: Publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7, 555_561.
Greve, W., Bröder, A., & Erdfelder, E. (2013). Result_blind peer reviews and editorial decisions: A missing pillar of scientific culture. European Psychologist, 18, 286_294.
Heward, W. L. (2009). Exceptional children: An introduction to special education (9th ed.). Upper Saddle River, NJ: Merrill/Prentice Hall.
Howard, G. S., Lau, M. Y., Maxwell, S. E., Venter, A., Lundy, R., & Sweeny, R. M. (2009). Do research literatures give correct answers? Review of General Psychology, 13, 116_121.
International Committee of Medical Journal Editors. (2004). Recommendations for the conduct, reporting, editing, and publication of scholar work in medical journals. Retrieved from http://www.icmje.org/Individuals with Disabilities Education Improvement Act [IDEA], 34 C.F.R. §300.7 (2004).
Ioannidis, J. P. A. (2008). Why most published research findings are false [Editorial material]. PLoS Medicine, 2, 696_701. doi: 10.1371/journal.pmed.0020124
Järvholm, B., & Bohlin, I. (2014). Evidence_based evaluation of information: The centrality and limitations of systematic reviews. Scandinavian Journal of Public Health, 42, 3_10.
Kemp, A. M., Kemp, K. W., Evans, R., Murray, L., Guildea, Z. E. S., Dunstan, F. D. J., & Sibert, J. R. (1998). Diagnosing physical abuse using Bayes’ theorem: A preliminary study. Child Abuse Review, 7, 178_188.
Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108, 480_498.
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., . . . Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta_analyses of studies that evaluate health care interventions: Explanation and elaboration. PLoS Medicine 6(7), e1000100. doi:10.1371/journal.pmed.1000100
Maag, J. W., Losinski, M., & Katsiyannis, A. (2014). Meta_analysis of psychopharmacologic treatment of child and adolescent depression: Deconstructing previous reviews, moving forward. Journal of Psychology and Psychotherapy, 4, 158_167.
McDaniel, M. A., Rothstein, H. R., & Whetzel, D. L. (2006). Publication bias: A case study of four test vendors. Personnel Psychology, 59, 927_953. doi:910.1111/j.1744_6570.2006.00059.x
Moreno, S. G., Sutton, A. J., Ades, A. E., Stanley, T. D., Abrams, K. R., Peters, J. L., & Cooper, N. J. (2009). Assessment of regression_based methods to adjust for publication bias through a comprehensive simulation study. BMC Medical Research Methodology, 9(2). doi: 10.1186/1471_2288_9_2
Mundy, K. M., & Stein, K. F. (2008). Meta_analysis as a basis for evidence_based practice: The question is, why not? Journal of the American Psychiatric Nurses Association, 14, 326_328.
Neuliep, J. W., & Crandall, R. (1990). Editorial bias against replication research. Journal of Social Behavior and Personality, 5, 85_90.
Odom, S. L., Brantlinger, E., Gersten, R., Horner, R. H., Thompson, B., & Harris, K. R. (2005). Research in special education: Scientific methods and evidence_based practices. Exceptional Children, 71, 137_148.
Orwin, R. G. (1983). A fail_safe N for effect size in meta_analysis. Journal of Educational Statistics, 8, 157_159.
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2006). Comparison of two methods to detect publication bias in meta_analysis. Journal of the American Medical Association, 295, 676_680.
Potvin, S., Sepehry, A. A., & Stip, E. (2007). Co_morbid substance_use in schizophrenia: The file drawer effect. Schizophrenia Research, 90, 351_352.
Render, G. F., Nell, J. E., Padilla, M., & Krank, H. M. (1989). What research really shows about assertive discipline. Educational Leadership, 46(6), 72_75.
Reysen, S. (2006). Publication of nonsignificant results: A survey of psychologists’ opinions. Psychological Reports, 98, 169_175.
Rosenberg, M. S. (2005). The file_drawer problem revisited: A general weighted method for calculating fail_safe numbers in meta_analysis. Evolution, 59, 464_468.
Rosenthal, R. (1979). The “file drawer problem” and tolerance for null results. Psychological Bulletin, 86, 638_641.
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005). Publication bias in meta_analyses. West Sussex, England: John Wiley & Sons.
Rotton, J., Foos, P., Van Meek, L., & Levitt, M. (1995). Publication practices and the file drawer problem: A survey of published authors. Journal of Social Behavior and Personality, 10, 1_13.
Schünemann, H. J., Oxman, A. D., & Fretheim, A. (2006). Improving the use of research evidence in guideline development: 6. Determining which outcomes are important. Health Research Policy and Systems, 4, 18.
Shadish, W. R., Doherty, M., & Montgomery, L. M. (1989). How many studies are in the file drawer? An estimate from the family/marital psychotherapy literature. Clinical Psychology Review, 9, 589_603.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2013). P_curve: A key to the file_drawer. Journal of Experimental Psychology: General, 143, 534_547.
Song, F., Parekh, S., Hooper, L., Loke, Y. K., Ryder, J., Sutton, A. J., . . . Harvey, I. (2010). Dissemination and publication of research findings: An updated review of related biases. Health Technology Assessment, 14, 1_220. doi: 10.3310/hta14080
Spring, B., Neville, K., & Russell, S. W. (2012). Evidence_based practice. In V. S. Ramachandran (Ed.), Encyclopedia of human behavior (pp. 86_93). New York, NY: Academic Press.
Sterne, J. A. C., & Egger, M. (2005). Regression methods to detect publication and other bias I meta_analysis. In H.
R. Rotherstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta_analysis: Prevention, assessment and adjustments (pp. 99_110). Chichester, UK: John Wiley & Sons, Ltd. Stroup, D. F., Berlin, J. A., Morton, S. C., Olkin, I., Williamson, G. D., Rennie, D., . . . Thacker, S. B. (2000). Meta_analysis of observational studies in epidemiology: A proposal for reporting. Journal of the American Medical Association, 283, 2008_2012.
Sutton, A. J. (2009). Publication bias. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta_analysis (pp. 435_455). New York, NY: Russell Sage Foundation.
Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003). Adjusting for publication bias in the presence of heterogeneity. Statistics in Medicine, 22, 2113_2126.
U.S. Department of Education, Institute of Education Sciences. (2003). What Works Clearinghouse. Retrieved from http://ies.ed.gov/ncee/wwc/default.aspx
Walster, G. W., & Cleary, T. A. (1970). A proposal for a new editorial policy in the social sciences. American Statistician, 24(2), 16_19.
