How good are we? A meta-analytic study of effect sizes in medicine

Abstract

Background. Although Cohen's convention regarding small (∼0.2), medium (∼0.5), and large (∼0.8) effect sizes (ES) that originated in the socio-behavioral sciences approximately 40 years ago has been used heuristically in medicine as well, the extent to which it characterizes the range of ES for various medical therapies remains unknown. Objectives. (1) To calibrate the robustness of current interventions in medicine using different ES indices as the underlying metric, and (2) to examine whether the efficacy of therapeutic interventions in medicine have changed over the years. Methods. Three complementary studies have been conducted. Study I, which provides an indication of the range of ES in medicine currently, is a meta-meta analysis that summarizes 91 different meta-analyses of various treatments, from medicine to surgery to psychology. Study II used a pool of 250 different ES computed from 52 original clinical trials published over the last 25 years to examine time trends associated with ES. Study III used traditional and cumulative meta-analyses of 76 trials representing 30 different treatments for sepsis. Number-needed-to-treat (NNT) was calculated and adjusted for different control death rates to assess systematically objectives (1) and (2). Results. Study I: The mean ES for continuous variables was d 0.50 (SD 0.35, 95%CI 0.32-0.69) and for dichotomous variables odds ratio 1.51 (SD 1.39, 95%CI 0.9-2.13; p > 0.05). Study II: A parsimonious theory-specified model accounted for 93.2% of the variance associated with ES estimates over the years. ES was statistically significant correlated with many design features but not with year of publication. Study III: Neither traditional meta-analysis nor cumulative meta-analysis found most treatment categories for sepsis to be effective. Whereas the pooled relative risk estimate for the entire cohort of almost 22,000 patients was statistically significant with very narrow confidence interval (RR 0.92; 95%CI 0.88-0.96; p = 0.00028), the NNT adjusted analysis had a wide range, including the potential for harm. Conclusion. Cohen's convention applies well to medicine. Most therapeutic interventions in this study had a small to moderate ES, indicating they have not changed dramatically over the years. These findings suggest that the ever-increasing biomedical knowledge does not result in ever-improving therapeutic efficacy.