A critical review of the design and analysis of experiments using replications factors.

Abstract

Explanatory factors are theoretically interesting classifications or variables manipulated by researchers in order to determine their influence on an outcome (commonly called the dependent variable). Replications factors are those whose levels are sampled by the researcher as examples of a general class of cases or as vehicles for a manipulation. Replications factors are used in research in communication and social psychology to avoid confounds and to increase generalizability. This dissertation reports two studies designed to assess research practices related to replications. The 1991 and 1992 volumes of 4 social science journals (Human Communication Research, Communication Monographs, Journal of Communication, and Journal of Experimental Social Psychology) were reviewed. Taken together, Study 1 and Study 2 demonstrated the importance of adapting design and analysis strategies to the requirements of experiments with replications factors. Study 1 demonstrated that replications are important in social research: A large number of studies in the literature included replications, and many more should have done so. Researchers often explicitly acknowledged the role of replications in strengthening their claims through increased generality and control of potential confounds. Study 2 showed that including replications is only part of the solution to problems of generalization. Researchers often chose forms of analysis that failed to account for replications-related variability. Most commonly replications were ignored in analyses, a strategy that results in a loss of control over Type I error rates. Another common choice, treating replications as fixed effects, results in inflated Type I error rates. When replications are treated as a random factor, it is important to remember that both replications and subjects contribute to the power of the study. Two final cautions are given in Chapter 4: (1) Because replications are understood as samples of possible materials, the problem of how to select materials for study needs to be addressed; and (2) because treating replications as random often creates substantial analytic complexity, more careful consideration needs to be given to the design of experiments with replications.