Method variance in the social sciences

Abstract

A preliminary taxonomy has been developed that differentiates between trait characteristics, the situation, manifest and subtle methods and interaction effects. The proposed taxonomy divides method into manifest and subtle categories. The obvious, surface characteristics of method are considered manifest, while the deeper structures of method that are not usually seen are considered subtle. Seven manifest method categories are described: stimulus format, response format, response categories, raters, whether the measure is direct or summative, rating the stimulus or the response, and opaque or transparent measures. Numerous subtle method categories can also be seen within the method rather than on its surface. These include semantic or verbal characteristics, direction of wording, measures of amount, ability, latency of response, possessions, situational context, associations, and behavior. Other method categories include report of others reactions, body symptoms, and time frame. Seven item level methods were tested in this study in the context of measurement of Introversion-Extraversion (IE). They included direct questions, direction of wording, situational context, time frame, report of others reactions, preference (semantic or verbal structure), and behavior. Three of these methods (preference, behavior, and direction of wording) introduced substantial method variance in the measurement of IE, and there were also several sizable trait*method interactions: direction of wording, time frame, and preference. Generalizability theory analysis (GT) proved to be quite useful in estimating method effects and interactions. By comparing the findings from GT analysis to confirmatory factor analysis (CFA) results, it became clear that there were problems with the CFA results that could not be ignored. Although generalizability analysis is limited in its ability to provide estimates of the trait and method contributions for individual measures, unfortunately, it informs us that the estimates provided by CFA are probably erroneous.