Assessment of dimensionality in dichotomously-scored data using multidimensional scaling.

Abstract

The effectiveness of multidimensional scaling (MDS) techniques in recovering the underlying dimensionality of dichotomously-scored data was examined for unidimensional and multidimensional data. Thirty-three data sets of varying numbers of dimensions with differing patterns of item discrimination were generated using a multidimensional latent trait model in a Monte Carlo simulation study. Margin-sensitive measures (agreement, phi, and kappa) and margin-free measures (Φ/ Φ(max), Yule's Q, and the tetrachoric correlation) were used as measures of similarity and the resulting matrices were scaled in one through five dimensions. Values of the stress coefficient, S₁, S₁ by dimensionality plots, and plot configurations were examined to determine the dimensionality of the item set. Principal components analyses (PCAs) of phi and tetrachoric matrices were carried out as a basis for comparison. In addition, MDS and PCA were used to examine a data set comprised of items obtained from the routing tests of the Head Start Measures Battery. Two effects of item discrimination on MDS results were especially noteworthy. First, factors tended to be located equally distant from each other in the MDS space. Items were located closest to the factor for which the primary factor loading occurred. Second, as item discrimination decreased, items tended to be more widely dispersed from their appropriate locations in space. Extra dimensions in the MDS representational space were required for margin-sensitive coefficients to accommodate difficulty effects. Margin-free coefficients generally eliminated difficulty-related dimensions, although occasional problems were noted with the tetrachoric correlation. Analysis of the HSMB revealed that the data were primarily unidimensional, although specific effects due to each subtest were clearly present in the analysis. MDS was found to be a useful technique and its use in conjunction with PCA or factor analysis is recommended.