Trace lines for classification decisions.

Abstract

Referral, placement and retention decisions were analyzed using item response theory in order to investigate several previously unaddressed questions. One question was whether classification decisions could be placed on the latent continuum of ability normally associated with conventional test items. A second set of questions pertained to the existence of differential item functioning (DIF) and testlet functioning (DTF) for the various classification decisions using ethnicity and gender as the grouping variables. Since referral and placement are dependent, two different types of testlets were formed; a referral and placement testlet and a referral, placement, and retention testlet. Test data and educational classification decisions were analyzed for 352 kindergarten children. The resulting "item" parameters were similar to those that might be expected from conventional test items. The a parameters where high and positive for both the individual classification decisions and for the testlets indicating adequate discrimination for the various decisions as a function of ability and that the decisions are related to a single underlying variable. The location parameters for the three decisions were low on the ability continuum. The location parameter for placement was lower than the estimate for referral while the estimate for retention was close to the value obtained for placement. Both testlets were graded and had correspondingly low location parameters. Item information was found to be high in the ability range where decisions are made for both individual decisions and testlets. No DIF was found for the Rasch models but was detected for referral for different ethnic groups using the two-parameter model. Using the Rasch model ignored an important source of DIF contained in the discrimination parameter. DTF was found for the referral and placement testlet when ethnicity was analyzed. Referral decisions for ethnicity were found to be functioning differently for Caucasians versus non-Caucasians. Teachers, in this sample, did not take ability into account when making referral decisions for this group. No DIF was found for placement indicating that evaluation teams did incorporate ability into the decision. Item response theory represents a powerful methodology that could be applied to a variety of new problem types.