Generalizability of Universal Screening Measures for Behavioral and Emotional Risk

Abstract

Data derived from universal screening procedures are increasingly utilized by schools to identify and provide additional supports to students at-risk of behavioral and emotional concerns. As screening has the potential to be resource intensive, effort has been placed on the development of efficient screening procedures, namely brief behavior rating scales. This study utilized classical test theory and generalizability theory to examine the extent to which differences among students, raters, occasions, and screening measures affect the meaningfulness of data derived from universal screening procedures. Teacher pairs from three middle school classrooms completed two brief behavior rating scales during fall and spring screening administrations for all students in their respective classrooms. Correlation coefficients examining interrater reliability, test-retest reliability, and concurrent validity were generally strong. Generalizability analyses indicated that the majority of variance in teacher ratings were attributable to student differences across all score comparisons, but differences between teacher ratings for particular students accounted for relatively large percentages of error variance among student behavior ratings. Although decision studies showed that increasing the number of screening occasions resulted in more generalizable data, the impact of increasing the number of raters resulted in more efficient screening procedures.