A Comparison of Treatment Integrity Assessment Methods for Behavioral Intervention

Abstract

The purpose of this study was to examine the similarity of outcomes from three different treatment integrity (TI) methods, and to identify the method which best corresponded to the assessment of a child’s behavior. Six raters were recruited through individual contact via snowball sampling. A modified intervention component list and 19 video clips were derived from Stahr’s (2005) study, “An Intervention for Children with Autism Spectrum Disorders (ASD) who Have Food Selectivity.” The raters, randomly and evenly assigned to three dyads. Each dyad received an average of six hours training and reached 85% interobserver agreement (IOA) with a 0.60 kappa score. After training, each dyad watched 5 video clips per day and measured both the child’s behavior and TI. The percentages of IOA, kappas, and indices of dependability for assessment of the child’s behavior and TI were analyzed. The data revealed that all raters reached over 80% IOA and the whole interval (WI) and yes/no (Y/N) dyads reached .60 kappa, but the two raters in the Likert-type (LIK) dyad could not reach .60 kappa. The indices of dependability indicated that the six raters consistently observed and rated both the child’s behavior and TI, but there was a discrepancy in scores (i.e., percentages of TI) between the two raters in the two indirect measure dyads (i.e., Y/N and LIK). An analysis of the percentages of total variance showed that the two indirect TI methods may affect the discrepancy between the two raters’ rating scores. A comparison of the three different TI methods and correlation between the child’s behavior and TI were examined using the PASW Statistics 18 software program. There was no significant difference between the WI and the Y/N dyads, while the assessments from the LIK dyad indicated a significant difference from the other two dyads. Both the WI and the Y/N dyads showed correlations between the degree of the child’s behavior and the degree of TI, but there was no significant difference between the two correlation coefficients. Questions about reliability with the indirect TI measures suggest one should be careful in considering these results.