Establishing the Validity of the Task-Based English Speaking Test (TBEST) for International Teaching Assistants

Abstract

This dissertation follows an oral language assessment tool from initial design and implementation to validity analysis. The specialized variables of this study are the population: international teaching assistants and the purpose: spoken assessment as a hiring prerequisite. However, the process can easily be applied to other populations and assessment goals.While evaluating the TBEST (Task-Based English Speaking Test) and TAST (TOEFL Academic Speaking Test), I search for a preponderance of evidence for assessment validity that indicate the most appropriate tool for evaluating potential ITAs. The specific evidences of assessment validity that are examined are:1. Evidence of Domain (Content) Validity: Which test, the TBEST or the TAST most closely measures the actual skills needed to be an ITA?2. Evidence of Predictive Criterion Validity: Which test, the TBEST or the TAST, is more valid in predicting ITA teaching success based on end of semester student evaluation (TCEs)?Following the analyses of these points of evidence, the results of a follow-up survey of ITA impressions about the ITA training and evaluating process are reviewed. Reviewing the results of this survey places the language assessment and hiring process recommendations within its larger context, directing attention toward suggestions for improvement of ITA training and evaluating procedures.Over the course of 18 months, 335 ITAs were assessed using the TBEST. 193 ITAs took the TAST prior to taking the TBEST, and those scores are used for correlation analysis. 119 ITAs participated in a follow up survey about their ITA experience.Analysis of domain validity shows that the TBEST is better suited for assessing ITAs than the TAST due to specialized assessment content not present on the more generic TAST. The TBEST is marginally better at predicting teaching success, though the results were statistically insignificant and recommendations are made for a follow-up study. Post-hoc analysis of the discriminative utility of both tests show that the TBEST results show more useful shades of distinction between candidates while the TAST results place the majority of students in a `fair' category which requires secondary interviews to assess teaching ability.