The validity of computer-mediated communicative language tests

Abstract

A recent innovation in language testing involves the use of computer-mediated communicative language tests i.e., assessment of individuals' second language ability from transcripts of their interactions via computer-mediated communication (CMC). Studies have shown that such interactions in the first language involve a hybrid discourse with features of both written and spoken language, which suggests the possibility of making inferences about oral language ability from performance in a CMC environment. The literature to date offers little guidance on this matter. Research on computer-mediated communication has focused on its use in the second language classroom rather than in a testing context while studies of the linguistic and interactional features of second language learners' CMC discourse have mostly been descriptive with little direct comparison of CMC and face-to-face discourse. This study, therefore, examines the validity of making inferences from computer-mediated discourse to oral discourse through a comparison of the performance of 24 third-semester French students on two tests: a computer-mediated communicative French test; and its nearest equivalent format in face-to-face testing, the group oral exam. Using a within-subjects design, counterbalanced for testing condition and discussion topic, the present study focuses on five areas which have important implications for validity: (a) the predictability of ratings of pronunciation on the group oral test; (b) the similarity of scores achieved on the CMC and group oral tests; the presence of similar (c) linguistic and (d) interactional features in the discourse of both tests; and (e) students' attitudes to the two tests. Results show that although scores on the two tests showed no statistically significant difference, students' discourse differed in many respects which would, thus, invalidate any inferences made about oral ability from computer-mediated performance. Moreover, this study raises an important question about the role of computer-mediated communication in promoting second language acquisition since the computer-mediated discourse contained fewer examples of the negotiation of meaning routines that interactionist theories hold to be important to language acquisition.