ArabiPro and ArabiBot: Arabic Language Learning with Automatic Speech Recognition and Conversational AI

Abstract

This dissertation investigates end-to-end Automatic Speech Recognition (ASR) in relation to second language acquisition. It explores the application of end-to-end ASR in Computer-Assisted Pronunciation Training (CAPT) systems and conversational AI agents, specifically chatbots, for Arabic second language acquisition. Therefore, it presents three distinct implementations subjected to empirical evaluation through two pretest-posttest empirical studies. ArabiPro, a CAPT system, harnesses a phoneme-based, fine-tuned Wav2vec 2.0 (Baevski et al., 2020) speech model. It delivers explicit, corrective feedback on pronouncing the feminine marker /taa/ (i.e., final tied /taa/) in three Arabic phrases: the construct state, the definite noun-adjective, and the indefinite noun-adjective phrases. The pronunciation of the final tied /taa/ varies between /h/ and /t/ based on the type of phrase. ArabiBot is a rule-based, speech-based chatbot that engages users in diverse conversations, assuming the role of a questioner with the user responding. ArabiCaf, meanwhile, is an implementation of several features derived from the Complexity, Accuracy, and Fluency (CAF) (Skehan et al., 1998; Ellis et al., 2003; Ellis and Barkhuizen, 2005, among others) model for articulation characteristics. The first empirical study uses ArabiPro to measure the effectiveness of providing immediate corrective feedback on pronouncing the final tied /taa/ in three Arabic phrases. The objective is to assess the efficacy of explicit corrective feedback in enhancing the pronunciation of specific complex structures for novice and intermediate Arabic learners. Two linear mixed models are fitted to discern the influence of a specified intervention, the explicit feedback on the pronunciation of the final tied /taa/, and score the changes between the pretest and the posttest stages. This study revealed two findings. First, it indicated that the explicit, corrective feedback helped second-language learners improve their pronunciation of the three phrases. Second, the analysis unveiled that, in both the pretest and posttest stages, participants were more knowledgeable of the definite noun and adjective structure, as well as the indefinite noun and adjective structure, compared to the construct state. In the second empirical study, ArabiBot and ArabiCaf are used to assess the extent to which learners’ conversational skills improved during the experiment. We administered free-speech tests before (pretest) and after (posttest) the intervention for five days. This approach allows us to measure the effectiveness of incorporating chatbot-mediated interactions in fostering conversational proficiency among learners and, thus, provides empirical evidence to support or refute the proposed hypothesis. This experiment is followed by a questionnaire to collect learners’ opinions about their experience with the chatbot. Learners’ speech is analyzed using the ArabiCaf regarding lexical complexity and fluency features. Seven linear mixed models are fitted to account for any intervention. Findings suggest that students’ speaking performance was significantly better with the chatbot, supported by learners’ positive experience while conversing with the ArabiBot. The importance of the systems devised from scratch within this dissertation extends beyond mere theoretical exploration. It sets the groundwork for more sophisticated Arabic language learning technologies, particularly in the burgeoning Large Language Models (LLMs) era.