Human and Machine Judgment of Non-Native Speakers’ Speech Proficiency

Abstract

This dissertation aims to investigate how human listeners judge the speech proficiency of non-native speakers and explore whether a computational model can approximate human listeners' judgments of proficiency. To be specific, this dissertation investigates the relationship between the timing of speech (i.e., speech rhythm) and human listeners' ratings of accentedness, fluency, and comprehensibility. It examines whether a computational model can measure accentedness, fluency, and comprehensibility of the speech in the same way as human listeners do. In the first study, English native listeners were asked to rate accentedness, fluency, and comprehensibility of speech produced by native and non-native speakers of English. Various speech rhythm measures, including the speech rate, the variability of vowels' durations, and the duration of pauses were calculated. The correlation between those rhythm measures and listeners' ratings of the speech were examined. The results showed that there were significant correlations between speech rhythm and perceived accentedness, fluency, and comprehensibility. In the second study, I built an automatic proficiency judgment model using three features (wav2vec, acoustic, speech rhythm) and collected human ratings of accentedness, fluency, and comprehensibility. The results showed that the model can approximate human judgments of speech proficiency. In the third study, I examined whether the automatic proficiency judgment model of the second study can generalize to new data and still provide accurate judgments, by replicating previous work on L2 proficiency and acoustic characteristics using new data and machine-predicted scores. The results showed that I could replicate the previous findings even though I used different data with predicted proficiency scores from the automatic proficiency judgment model. Furthermore, the model produced results which matched previous findings on degree of accentedness using different acoustic measures some of which were not used in the training of the model. One additional result found in these studies is that the rhythm measures reflect not only the overall timing of the speech but also the speech errors and pauses that the speakers produced, which can reflect speakers' proficiency. In addition, the results from the automatic proficiency judgment model suggest that segmental acoustic features have a stronger effect on the judgment of accentedness, while speech rhythm affects judgments of fluency more strongly. This finding may have implications for foreign language teaching. This dissertation also suggests that an automatic proficiency model can be generalized for different tasks, so it can be used to determine the proficiency of non-native speakers automatically when conducting other linguistic research, for example for automatically determining accentedness of stimuli for an experiment. To sum up, the results of this dissertation show that when the speech is accented, it can still be perceived as fluent and comprehensible since various characteristics of the speech signal affect accentedness, fluency, and comprehensibility differently, and this is supported by both human and machine judgments.