An Investigation of the Substantive Process Validity of Multistate Bar Examination Items through Verbal Protocol Analysis

Abstract

The dissertation describes a think-aloud study that investigated the internal processes of 25 participants responding to selected items from the Multistate Bar Exam (MBE). The MBE is a nationally administered high-stakes licensure test used in 53 U.S. jurisdictions. The following questions were addressed by the study: Which mental processes are most frequently used in answering the selected items? What is the relative effect on performance of reasoning from legal principles as compared to general reasoning or use of testwiseness principles? When variance in performance due to similarity of responses to expert problem-solving models is accounted for, do divergent mental processes add to prediction? Do minority ethnic examinees use different cognitive processes or commit different types of errors when answering items in selected MBE content areas as compared to majority ethnic examinees?The dissertation discusses the construct of legal reasoning, the method of verbal protocol analysis, and potential method effects of reactivity and veridicality. The study method, including the development and reliability of the rating systems used to quantify the verbal transcripts for analysis are also described.Findings of the study are presented. Using legal principles and rehearsing facts were the most commonly used mental processes. Participants used a number of other mental processes which are described. Using legal principles and avoiding extraneous inferences were associated with increased performance. Drawing premature conclusions and making errors in legal principles negatively affected performance. The effects of different error types and construct-irrelevant thinking such as cue-using strategies are also be reported. Similarity of responses to expert problem-solving models predicted performance on verbalized items, and adding divergent mental processes as predictors did not significantly increase the amount of variance accounted for. No evidence was found in the verbal responses that minority ethnic examinees used different cognitive processes when answering items in selected MBE content areas as compared to majority ethnic examinees; however, a method effect appeared for the minority examinees in the study that complicates interpretation. Some evidence was found of differences in response patterns for minority examinees that could not be explained by the verbal responses.