Understanding the oral examination process in professional certification examinations

Abstract

The subjective nature of oral examinations often lead to reliability estimates that are lower than other types of examinations (i.e., written examinations). The potentially biasing individual attributes of examiners (i.e., experience) are of particular concern since the oral examination process depends specifically upon the quality of their assessments. In addition, traditional reliability estimation procedures are not always possible for some oral exams due to the utilization of incomplete measurement designs (i.e., one examiner per candidate) resulting from the inherent high costs and complicated logistics associated with large scale oral examinations. Consequently, the current study attempts to evaluate the quality of one such exam by developing alternative indicators of exam quality using a pre-existing data set. A series of examiner agreement variables were calculated for low, moderate, and high ability candidates and subsequently correlated with each other. A series of exploratory multiple regressions were also used to evaluate the potential impact of several examiner characteristics (experience, gender, specialty, variance of scale use, and fail rate) confined in the data set. Finally, a generalizability (G) study was conducted on a subset of the examination that utilizes a complete measurement design (i.e., two examiners evaluating the same candidate, and all examiners examine all candidates) for lower ability candidates. The G study was then followed by a decision (D) study to determine both the current level of dependability with two examiners, and how much the dependability of the process would improve by adding mure examiners. The results of the current study suggest that evaluating lower ability candidates is different and more difficult than evaluating higher ability candidates. Furthermore, systematic sources of error related to examiners appears to be less or a concern than previously anticipated. Finally, the results of the G-D studies suggest that the current dependability of evaluating lower ability candidates with two examiners could be greatly improved by adding additional examiners to the process.