Design and Development of Objective, Structured Management Examinations (OSMES) on Management Skills Among Pharmacy Students

Abstract

The purpose of this study was to design, develop, and administer an Objective, Structured Management Exam (OSME) on management skills for pharmacy students. Pharmacy preceptors for the University of Arizona College of Pharmacy participated in focus groups that identified business, management, and human resource skills needed by pharmacy graduates. Once the skills were identified, gaps were identified for inclusion into the OSMEs. The OSMEs evaluated pharmacy students' performance on four skills: 1) managerial communication (oral and written); 2) conflict resolution; 3) decision-making; and 4) professionalism. The OSME consisted of a role-playing and a writing document. Both were graded using a developed scoring rubric. The role-playing scoring rubric contained 17 skills and the written document scoring rubric contained 8 skills. During the OSME, students interacted with a trained actor and were graded by a judge on their performance. Upon completion of the OSME, students completed a 29-question survey on a) their perceived ability and confidence to accomplish 8 selected skills; b) their opinion about the OSME and suggestions to improve the process in the future; and c) their background characteristics. Many-facet Rasch analysis provided detailed information with which to evaluate content validity and student performance taking into account difficulty of skills, rating scale function of the scoring rubric, judge leniency and severity. Rasch analysis provided detailed information on the scale performance and student ability and confidence. A step-wise linear regression was used to determine if any student characteristics predicted a higher OSME performance score. Ninety-six student pharmacists completed the OSME and ninety-five students completed the questionnaire. No student failed the role-playing scenario and 1 student did not complete the written document. Significant gaps were calculated between the easiest two skills, empty sounds and eye contact. Additionally, the seventeen items did not align with student performance measures. Finally, there were two separate groups of graders. The writing document scoring rubric had poor fit with the model and significant gaps were calculated. Student measures did not adequately align with item difficulty measures. For students' perceived ability, there was no significant change in overall mean student ability scores from before the educational lectures to after the OSME. However, 46% of students (n=44) had a significant change in ability scores. Three significant gaps were calculated between skills a) between decision-making and communication and between communication and active listening on the pre-educational lectures; and b) between decision-making and communication on after the OSMEs. Students appeared confident with their ability on the 8 skills. The overall mean student confidence measure did not significantly change between the three time points: 1) before the practice cases; 2) after the practice cases; and 3) after the OSMEs. However, 46% of students (n=44) had a significant change in their confidence from before the practice cases to after the completion of the OSMEs. Item difficult measures did not appropriately align with student confidence measures, meaning the items were too easy for students. Significant gaps were also calculated: a) between the motivation and communication items and between the communication and active listening items in the before the practice cases responses; b) between the motivation and decision-making items and between the communication and active listening items for the after the practice cases responses; and c) between the motivation and decision-making items; between the decision-making and communication items; and between the communication and active listening items for after the OSME responses. Based on the result of the regression, three characteristics predicted a higher performance score on the role-playing portion of the OSME: 1) previous management experience; 2) previous leadership experience; and 3) the total grade on the educational lecture quizzes. Three characteristics predicted a higher writing document score: 1) age between 26 and 28 years; 2) English as the primary language; and 3) previous pharmacy experience classified as "other" (i.e., pharmacy experience that was not hospital, community, or managed care). This was one of the first studies to develop a role-playing exercise on management skills in pharmacy education. The grading rubrics provided an initial structure for assessing student performance on these management skills. Some changes to the ability questions and the rubrics are suggested in order to improve the content validity. The findings provide the outline for the use of an OSME at schools/colleges of pharmacy as part of their curriculum. As exposure to management scenarios showed higher performance, educators should include these skills in the education of student pharmacists in order to prepare them for a future career in pharmacy.