Accuracy of a Chatbot Assessing OTC Medications for Safety in Pregnancy

Abstract

Specific Aims: To assess the accuracy, completeness, and fetal safety implications of ChatGPT-3.5 responses in determining the safety of over-the-counter medications in pregnancy. Methods: Eighty-seven of the top 100 pharmacist-recommended OTC drugs in the U.S. were assessed for safe use in pregnancy using ChatGPT-3.5. A standard prompt was input into ChatGPT-3.5, and responses were recorded in a spreadsheet. Two groups independently rated the responses compared to UpToDate on accuracy, completeness, and safety using a 5-point Likert scale. After independent evaluations, the groups discussed the findings to reach a consensus, with a 3rd investigator giving final ratings. Descriptive statistics were used to analyze the results. Results: For accuracy, the median score was 5 (IQR 5,5). For completeness, the median score was 4 (IQR 4,5). For safety, the median score was 5 (IQR 5,5). Conclusions: Overall, ChatGPT scored favorably in accuracy, completeness, and safety. Despite high median scores in these categories, ChatGPT failed to acknowledge situations in which certain medications should be avoided in pregnancy due to the risk of serious complications. Currently, we do not recommend the use of ChatGPT when seeking medication safety information in pregnancy. Conversely, a chatbot may be developed as a reputable and comprehensive source in the future.