LEVERAGING RETRIEVAL-AUGMENTED GENERATION FOR ACCURATE ADVICE REGARDING HEALTH AND HARSH WEATHER

Abstract

This paper explores the challenges of hallucination and misinformation spread by large language models (LLMs), focusing on the importance of their accuracy and reliability for use in health and inclement weather information dissemination. To address these issues, we introduce the AZX chatbot, which utilizes a retrieval-augmented generation (RAG) process. This method involves compiling trusted sources into a dense passage retrieval database of vectorized text documents to support informed response generation. The chatbot is designed to utilize these sources in providing its response in order to provide users with transparency regarding its logic and enable them to verify its responses. This chatbot, alongside ChatGPT and Gemini, was assessed in its responses to various prompts related to shelter, inclement weather alerts, health hazards, and disease. It slightly outperformed these baselines for prompts related to shelter and health hazards, but only succeeded on about half of the prompts in these categories, indicating the need for further improvement. However, it excelled and far outperformed the baselines for prompts related to weather and disease. Further investigation should look into improving the formerly described areas, as well as methods for efficiently updating the sources within the database.