WCLC 2025 - Posters & ePosters-P1.17.32 Comparing ChatGPT's and Google Gemini'S Responses to Lung Cancer Questions

Pdf Summary

This study compares the responses of two AI systems—OpenAI's ChatGPT and Google's Gemini—to patient questions about lung cancer, focusing on accuracy, reliability, and readability. Given that lung cancer patients often seek information online, where misinformation is common and specialists may be unavailable, the study explores whether AI can provide trustworthy education.<br /><br />Methods involved evaluating AI answers for accuracy, relevance, sufficiency, and the quality of references cited. Responses were assessed by physicians and nurses, and readability was measured using standard tools such as Flesch Reading Ease and Flesch-Kincaid Grade Level.<br /><br />Results showed that while both AIs performed comparably in accuracy, relevance, and sufficiency (no statistically significant differences), ChatGPT’s answers were more consistent and understandable. ChatGPT demonstrated higher consistency with 91.16% of references rated consistent versus Gemini’s 48.70%, with ChatGPT avoiding hallucination—fabricated or misleading content—entirely, whereas Gemini exhibited hallucination in 6.09% of responses. ChatGPT’s references predominantly originated from American sources, while Gemini cited mainly Korean sources. Both used a mix of organizations, hospitals, journals, and news articles as reference types. ChatGPT also achieved better readability scores. For instance, regarding pneumonia or tuberculosis progressing to lung cancer, both scored equally well (4.5/5), but ChatGPT cited reputable American cancer organizations, while Gemini referenced Korean associations and showed some hallucination.<br /><br />The study concludes that ChatGPT offers more consistent, reliable, and understandable information without hallucinations, making it potentially more suitable for patient education on lung cancer than Google Gemini. This highlights ChatGPT's promise as a useful tool for disseminating trustworthy medical information to patients who lack immediate specialist access.

Asset Subtitle

Chanwoo Kim

Meta Tag

Speaker Chanwoo Kim

Topic Global Health, Health Services, and Health Economics

Keywords

AI systems

ChatGPT

Google Gemini

lung cancer

patient education

accuracy

reliability

readability

hallucination in AI

medical information dissemination