2024 Latin America Conference on Lung Cancer (LALCA) - Posters-PP01.21: Evaluating the Accuracy & Reproducibility of ChatGPT Models in Answering Patient’s Lung Cancer Queries

PP01.21: Evaluating the Accuracy & Reproducibility of ChatGPT Models in Answering Patient’s Lung Cancer Queries

Pdf Summary

The study "Evaluating the Accuracy & Reproducibility of ChatGPT Models in Answering Patient’s Lung Cancer Queries" investigates the effectiveness of ChatGPT models (versions 3.5 and 4.0) in providing accurate information in response to lung cancer-related queries. As patients increasingly seek medical information online, accurate responses from AI-driven tools like ChatGPT are crucial to avoid misinformation.<br /><br />The research involved posing 47 lung cancer-related questions to both models, focusing on areas such as general information, diagnosis, treatment, and follow-up. Each question was asked twice to assess response accuracy and reproducibility. The study found that GPT-4.0 delivered more accurate and consistent responses, with 87.2% of answers rated accurate and comprehensive, compared to 76.6% for GPT-3.5. GPT-4.0 also showed better reproducibility in responses (95.7%) compared to GPT-3.5 (89.3%).<br /><br />Despite better performance, both models struggled notably with lung cancer staging, suggesting areas that require improvement. An interesting observation was the empathetic tone adopted by both models, which could enhance patient trust and communication. However, issues with excessive detail or potentially incorrect information persisted, highlighting the models' limitations in critical medical accuracy.<br /><br />The study emphasizes the ongoing need for refinement of AI models to ensure reliable medical information dissemination, suggesting that as more comprehensive data is used in training, further improvements can be expected. It also underscores the importance of blending empathetic communication with factual accuracy in AI systems for healthcare, while recommending continued oversight by medical professionals to mitigate risks of misinformation and support optimal patient outcomes.

Asset Subtitle

Zishan Allibhai

Keywords

ChatGPT

lung cancer

AI models

medical accuracy

patient queries

healthcare AI

GPT-4.0

reproducibility

empathetic communication

misinformation