Introduction
Patients with cirrhosis often lack knowledge about how to manage and prevent complications related to their disease. ChatGPT, a natural language processing model developed by OpenAI, could offer support by providing clear and understandable answers to patients. While it has been used in medical applications, such as test preparation and report writing, few studies have evaluated its ability to accurately answer clinical questions specific to diseases like cirrhosis.
Aims & Methods
This study aims to evaluate the accuracy and completeness of ChatGPT's responses in order to assess its effectiveness as an informational tool for patients.
We analyzed 44 frequently asked questions about cirrhosis posed by patients in Facebook support groups, categorizing them into four groups: basic knowledge (19 questions), diagnosis (3 questions), treatment (15 questions), and lifestyle (7 questions). A liver specialist evaluated the answers on a scale from 1 to 4:
- Complete
- Correct but insufficient
- Contains both correct and incorrect information
- Completely incorrect"
Results
The results show that for basic knowledge, 47.3% of answers were complete, 31.5% were correct but insufficient, and 21.05% contained incorrect information. Regarding diagnosis, 66.6% of answers were correct but lacked detail, while 33.3% were complete. For treatment, 33.3% of answers were complete, 40 % were correct but insufficient, and 20% contained errors. Finally, for lifestyle, 42.8% of answers were complete, 28.5% were correct but insufficient, and 14.28% contained errors. Notably, no ChatGPT answer was deemed completely incorrect.
The model demonstrated its ability to provide detailed answers to questions related to basic knowledge and lifestyle. It offered comprehensive explanations of the symptoms, etiology, and prognosis of both compensated and decompensated cirrhosis, as well as risk factors and lifestyle changes that may influence outcomes. While the model provided correct answers in areas like diagnosis and treatment, most responses were rated as correct but insufficient.
Conclusion
This study shows that while the model was able to provide complete and useful answers on basic and lifestyle topics, there is still room for improvement, particularly in more technical areas such as diagnosis and treatment of cirrhosis. It is crucial that medical information is regularly updated to ensure its accuracy and relevance. Additionally, this analysis emphasizes the importance of consulting healthcare professionals for personalized medical advice tailored to each individual situation.
