Introduction

Currently available artificial intelligence (AI) large language models (LLM) fail to translate effectively to clinical medicine, largely being relegated to question answering roles, documentation, and medical literature summarization. Recognizing this potential, we developed a proof of concept of a specialty-specific and multi-task clinical LLM: GastroGPT. Our study evaluated GastroGPT against leading general-purpose LLMs across key clinical tasks and diverse gastroenterology case scenarios.

Aims & Methods

In this structured analysis, GastroGPT was compared to three state-of-the-art general purpose LLMs (LLM-A: GPT4, LLM-B: Bard, LLM-C: Claude) with highest known utility metrics (training and model parameters, published performance benchmarks, user adoption). All models were assessed on seven key clinical tasks and overall performance. Ten simulated cases covered general gastroenterology and sub-specialties across varying complexity, frequency, and demographics. Standardized prompts ensured structured comparisons. Expert panels, representing varied backgrounds, blindly rated model outputs overall and per task using a 10-point Likert scale, judging utility, style, practicality, and knowledge depth. 480 expert scorings were obtained. Comprehensive statistical analyses were performed.

Results

GastroGPT significantly outperformed other LLMs overall (8.05±1.9 vs 4.95±2.95, 5.63±3.35, 6.92±2.69; p<0.001). GastroGPT also scored higher in models overall comparison in each case (p<0.05). GastroGPT demonstrated superior performance in six of seven tasks: 1) Initial assessments (p<0.001 vs LLM-A and LLM-B, p=0.39 vs LLM-C), 2) Additional history gathering (p<0.001 for all), 3) Diagnostic test recommendations (p<0.001 vs LLM-A, p=0.83 vs LLM-B, p=0.35 vs LLM-C), 4) Management plans and treatment strategies (p<0.001 vs LLM-A and C, p=0.57 vs LLM-B), 6) Multidisciplinary care organization and specialist referrals (p<0.001 vs all) and 7) Patient education and counseling (p<0.001 for LLM-A and B, p=0.07 vs LLM-C). In task 5) follow-up planning LLM-B scored comparably. (7.45±2.04 vs 7.82±1.88, p=0.35) (Table 1). GastroGPT showed more consistent performance across cases (Levene's test p<0.001 for all tasks). Subgroup analysis illustrated that GastroGPT showed higher performances across different complexity levels and frequencies. In multivariable analysis accounting for complexity, frequency, and clinical task, LLM choice emerged as a significant predictor of performance (p<0.001). Reviewer panel scores showed acceptable consistency (Cronbach’s alpha: 0.76 CI:0.71–0.79).

Table 1. Evaluation scores for models’ performances in key clinical tasks

Conclusion

Our study pioneers the development and comparison of a specialty-specific, clinically oriented AI model to state-of-the art general-purpose LLMs. GastroGPT demonstrated superior utility overall and in key clinical tasks of gastroenterology practice. These findings underscore the potential for specialty-focused, clinically- and task-oriented AI models in medicine.

Disclosure

Olga P Nysse: IDEM
Rajiv Jalan: Inventor of OPA, which has been patented by UCL and licensed to Mallinckrodt Pharma. He is also the founder of Yaqrit Discovery, a spin out company from University College London, Hepyx Limited and Cyberliver (the company that holds the intellectual property for AlcoChange). He had research collaborations with Yaqrit Discovery

Introduction

Enteral feeding (EF) is recommended in the severe or predicted severe forms of acute pancreatitis (AP) to reduce mortality.

Aims & Methods

We aimed to investigate the factors leading to initiating EF in the acute phase (first three days of hospitalization) and the associations between EF and the disease outcome. 1429 prospectively collected cases from 31 centers and 12 countries were included in the analysis. EF and Non-EF patient groups were formed concerning tube feeding in the first three days of the hospital stay. We investigated the associations between pancreatic and inflammatory laboratory parameters (amylase, lipase, CRP, WBC, LDH), imaging alterations (necrosis, fluid collections, etc.), and clinical symptoms influencing the initiation of EF (abdominal pain, vomiting, etc.). We used Fisher's exact test for categorical variables and the Mann-Whitney U test for not normally distributed continuous variables. A p-value <0.05 indicates a statistically significant difference.

Results

EF was applied during hospitalization in 30%, 41%, and 55% of mild, moderately severe, and severe cases, respectively. Significantly higher WBC (p=0.042 p=0.019, p=0.002 on D1, D2, and D3), CRP (p<0.001 on D3), and LDH (p=0.004, p=0.036 on D1 and D3) were detected in the EF group before nutritional support compared to the non-EF cases. The frequency of necrosis, pseudocyst, abdominal fluid collection (all with p<0.004), and the intensity of abdominal pain (p<0.001) were also significantly higher in the EF group before nutrition initiation. The overall mortality in this cohort was 2.8%. Notably, the mortality in the non-EF severe group was 10% higher than in the EF severe group (38% vs. 28%, p=0.384).

Conclusion

All parameters influencing the EF initiation here are not part of the current guidelines. In conclusion, decision-making mechanisms for initiating EF rely on personal preferences; therefore, there is still a substantial proportion of patients with severe AP without EF (45%). Since EF is cost-effective and has a very low complication rate if applied correctly, it should be introduced to all AP cases regardless of laboratory parameters and imaging findings.