Introduction

Advances in AI have improved lesion detection in upper gastrointestinal (GI) endoscopy. However, most systems still behave as black boxes, providing indings without explanation. In a clinical setting, this lack of interpretability limits adoption. In contrast, vision-language models (VLMs) can provide explanatory outputs, but they lack performance and efficiency. We propose a hybrid system that combines a high-performance lesion detector with a compact VLM to produce structured explanations for upper GI endoscopic images, improving both accuracy and transparency. This system aims to improve classification accuracy for neoplastic vs. non-neoplastic lesions and enhance clinical explainability while enabling effective inference with realistically limited datasets, maintaining clinically practical inference speed during procedures. During the endoscopic procedure, CNN-based detectors can perform at around 10ms per input. Down-stream small VLM can generate 92.8 tokens per second, which is suitable for post-procedure evaluation.

Aims & Methods

We developed a multimodal pipeline integrating a CNN-based lesion detector with a lightweight VLM (Gemma-3-4B). Given an input endoscopic image, the DETR-RT detector predicts lesion type (p0/p1) and bounding boxes, which are passed as metadata inputs to a SigLIP image encoder and a Gemma-3 small language model. Finally, our pipeline outputs description of the context including organ identification, lesion classification, and explanation of the detector’s prediction. We employed a distillation process, in which explanations and descriptions of the input endoscopic image are generated by a very large model such as Gemini-2.5-Pro and GPT-4.1. These descriptions are then used to train a small language model based on Gemma-3 with 4 billion parameters. We prepared 1560 images from 246 clinical cases as experimental data. 1247 images (197 cases) are used for training and the remaining 313 images (49 cases) are used for testing.

Results

On a held-out test set of gastric cases, the system model achieved a 67% sensitivity and 85% specificity of distinguishing neoplastic from non-neoplastic lesions —representing 3% and 2% improvement respectively over the detection model (DETR-RT) alone. Across the held-out test set, structured explanations included consistent fields (organ, lesion type, morphology, rationale) with 100% completeness. Component-wise accuracy: organ (85%), light color (100%), lesion type (75%), and JGES classification (83%).

Conclusion

We developed a compact VLM that generates structured, clinically meaningful explanations for endoscopy images. By fine-tuning with synthetic data and integration of a detector, the model not only identifies abnormalities but explains them based on key morphological and contextual features. Despite limited train data, it achieved high accuracy and interpretability, outperforming detection-only baselines.This approach adds structured explanations without sacrificing performance, enabling efficient training and improving both diagnostic accuracy and transparency. By encoding established endoscopic diagnostic criteria into the output, the system presents clinicians with synthesized clinical knowledge—supporting informed interpretation and enhancing trust in AI-assisted decision making. Notably, the explanations for false positives often referenced benign patterns (e.g., folds, shadows), which are anticipated to support clinicians in interpreting uncertain outputs and assessing potential misjudgments.

References

1. Japanese Gastric Cancer Association. Japanese classification of gastric carcinoma: 3rd English edition. Gastric Cancer. 2011;14(2):101–112.
2. Endoscopic Classification Review Group. Update on the Paris classification of superficial neoplastic lesions in the digestive tract. Endoscopy. 2005;37(6):570–578.
3. Yao K. The endoscopic diagnosis of early gastric cancer. Ann Gastroenterol. 2013;26(1):11–22.
4. Sugimoto M, et al. Risk stratification using the Kyoto classification of gastritis. JGH Open. 2017.
5. Zhang L, et al. AI-EARS: an AI-based endoscopic automatic reporting system improves reporting completeness and diagnostic accuracy. JAMA Netw Open. 2023.
6. Park SH, et al. Vision-language models in gastrointestinal endoscopy: current status and future potential. Dig Endosc. 2023.
7. Mu Y, et al. Development and validation of a deep learning system for gastric inflammation classification based on the Kyoto classification. Gastric Cancer. 2021.
8. Banks M, et al. British Society of Gastroenterology guidelines on the diagnosis and management of patients at risk of gastric adenocarcinoma. Gut. 2019;68(9):1545–1575.
9. Dinis-Ribeiro M, et al. Management of precancerous conditions and lesions in the stomach (MAPS II): ESGE/EHMSG/ESP/SPED guideline update 2019. Endoscopy. 2019;51(4):365–388.
10. OpenAI. GPT-4 Technical Report. 2024. https://openai.com/research/gpt-4
11. Google Research. PaliGemma: Open Vision-Language Models. 2024. https://ai.googleblog.com/2024/03/paligemma.html

Clinical Case Summary

Introduction: Patients with inflammatory bowel disease (IBD) face increased cardiovascular risk due to systemic inflammation and corticosteroid exposure. When acute severe colitis coincides with recent acute coronary syndrome (ACS) requiring dual antiplatelet therapy (DAPT), colectomy presents significant clinical challenges.
Case presentation: A 65-year-old male with hypertension, hyperlipidemia, and past smoking history was diagnosed with mild to moderate left-sided ulcerative colitis (UC) in March 2023. Initial response to mesalazine and budesonide was followed by deterioration and steroid dependence. Vedolizumab was initiated in November 2024 but stopped after three doses due to clinical and endoscopic worsening (MES 3). Infliximab, started as rescue therapy in January 2025, led to partial improvement, but disease flared again before the third dose. C. difficile and CMV were repeatedly negative. MRI showed diffuse colonic wall edema and reactive lymphadenopathy. A short course of metronidazole and rifaximin gave transient relief. Despite therapeutic Infliximab levels and no antibodies, inflammation persisted. Three days after the fourth dose, the patient was hospitalized with NSTEMI and received a stent in an occluded diagonal branch; additional significant stenoses required planned re-coronarography. DAPT was initiated. C. difficile infection diagnosed on admission postponed the procedure but responded to vancomycin. After brief improvement, ASUC in April 2025 left the patient bedbound. LMWH was introduced due to elevated thromboembolic risk. Surgical evaluation confirmed the need for urgent total colectomy.
Conclusion: In a patient with ASUC following recent ACS, decision-making must be guided by clinical urgency. When medical therapy fails, urgent colectomy becomes unavoidable - even in the setting of ongoing DAPT. Multidisciplinary coordination is key to balancing perioperative ischemic and bleeding risks.

Introduction

Over the past decade, endoscopic retrograde appendicitis therapy (ERAT) has emerged as a novel, minimally invasive treatment option for patients with acute uncomplicated appendicitis (AUA). However, it is not yet known whether ERAT is a feasible alternative to antibiotic therapy (ABT).

Aims & Methods

This study aims to evaluate the efficacy of ERAT versus ABT in patients with AUA. We performed a systematic review across PubMed, Embase, and Cochrane CENTRAL databases, searching for studies comparing ERAT to ABT in this population. Outcomes of interest included therapeutic success, — defined as the absence of reintervention up to 1 year of follow-up — recurrence rates, mean time to disease recurrence, and length of hospital stay (LOS). Risk ratios (RR) were used to evaluate binary outcomes with their corresponding 95% confidence intervals (CI), while the mean difference (MD) was calculated for continuous outcomes. All statistical analyses were performed using R software (version 4.4.1) with a random-effects model.

Results

This meta-analysis included 3 studies — 2 control-matched cohorts and 1 randomized trial — comparing 457 ERAT patients to 784 patients receiving ABT for AUA. Baseline clinical characteristics were equally distributed between the groups and the time of follow-up ranged from 3 to 12 months. There was no statistically significant difference between ERAT and ABT in treatment success (RR 1.06; 95% CI 1.00 to 1.12; p = 0.07) or recurrence rate (RR 0.36; 95% CI 0.11 to 1.22; p = 0.10). The difference in time to recurrence was also not significant (MD 116 days; 95% CI -49 to 282; p = 0.17). However, ERAT was associated with a significantly shorter LOS (MD -1.85 days; 95% CI -3.15 to -0.55; p = 0.005).

Conclusion

In patients with acute uncomplicated appendicitis, ERAT is non-inferior to antibiotic therapy, while considerably reducing the length of hospital stay. Further studies are warranted to better assess the efficacy of ERAT compared to ABT, and whether it decreases the risk of appendicitis recurrence.

References

1. Kang, J., et al. (2021). The modified endoscopic retrograde appendicitis therapy versus antibiotic therapy alone for acute uncomplicated appendicitis in children. Surgical Endoscopy, 35(11), pp.6291-6299. Available at: https://doi.org/10.1007/s00464-020-08129-8.
2. Li, D., et al. (2023). Endoscopic retrograde appendicitis therapy or antibiotics for uncomplicated appendicitis. British Journal of Surgery, 110(6), pp.635-637. Available at: https://doi.org/10.1093/bjs/znad023.
3. Zhan, K., et al. (2024). Visual Endoscopic Retrograde Appendicitis Therapy Vs Antibiotic Therapy for Treatment of Uncomplicated Acute Appendicitis. The American Journal of Gastroenterology. Available at: https://doi.org/10.14309/ajg.0000000000003118.