Introduction

Advances in AI have improved lesion detection in upper gastrointestinal (GI) endoscopy. However, most systems still behave as black boxes, providing indings without explanation. In a clinical setting, this lack of interpretability limits adoption. In contrast, vision-language models (VLMs) can provide explanatory outputs, but they lack performance and efficiency. We propose a hybrid system that combines a high-performance lesion detector with a compact VLM to produce structured explanations for upper GI endoscopic images, improving both accuracy and transparency. This system aims to improve classification accuracy for neoplastic vs. non-neoplastic lesions and enhance clinical explainability while enabling effective inference with realistically limited datasets, maintaining clinically practical inference speed during procedures. During the endoscopic procedure, CNN-based detectors can perform at around 10ms per input. Down-stream small VLM can generate 92.8 tokens per second, which is suitable for post-procedure evaluation.

Aims & Methods

We developed a multimodal pipeline integrating a CNN-based lesion detector with a lightweight VLM (Gemma-3-4B). Given an input endoscopic image, the DETR-RT detector predicts lesion type (p0/p1) and bounding boxes, which are passed as metadata inputs to a SigLIP image encoder and a Gemma-3 small language model. Finally, our pipeline outputs description of the context including organ identification, lesion classification, and explanation of the detector’s prediction. We employed a distillation process, in which explanations and descriptions of the input endoscopic image are generated by a very large model such as Gemini-2.5-Pro and GPT-4.1. These descriptions are then used to train a small language model based on Gemma-3 with 4 billion parameters. We prepared 1560 images from 246 clinical cases as experimental data. 1247 images (197 cases) are used for training and the remaining 313 images (49 cases) are used for testing.

Results

On a held-out test set of gastric cases, the system model achieved a 67% sensitivity and 85% specificity of distinguishing neoplastic from non-neoplastic lesions —representing 3% and 2% improvement respectively over the detection model (DETR-RT) alone. Across the held-out test set, structured explanations included consistent fields (organ, lesion type, morphology, rationale) with 100% completeness. Component-wise accuracy: organ (85%), light color (100%), lesion type (75%), and JGES classification (83%).

Conclusion

We developed a compact VLM that generates structured, clinically meaningful explanations for endoscopy images. By fine-tuning with synthetic data and integration of a detector, the model not only identifies abnormalities but explains them based on key morphological and contextual features. Despite limited train data, it achieved high accuracy and interpretability, outperforming detection-only baselines.This approach adds structured explanations without sacrificing performance, enabling efficient training and improving both diagnostic accuracy and transparency. By encoding established endoscopic diagnostic criteria into the output, the system presents clinicians with synthesized clinical knowledge—supporting informed interpretation and enhancing trust in AI-assisted decision making. Notably, the explanations for false positives often referenced benign patterns (e.g., folds, shadows), which are anticipated to support clinicians in interpreting uncertain outputs and assessing potential misjudgments.

References

1. Japanese Gastric Cancer Association. Japanese classification of gastric carcinoma: 3rd English edition. Gastric Cancer. 2011;14(2):101–112.
2. Endoscopic Classification Review Group. Update on the Paris classification of superficial neoplastic lesions in the digestive tract. Endoscopy. 2005;37(6):570–578.
3. Yao K. The endoscopic diagnosis of early gastric cancer. Ann Gastroenterol. 2013;26(1):11–22.
4. Sugimoto M, et al. Risk stratification using the Kyoto classification of gastritis. JGH Open. 2017.
5. Zhang L, et al. AI-EARS: an AI-based endoscopic automatic reporting system improves reporting completeness and diagnostic accuracy. JAMA Netw Open. 2023.
6. Park SH, et al. Vision-language models in gastrointestinal endoscopy: current status and future potential. Dig Endosc. 2023.
7. Mu Y, et al. Development and validation of a deep learning system for gastric inflammation classification based on the Kyoto classification. Gastric Cancer. 2021.
8. Banks M, et al. British Society of Gastroenterology guidelines on the diagnosis and management of patients at risk of gastric adenocarcinoma. Gut. 2019;68(9):1545–1575.
9. Dinis-Ribeiro M, et al. Management of precancerous conditions and lesions in the stomach (MAPS II): ESGE/EHMSG/ESP/SPED guideline update 2019. Endoscopy. 2019;51(4):365–388.
10. OpenAI. GPT-4 Technical Report. 2024. https://openai.com/research/gpt-4
11. Google Research. PaliGemma: Open Vision-Language Models. 2024. https://ai.googleblog.com/2024/03/paligemma.html

Introduction

Artificial intelligence (AI) characterization (CADx) systems for colonic polyps exhibit high diagnostic accuracy. However, they may only support, not replace, clinical decision-making and little is known of this human-AI interaction – even less so with trainee endoscopists. This issue is difficult to explore, but data from a study investigating CADx-supported trainee endoscopists compared to AI-blinded experts could give insight.

Aims & Methods

This analysis draws on data from a prospective observational study comparing the diagnostic performance of CADx-supported trainee endoscopists (using GI Genius®, Medtronic) to expert endoscopists blinded to AI support. Two key scenarios were explored: (a) instances in which the CADx system produced a ‘no prediction’ output, and (b) cases where trainees actively overruled AI-generated diagnoses. These were assessed to examine the impact of AI absence and human-AI discordance on diagnostic accuracy.

Results

The baseline cohort included 11 trainees performing 225 colonoscopies, mean age 63.8 years (SD 12.7), 48.9% male patients. Baseline ADR was 57.8%. Of 630 resected lesions (median size 3 mm, IQR 2;4), 40.0% were in the rectosigmoid, and 53.6% were non-adenomatous. CADx failed to provide a prediction in 19.2% (n=121) of cases. These ‘no prediction’ polyps were smaller (median 3 mm, p=0.048) and more often located in the right colon (72% vs. 58%, p=0.005. In the absence of AI support, trainee diagnostic accuracy declined significantly by 12.8% (63.6% vs. 76.4%, p=0.007). Interestingly, expert endoscopists—despite being blinded to AI—also performed worse on these polyps (64.7% vs. 73.9%, p=0.045), suggesting an inherent difficulty in classifying these lesions.
Trainees overruled AI suggestion leading to discordant diagnoses in 9.0% (57/630) of polyps, comprised of 20 (35.1%) non-adenomatous, and 37 (64.9%) adenomatous lesions. Trainees incorrectly overruled AI in 59.6% (34/57), which mostly consisted of downgrading lesions to NICE 1. Discordant polyps did not differ in size (3 mm vs. 3 mm, p=0.413), but were more often located proximally (77.2% vs. 58.1%, p=0.005). The difference in false negative rate (FNR) between trainees and AI was 62.16% (81.07% vs. 18.92%, p<0.0001), demonstrating a risk of missed adenomas when trainees override AI. Blinded expert endoscopists achieved identical overall diagnostic accuracy in those polyps as the AI system (59.6%). However, their error profiles differed – In this subset, blinded experts and AI performed equally in overall accuracy (59.6%), although with differing error profiles: AI showed a numerically lower FNR (18.9% vs. 35.1%, Δ16.2%, p=0.17), offset by a higher false positive rate (80.0% vs. 50.0%).

Conclusion

CADx systems generate ‘no prediction’ outputs in a substantial proportion of polyps, significantly impairing trainee diagnostic performance. These cases may represent inherently more challenging lesions. Despite CADx’s high accuracy, trainees overrule AI in almost every tenth polyp, most often incorrectly downstaging polyps. Structured training on appropriate use of CADx, emphasizing both its capabilities and limitations, may enhance clinical effectiveness. Further research on human-AI interaction and trust is warranted, as these factors will be critical for successful integration of CADx into routine endoscopic practice.

Disclosure

The original study (AC-CADx) received technical support by Medtronic. Medtronic was not involved in the design of the study or analysis of the data, neither in the original study nor this exploratory analysis.

Clinical Case Summary

A 24-year-old male presented with jaundice and mild vertigo after a 10-day prodrome (asthenia, nausea, headache, fever). Ibuprofen (1200 mg/d) was taken for these symptoms only during the first 5 days. Labs revealed acute hepatocellular injury and elevated cholestasis (AST 215, ALT 862, FA 134, GGT 126 IU/L), marked mixed hyperbilirubinemia (Total 13.5 mg/dL, Direct 7.58 mg/dL) and coagulopathy (INR 1.59). Ultrasound showed hepatosplenomegaly. Viral hepatitis (HAV/HBV/HCV/CMV) and autoimmune/metabolic causes were excluded. Epstein-Barr Virus (EBV) serology showed low-titer positive VCA IgM but negative VCA IgG, EBNA IgG, and serum EBV DNA PCR, indicating recent non-active infection. Given ibuprofen timing and high RUCAM score, drug-induced liver injury (DILI) with a hepatocellular pattern was suspected. Liver biopsy confirmed acute hepatitis with lymphomonocytic infiltrate, necrosis and canalicular cholestasis. Despite improving liver function tests post-ibuprofen cessation, he developed worsening vertigo, severe ataxia, nausea, vomiting, hypophonia, necessitating nasogastric feeding/bed confinement. Cerebrospinal fluid (CSF) showed mild mononuclear pleocytosis (16 cells/mm³); a multiplex PCR pathogen panel (including EBV) was negative. Brain MRI/EEG were normal. Post-infectious EBV acute cerebellar ataxia was diagnosed concurrently with DILI, based on timing post-recent EBV, exclusion of active neurological infection (negative CSF PCR, normal imaging), mild pleocytosis, and supported by later IgG seroconversion. IV methylprednisolone treatment, chosen for clinical severity, led to significant neurological improvement within days and eventual recovery after physiotherapy. This case highlights the significant challenges of diagnosis and management of concomitant DILI and recent EBV infection with neurological sequelae, requiring careful differentiation of liver damage patterns (hepatocellular DILI vs cholestatic EBV) and optimal management of neurological symptoms.

References

Illustrative Material References:
- Liver biopsy histology slides (highlighting lymphomonocytic infiltrate, necrosis, and canalicular cholestasis characteristic of DILI).
- Table/Graph showing serial Liver Function Tests (AST, ALT, Bilirubin) over time, depicting injury peak and recovery trend.
- Table summarizing sequential Epstein-Barr Virus (EBV) serology results (VCA IgM, VCA IgG, EBNA IgG) demonstrating seroconversion.
- Table summarizing Cerebrospinal fluid (CSF) analysis results (cell count, protein, glucose, negative multiplex PCR).
- Representative Brain MRI images (axial/sagittal views) confirming absence of structural lesions or encephalitis signs.

Introduction

Lynch syndrome (LS) is associated with an increased risk of upper gastrointestinal (GI) cancers. While upper GI surveillance is recommended in several guidelines, there remains international variability and ongoing debate regarding its implementation, timing, and extent. This study evaluated the incidence of gastric (GC), duodenal (DC), and jejunal cancers (JC) in LS patients using data from the German Consortium for Familial Intestinal Cancer (GCFIC). In addition, the diagnostic yield of esophagogastroduodenoscopy (EGD) and push enteroscopy (PE), performed at a tertiary care center, was retrospectively analyzed.

Aims & Methods

A total of 960 endoscopic procedures (494 EGD, 466 PE) were retrospectively analyzed in 349 LS patients documented at a tertiary care center. Push enteroscopy (PE) was performed using a colonoscope to allow deeper insertion into the small intestine, enabling inspection of the proximal jejunum. Clinically relevant findings included intestinal metaplasia, Barrett’s esophagus, adenomas, and carcinomas. These findings were systematically compared between EGD and PE to evaluate diagnostic yield. The GCFIC database was evaluated to determine cancer incidence by genotype.

Results

In the GCFIC cohort (n=2859), the majority of GC (2.2%), DC (2.1%), and JC (1.9%) occurred in high-risk pathogenic variant carriers (PGV) (MLH1, MSH2, EPCAM). However, a small proportion of cancers was also observed in low-risk PGV carriers: GC in two MSH6 carriers, DC in five patients with MSH6 or PMS2, and JC in seven low-risk carriers (MSH6: 3, PMS2: 4). In the NZET cohort, 22% of procedures and 50 % of patients revealed relevant findings. Gastric adenomas were found in 3.7%, gastric cancer in 1.1%, and intestinal metaplasia in 28% of patients. Duodenal adenomas were found in 3.7% and cancer in 0.9%. Jejunal neoplasia was detected exclusively by PE (adenomas 1.7%, cancer 0.6%).

Conclusion

Upper gastrointestinal surveillance is essential in patients with Lynch syndrome. Push enteroscopy (PE) provides diagnostic value beyond conventional esophagogastroduodenoscopy (EGD), particularly in detecting jejunal neoplasia, and should be considered in high-risk surveillance protocols.

Disclosure

Robert Hüneburg was funded by Deutsche Krebshilfe (DECADE), and Wilhelm-Sander-Stiftung.
Jacob Nattermann was funded by Wilhelm-Sander-Stiftung.