Introduction
Our previously developed UC-SCALE algorithm[1] predicts UC severity at the colon-section level by assigning a Mayo Clinic Endoscopic Subscore (MCES) to the rectum, sigmoid, or descending colon. Yet, manually annotating anatomical sections—especially after the procedure—is exceedingly challenging for expert readers and is typically not part of clinical practice. To streamline UC-SCALE usage and reduce manual labeling, we propose an AI algorithm that automatically identifies each frame’s anatomical location in colonoscopy videos.
Aims & Methods
We used the same five phase-III etrolizumab clinical trial datasets underlying UC-SCALE: three trials (3028 videos, 1309 patients, 391 sites) for training, one trial (692 videos, 327 patients, 80 sites) for validation, and one trial (607 videos, 317 patients, 83 sites) for testing. We tuned model hyperparameters by comparing multiple training runs and selected the best-performing configuration on the validation set for final testing. For each frame, we extracted “embedding vectors” generated by our internal IBD foundation model (adapted from DINO[2]), which condenses frame image content into relevant features.
For comparison, these two models were trained to classify each frame as rectum, sigmoid, or descending colon:
- Frame-Only Model (no video context): Predicts anatomic location using only the single-frame embedding.
- Full Video-Context Model: Uses a transformer-based architecture, similar to the one proposed by Mobadersany et al [3], that incorporates the entire video’s sequence of embeddings, mimicking how clinicians view videos in context.
Results
The Frame-Only Model yielded modest performance (AUC=0.71, F1=0.59), confirming the difficulty of labeling isolated frames. By contrast, the Context-Aware Model significantly improved metrics (AUC=0.93, F1=0.82), highlighting the value of incorporating sequential information—similar to real-time endoscopic navigation.
| Model | AUC | F1-score |
| Frame-Only | 0.71 | 0.59 |
| Full Video-Context | 0.93 | 0.82 |
Table 1. Performance of the two section classification models
Conclusion
Incorporating video-wide context remarkably enhances frame-by-frame anatomical labeling of UC colonoscopy. This approach can be coupled with UC-SCALE’s section-level severity scoring to automate both localization and disease severity estimation, reducing the need for manual section annotation. Future work will validate performance across diverse datasets, potentially lowering trial costs and broadening clinical adoption of IBD AI-based algorithms.
References
[1] Gutierrez-Becker, Benjamin, et al. "Ulcerative Colitis Severity Classification and Localized Extent (UC-SCALE): An Artificial Intelligence Scoring System for a Spatial Assessment of Disease Severity in Ulcerative Colitis." Journal of Crohn's and Colitis 19.1 (2025): jjae187.
[2] Caron, Mathilde, et al. "Emerging properties in self-supervised vision transformers." Proceedings of the IEEE/CVF international conference on computer vision. 2021.
[3] Mobadersany, Pooya, et al. "Harnessing Temporal Information for Precise Frame-Level Predictions in Endoscopy Videos." International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2024.
Disclosure
Alvaro Gomariz, Bernhard Stimpel, Samaneh Abbasi-Sureshjani, Steven Levitte and Teresa Karrer are employees of F. Hoffmann-La Roche Ltd.