Introduction
Regulatory guidance recommends the endoscopy subscore as the index to assess the endoscopic component of the primary endpoint in ulcerative colitis (UC) trials1. Inter-reader variability in assessments may impact the reliability of trial results2,3. Currently, there is no metric in place to assess the certainty by which a reader is assigning an endoscopy subscore. Machine learning (ML) provides an opportunity to assess the endoscopy subscore and provide a measurement of its certainty in a standardized manner. Artificial Intelligence Assessment of Endoscopic Severity (AI-ES) accurately assesses the endoscopy subscore4. The objective of this study is to evaluate the calibration of AI-ES - how well its predicted probabilities reflect true likelihoods - to assess the reliability of its measurement of certainty in endoscopy subscore assessments in UC trials.
Aims & Methods
AI-ES is a deep learning algorithm that assesses the endoscopy subscore in UC endoscopic videos. AI-ES measures probability for the four ordinal endoscopy subscore classes. The endoscopy subscore with the highest probability is assigned as the final score by AI-ES. We assessed calibration on a holdout test set of 639 videos (~25%) from the Phase 3 induction trial for mirikizumab in UC (NCT03518086). Videos had a 2+1 centrally read endoscopy subscore, randomly selected from week 0 and 12 with a distribution of endoscopic severity similar to the overall study population. Calibration plots were generated across endoscopy subscore classes with probabilities grouped into septiles (~100 videos per group) for primary analysis and deciles for confirmation. Brier scores, ranging from 0 (perfect calibration) to 1 (worst calibration), were calculated, with values <0.25 considered informative5.
Results
AI-ES demonstrated strong calibration, with Brier scores below <0.25 for each endoscopy subscore (0: 0.037, 1: 0.082, 2: 0.162, 3: 0.112). The Brier score for evaluation of endoscopic improvement (0,1 vs 2,3) also showed excellent calibration (0.066). Findings were consistent when assessing probabilities by deciles.
| Assessment | Brier Score |
| Ordinal endoscopy subscore |
| Endoscopy subscore 0 | 0.037 |
| Endoscopy subscore 1 | 0.082 |
| Endoscopy subscore 2 | 0.162 |
| Endoscopy subscore 3 | 0.112 |
| Endoscopic improvement |
| Endoscopy subscore 0, 1 vs 2, 3 | 0.066 |
Conclusion
Whereas data on the certainty of human readers in endoscopy subscore assessments are elusive, AI-ES is calibrated across all endoscopy subscore classes, providing reliable data on score probabilities. This novel measurement of certainty by AI-ES added to the score assessment may enable novel AI-based multi-reader or consensus workflows in trials, potentially improving the reliability of UC endpoint assessments.
References
- Food and Drug Administration. Ulcerative Colitis: Developing Drugs for Treatment. Draft Guidance for Industry. April 2022.
- Hashash JG, Yu Ci Ng F, Farraye FA, Wang Y, Colucci DR, Baxi S, Muneer S, Reddan M, Shingru P, Melmed GY. Inter- and Intraobserver Variability on Endoscopic Scoring Systems in Crohn's Disease and Ulcerative Colitis: A Systematic Review and Meta-Analysis. Inflamm Bowel Dis. 2024 Nov 4;30(11):2217-2226.
- Wils P, Jairath V, Sands BE, Magro F, Reinisch W, Rubin D, Danese S, Baumann C, Peyrin-Biroulet L. Comparison of treatment effect between phase 2 and phase 3 trials in patients with inflammatory bowel disease. United European Gastroenterol J. 2023 Oct;11(8):797-806.
- Battioui C, Brodskiy P, Gottlieb K, Haft-Javaherian M, Eastman W, Lehrer J, Yu E, Onken D, Thomason D, Colucci D, Reinisch W, Baxi S. DOP078 Machine learning assessment of endoscopic severity in Ulcerative Colitis trials: Model evaluation against the 2 + 1 reference standard. Journal of Crohn's and Colitis. 2025 Jan;19(1):229-230.
- Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010 Jan;21(1):128-38.
Disclosure
CB, KG, WE, and DO are employees of Eli Lilly and may hold stock or stock options in Eli Lilly.
PB, MH, JL, EY, DT, DC, and SB are employees of Iterative Health and may hold stock or stock options in Iterative Health.
WR has served as a speaker for AbbVie, Celltrion, Ferring, Janssen, Galapagos Medice, MSD, Roche, Pfizer, Sobi, Takeda, as a consultant for AbbVie, Amgen, AOP Orphan, Boehringer Ingelheim, Bristol Myers Squibb, Calyx, Celltrion, Eli Lilly, Galapagos, Gilead, Index Pharma, Janssen, Medahead, Microbiotica, Pfizer, Teva, Takeda; as an advisory board member for AbbVie, Amgen, Boehringer Ingelheim, Bristol Myers Squibb, Celltrion, Galapagos, Janssen, Pfizer, Teva and has received research funding from AbbVie, Janssen, Sandoz, Sanofi, Takeda.
