Introduction

Scoring colonoscopy videos of Ulcerative Colitis (UC) requires high level expertise, but even among trained readers there are disagreements. This can negatively impact subject selection and assessment of treatment response. Consistency among readers can be improved by algorithms that automatically process videos to identify salient features and estimate the level of disease activity. We propose an end-to-end machine learning (ML) system to process colonoscopy videos with the intent of providing an assessment of the Mayo Endoscopic Score (MES) used to determine subject eligibility and endoscopic response to treatment.

Aims & Methods

The dataset included UC videos with a representative range of disease activity acquired at a Ukrainian (n=505) and Israeli hospital (n=227). Video annotation was performed by 12 trained reviewers, supervised by an expert gastroenterologist (D.M., >10 years central reading experience). Out of almost 3 million frames, over 34% were classified by reviewers as non-informative (e.g., out of focus, motion blurring, stool). The remaining informative frames were annotated as containing ulcers (6%) classified as MES 3, erosions (22%) as MES 2, loss of vascularity or erythema (23%) as MES 1, or none of the above (49%) classified as MES 0. Informative frames were split to 80% training, 10% validation and 10% test set. The data set included low quality frames (e.g., not fully focused) on which MES could still be estimated. A pipeline of models, consisting of image processing algorithms, Convolutional Neural Networks (CNN), CNN+Long Short Term Memory (LSTM) model and classical ML classifiers were trained to filter out non-informative frames, score independently each frame for MES and finally scoring the full video for MES based on the aggregated results of the frame-based model.
Model performance was evaluated using Cohen’s Quadratic Weighted Kappa (QWK) and Accuracy metrics with 95% Confidence Interval [95% CI].

Results

The model results are evaluated for full videos, all annotated frames, and a subset of non-blurry ‘clear frames’ from various videos of the test set. Where possible, we compared the Human-Human (H-H) and Model-Human (M-H) performance. Agreement on both MES and dichotomous outcomes (e.g., 0-1 vs. 2-3) was investigated. The agreement of the model with HR was comparable to inter-reader agreement levels (Table 1).

	#Frames/Videos	MES QWK [95% CI]		0 vs. 1-3 Accuracy		0-1 vs. 2-3 Accuracy
		H-H	M-H	H-H	M-H	H-H	M-H
Clear frames	221	0.770	0.822 [0.760, 0.875]	0.917	0.945 [0.909, 0.968]	0.803	0.841 [0.792, 0.887]
All test frames	183,753	-	0.811 [0.809, 0.813]	-	0.883 [0.882, 0.885]	-	0.889 [0.887, 0.890]
Full video	127	0.814	0.763 [0.680, 0.846]	0.732	0.902 [0.842, 0.941]	0.732	0.874 [0.809, 0.919]

Conclusion

A full end-to-end pipeline was developed for processing colonoscopy videos that estimated MES with accuracy comparable to human-human agreement. Such model has the potential to make HR more efficient by highlighting pathology relevant frames. It can also aid inter-reader agreement by providing second-opinion MES. Future work will expand model training and testing with new data sources, test agreement with other expert readers and explore paradigms to combine the model with a HR in the context of clinical trial central reading.

Introduction

Data on using Intestinal Ultrasonography (IUS) for monitoring colonic lesions and treatment response in ulcerative colitis (UC) are still limited and most results are from single-center units usually performed in centers with a special expertise in IUS.

Aims & Methods

Our multicenter study aimed to identify predictive factors and monitor early response after different therapies in UC.
Methods: From July 2022 to Oct. 2023, we conducted a multicenter study at 5 IBD centers. UC pts starting biological therapies or small molecules were included, and underwent colonoscopy (CS), IUS, clinical assessments at baseline and at 12 weeks. All pts received standard of care therapy according to ECCO guidelines. Exclusion criteria were disease limited to the rectum, active infections, and inability or contraindication to perform CS and/or IUS within the time intervals established by the study. Colonic wall thickening (CWT), vascularization (CWF), Milan Ultrasound Criteria (MUC), Mayo endoscopic score (MES), total/partial Mayo score were evaluated.

Results

Eighty-four UC patients were enrolled and eligible for starting biological therapy or small molecules. Forty-nine out of 84 pts (58.4%) were male, 40 pts (47.6%) had extensive colitis and 44 pts (52.4%) had left colitis, with a disease duration of 20 months (1-336); 53 pts (63%) were steroid-dependent/refractory. Forty-one pts (48.9%) started anti-TNFs, 14 pts (16.6%) anti-IL12/23 or anti IL 23, 16 pts (19%) vedolizumab, and 13 pts (15.5%) JAKi. After 12 weeks, 39 out of 84 patients (46.4%) had partial Mayo score<2, 31 patients (36.9%) 2-4, 10 patients (11.9%) 5-7, and 4 patients (4.76%) >7. Forty-three out of 84 patients (51.2%) had MES=0-1, while 41 out of 84 patients (48.8%) had MES=2-3. Fifteen out of 84 patients (17.9%) had ultrasonographic response (defined as CWT<3mm) and 26 out of 84 patients (30.1%) had MUC<6.2 (1). Logistic regression was performed to evaluate the relationship between clinical/endoscopic and ultrasonographic characteristics and early IUS response at 3 months. Univariable analysis showed that active smoking ([RR], 1.9; 95% CI, 1.2–3.1, p<0.005), appendectomy ([RR], 2.5; 95% CI, 1.9–3.2, p<0.0001), partial Mayo score <2 ([RR], 2.18; 95% CI, 1.5–3, p<0.001) were associated with higher risk of response at 3 months. Inversely, severe endoscopic lesions (MES 3) (RR, 0.3; 95% CI, 0.2–0.6; p<0.0001), presence of lymph-nodes, mesenteric hypertrophy and vascularization (p<0.001, RR 0.36, 0.28, 0.45 respectively) were associated with a lower risk of response at 3 months. CWT was associated with a lower risk of response at 3 months (RR 0.66, 95% CI, 0.53-0.81, p<0.0001) and every increased mm was associated with a 34% reduction of IUS response. MUC was associated with a lower risk of response at 3 months (RR 0.78, 95% CI, 0.7-0.87) and every increased of one point was associated with a 22% reduction of IUS response. In the multivariable analysis, CWT and MUC were the only predictive factors associated with IUS response at 3 months (RR 0.70, 0.78 respectively, p<0.04). A CWT<5.5 mm showed a 2.5 risk of treatment response at 3 months.

Conclusion

Our multicenter study demonstrated that IUS allows close monitoring of treatment in UC. MUC, and in particular CWT, are valuable parameters to predict early response in UC.

References

1)Allocca M, et al. Predictive value of Milan ultrasound criteria in ulcerative colitis: a prospective observational cohort study. United European Gastroenterol J. 2022; 10:190-7.

Introduction

Placebo and treatment response rates in randomized ulcerative colitis (UC) trials impact clinical trial design and patients required to demonstrate efficacy. Prior efforts to predict endoscopic healing (EH) in UC trial patients using clinical measures have failed to produce meaningful predictive power. Using AI analysis of baseline endoscopy videos from the ustekinumab UNIFI registration Phase 3 trial in moderate to severe UC, we demonstrate robust accuracy to predict 8 week endoscopic healing in ulcerative colitis patients treated with placebo or drug.

Aims & Methods

300 screening endoscopy videos were obtained from UC patients receiving placebo, and 606 screening endoscopy videos were obtained from UC patients receiving ustekinumab (130mg or 6mg/kg) in the UNIFI induction trial. For all videos, visual features were extracted using EndoDINO, an AI foundation model pre-trained on the largest reported endoscopy video dataset. For each patient, 21 baseline clinical measures were also obtained as predictors. These included fecal calprotectin, albumin, C-reactive protein, lactoferrin, Geboes grading, prior biologics failure, prior steroids failure, UCEIS scores, and Mayo scores.
For each of the placebo and drug arms, we trained two binary classification models: one using the AI visual features and the other using the baseline clinical measures as predictors. Each model was trained to predict 8 week EH, defined as an endoscopy subscore of the Mayo score equal to 0 or 1. We performed 5-fold cross validation and reported the average Area Under the Receiver Operating Curve (AUROC).
This study, carried out under YODA Project # 2024-0668, used data obtained from the Yale University Open Data Access Project, which has an agreement with JANSSEN RESEARCH & DEVELOPMENT, L.L.C.. The interpretation and reporting of research using this data are solely the responsibility of the authors and does not necessarily represent the official views of the Yale University Open Data Access Project or JANSSEN RESEARCH & DEVELOPMENT, L.L.C.. The original proposal can be found: https://yoda.yale.edu/data-request/2024-0668/.

Results

For the placebo arm, our best performing model using AI visual features predicted 8 week endoscopic healing with an AUROC of 0.78, compared to 0.70 for our best performing model using the combination of 21 baseline clinical measures.
For the drug arm, our best performing model using AI visual features predicted 8 week endoscopic healing with an AUROC of 0.75, compared to 0.72 for our best performing model using the combination of 21 baseline clinical measures.
For models using baseline clinical measures, table 1 shows the importance factor for select measures across arms, calculated using XGBoost built in feature importance.

Clinical Measure	Placebo EH Model	Drug EH Model
Mayo endoscopic subscore	0.00	0.13
C-reactive protein	0.22	0.17
Fecal calprotectin	0.12	0.06
Screening erosion	0.32	0.02
Prior biologics failure	0.00	0.01

Conclusion

Our results demonstrate AI analysis of baseline colonoscopy videos with the EndoDINO AI foundation model provides robust accuracy for predicting 8 week EH in ulcerative colitis patients in both placebo and drug arms. AI placebo response prediction could be used in future trial design as covariate analysis to increase statistical power or reduce necessary sample size.
In future work, we aim to improve predictive power by combining our AI visual features with the most important baseline clinical measures through different modeling techniques.

References

Sands BE, Sandborn WJ, Panaccione R, et al. Ustekinumab as induction and maintenance therapy for ulcerative colitis. N Engl J Med. 2019;381(13):1201–1214. doi:10.1056/NEJMoa1900750

Disclosure

AK, PD, MC, and MS are employees of Virgo Surgical Video Solutions, Inc., which developed technology used in this work.

Introduction

Endoscopy represents the gold standard of diagnosis for UC patients. It is an invasive investigation that can assess and stratify disease activity, evolution and treatment. Over the past few years, machine learning (ML) has become a potent instrument in the field of medicine, thanks to its capacity for discrimination and decision-making. A suitable, non-invasive biomarker for assessing endoscopic disease activity (EDA) in ulcerative colitis (UC) has yet to be identified.

Aims & Methods

Our study aimed to develop a cost-effective and non-invasive machine learning (ML) method that utilizes the no-cost Inflammatory Bowel Disease Questionnaire (IBDQ-32) score and low-cost biological predictors to estimate EDA.Two random forest (RF) and 2 multilayer perceptron (MLP) classifiers were proposed. The first RF and MLP models were developed to predict EDA based on the selected variables, excluding IBDQ-32 score. The second RF and MLP classifiers were built to estimate EDA based on all the selected predictors, including IBDQ score.

Results

The results show that the inclusion of IBDQ-32 in the list of predictors that were fed to the models, improved accuracy and AUC for both the RF and the MLP algorithms. Moreover, the RF technique performed noticeably better than the MLP method on unseen data (the independent patient cohort).

Conclusion

This is the first study to propose the use of IBDQ-32 as a predictor in a ML model aimed to estimate UC EDA.The deployment of this ML model can furnish doctors and patients with valuable insights about EDA, a highly beneficial resource for individuals with UC who need long-term treatment.

Introduction

In ulcerative colitis (UC) clinical trials, endoscopic disease activity is commonly assessed via central reading of the overall Mayo Endoscopic Score (MES). Segmental MES can provide a more detailed evaluation of mucosal response, but it is limited by high inter-rater variability in both anatomical landmark localization and disease activity grading. Automatic MES grading can assist reducing reader discrepancies and detect local mucosal changes that may not be estimated by global scoring. In this study, we developed and evaluated an artificial intelligence (AI)-based system for automated segmental MES grading enabling disease extent quantification and granular endoscopic remission estimation.

Aims & Methods

In contrast to previous approaches, the proposed AI system introduces a novel hierarchical framework for automated MES grading at the frame-, segment-, and video-level. The system comprises three modules, trained on 5,897 retrospectively acquired, aggregated, and anonymized endoscopic videos. A foundation model (FM), trained on 22 million frames, was developed to extract clinically relevant frames from full-length colonoscopies. A colon segmentation module was built upon the FM to localize three anatomical segments: descending colon (DC), sigmoid colon (SI), and rectum (RM). An automated MES classification module was then trained to perform frame-level MES grading. Hierarchical aggregation of local scores, combined with temporal smoothing and non-informative filtering, enabled both segmental and global MES estimation. For independent testing, 915 colonoscopy videos were annotated for each segment for MES scoring by expert gastroenterologists using a 2+1 reading paradigm. Colon segment reference annotations were automatically extracted from endoscopic videos based on recording software annotations. System performance was evaluated using the Quadratic Weighted Kappa (QWK) for both segmental and overall MES agreement. Classification accuracy in detecting endoscopic granular remission (MES 0, 1 vs 2, 3) was reported for all colon segments.

Results

The AI system demonstrated strong agreement (QWK=0.752 [95% CI: 0.719, 0.782]) with gastroenterologist assessments for overall MES on the test dataset. Good agreement was confirmed for SI (QWK=0.647 [95% CI: 0.602, 0.687]) and RM (QWK=0.687 [95% CI: 0.646, 0.724]), moderate agreement was found for DC (QWK=0.592 [95% CI: 0.541, 0.638]). Frame-level colon segment classification accuracy was 41% for DC, 69% for SI, and 78% for RM; with lower performance in DC linked to high variability in reference annotations. Colon segmentation enabled a 37.72% reduction in mean colonoscopy video time by filtering out-of-body and insertion-phase sections. The system showed very high accuracy in detecting both overall (ACC=90.38%) and segmental (DC: 78.80%; SI: 82.84%; RM: 85.03%) endoscopic remission. Mean accuracy in detecting the number of colon segments per video with active disease was 59.13%.

Conclusion

We introduced a novel AI system for automated, granular MES scoring in ulcerative colitis patients, hierarchically grading colonoscopy videos. Our framework demonstrated strong agreement with gastroenterologists in both overall and segmental endoscopic activity assessments. By integrating continuous scoring with anatomical landmarking, the AI system can offer a more comprehensive view of total disease burden in UC patients which can lead to improved precision of disease estimation and treatment response evaluation in clinical settings.

Disclosure

All authors are employees of Clario Inc., Philadelphia, USA.

Introduction

To examine the evolution of the ulcerative colitis (UC) treatment pathway in advanced therapy (AT) naïve and later line EU4+UK patients over time, as well as identifying how treatment goals may have changed.

Aims & Methods

A multi-centre online medical chart review study of patients with UC was conducted between Jan – Mar 2018, Jul – Sep 2020 and Jan – Mar 2023, among EU4+UK (UK, FR, DE, IT & ES) gastroenterologists practicing across hospital and private practices. Recruited physicians were screened for duration of practice in their specialty and caseload. Respondents completed patient record forms of their next 4 UC patients seen following receipt of the survey.

Results

The total number of sampled gastroenterologists recruited in this survey and reported UC patients are in Table 1.

Table 1: Sampled physician and reported patient sample size

Timepoint	Q1 2018	Q3 2020	Q1 2023
Surveyed gastroenterologists	207	211	220
Reported UC patients	816	844	880

Between Q1’18 – Q3’20 - Q1’23 the percentage usage of total adalimumab volume in reported UC patients dropped (30% vs 15% vs 13%, respectively), as well as the proportion of these patients reported as first line adalimumab patients (90% vs 88% vs 87%, respectively). In addition, the proportion of reported patients who experienced a switch from one TNF to another also drops between Q1’18 – Q3’20 (14% vs 10%, respectively). Conversely, the proportion of reported UC patients receiving a non-TNF therapy increases over time (12% vs 29% vs 43%, respectively); specifically, the proportion of reported non-TNF first line patients increased (5% vs 12% vs 25%, respectively). Switches from non-TNFs to TNFs increased between Q1’18 – Q3’20 (8% vs 11%, respectively). The proportion of the overall sample occupied by first line patients decreased between Q1’18 – Q3’20 (84% vs 77%, respectively).

Conclusion

In this study cohort, use of adalimumab molecule decreases over time, seemingly in favour of non-TNF therapies, as more products become available in the gastroenterology landscape. This trend is seen across both the overall market and at first line of therapy, suggesting a move away from use of the more ‘traditional’ therapies in general, as well as more frequently choosing a non-TNF as a first line option. The increasing proportion of non-TNF 1st line patients, as well as the increasing proportion of those switching from non-TNFs to TNFs, suggests an increasing proclivity to use non-TNF products earlier in the treatment algorithm. TNF cycling also decreases, suggesting an increased tendency to use non-TNF products. This, along with the decreasing proportion of the sample occupied by first line patients, could indicate an increased propensity by gastroenterologists to switch patients between therapies sooner. Further investigation using comparator cohort is warranted to further investigate impact of increasing numbers of approved UC treatment options.