DEVELOPMENT AND VALIDATION OF AN ARTIFICIAL INTELLIGENCE MODEL SCORING THE ULCERATIVE COLITIS ENDOSCOPIC INDEX OF SEVERITY: A POTENTIAL IMPROVEMENT OVER EXPERT EVALUATIONS

UEG Presentation

Log in to access this content.

Free for all myUEG account holders. Your access level is set automatically based on your occupation. Medical professionals get full access to all content. If you are a non-medical user, you can only access UEG Week content from congresses you attended.

Log In Create a free account

Not sure what you can access? Learn more about account types.

Introduction

Endoscopic evaluation of disease severity remains the cornerstone of disease assessment in ulcerative colitis (UC). Endoscopic evaluation using scoring systems such as the Mayo Endoscopic Subscore (MES) is, however, subjective with interrater agreements typically ranging from a kappa of 0.4 to 0.7. Post-hoc analysis from clinical trials indicates that up to 30% of patients were incorrectly included due to misclassification of disease severity. Recent studies including our own using artificial intelligence (AI) to assess the MES have shown enhanced classification accuracy using only white light procedures¹.

Aims & Methods

In this study, we aimed to refine our AI model to assess disease severity using the Ulcerative Colitis Endoscopic Index of Severity (UCEIS).
We extracted 2000 images from recorded sigmoidoscopy/colonoscopy performed on UC patients at the Gastrounit, Copenhagen University Hospital Hvidovre, Denmark. Six IBD experts from The Digestive Disease Center, Copenhagen Univresity Hospital Bispebjerg, scored each image according to the UCEIS and MES. The majority score from the six experts for each subscore (Bleeding, Erosions and Ulcers, Vascular Pattern) was considered the gold standard. In cases of tied scores, the higher score was selected, preferring a model that overestimates rather than underestimates disease severity. The final UCEIS score was the sum of the gold standards for each subscore.
A convolutional neural network was trained on 80% of the images, and then validated on the remaining 20%. The network performance was evaluated against gold standard and individual experts’ scores.

Results

All six experts evaluated all 2000 images; the results are summarized in Table 1.

Table 1: Comparative kappa scores from the leave-one-out analysis of experts and the model versus gold standard (majority score from all six experts) for each subscore and the total of the Ulcerative Colitis Endoscopic Index of Severity (UCEIS) and Mayo Endoscopic Subscore (MES)
(linear) Kappa	Expert Mean	Min	Max	AI Model
Bleeding	0.62	0.44	0.73	0.78
Erosion and Ulcers	0.65	0.46	0.71	0.82
Vascular pattern	0.73	0.41	0.88	0.88
Total UCEIS	0.69	0.44	0.80	0.84
Total MES	0.68	0.50	0.77	0.87

The model surpassed the interrater reliability of expert IBD endoscopists in all subscores, showing particularly strong performance in the erosion and ulcers and vascular patterns subscores. The model’s moderate performance in the "bleeding" subscore reflects the lower kappa scores among experts, underscoring the challenges of visual assessments in this area. Notably, the model's performance on the MES was high, indicating significant overlap in the visual cues used for both UCEIS and MES scoring.

Conclusion

This study demonstrates the potential of using an AI model to enhance the accuracy of UC severity assessments with the UCEIS with results surpassing experts. The overlapping visual cues for MES and UCEIS suggest that the AI model can streamline training and application across different scoring systems, enhancing clinical practice efficiency.
We plan to validate the model on a new test set from another hospital to further evaluate its generalizability and effectiveness across various settings. This is essential for confirming its potential to significantly improve the consistency and precision of UC severity assessments.

References

1. Lo, B. et al. High accuracy in classifying endoscopic severity in ulcerative colitis using convolutional neural network. American Journal of Gastroenterology Publish Ah, (2022).

DEVELOPMENT AND VALIDATION OF AN ARTIFICIAL INTELLIGENCE MODEL SCORING THE ULCERATIVE COLITIS ENDOSCOPIC INDEX OF SEVERITY: A POTENTIAL IMPROVEMENT OVER EXPERT EVALUATIONS

Bobby Lo 1, Bjørn Leth Møller 2, Anja Poulsen 3, Malene Fey Kristiansen 3, Ida Benedikte Gottschalck 3, Mette Wøldike Brandt 3, Shaista Rashid 3, Eva Efsen Dahl 3, Ida Vind 1, Flemming Bendtsen 1, Christian Igel 2, Bulat Ibragimov 2, Johan Burisch 1

1 Copenhagen University Hospital – Amager and Hvidovre, Hvidovre, Hvidovre, Denmark|||Copenhagen University Hospital – Amager and Hvidovre, Hvidovre, Hvidovre, Denmark

2 University of Copenhagen, Copenhagen, Denmark

3 Copenhagen University Hospital – Bispebjerg and Frederiksberg, Bispebjerg, København N, Denmark

Event

UEG Week Vienna 2024

Topics

Endoscopy IBD

Submission format

Abstract

Session

AI in diagnostic and therapeutic endoscopy

Citation

United European Gastroenterology Journal 2024; 12 (Supplement 8)

Published

2024

Modality	Sensitivity (%)	Specificity (%)	Accuracy (%)	PPV (%)	NPV (%)
CH-EUS (hypoenhanced)	63.2	86.7	73.5	85.7	65.0
SRE (strain ratio >10)	88.2	75.0	82.8	83.3	81.8
DFI (hypovascularization)	83.3	84.6	83.8	90.9	73.3
CH-EUS + DFI (n = 27)	62.5	90.9	74.1	90.9	62.5
SRE + DFI (n = 24)	66.7	100.0	79.2	100.0	64.3
CH-EUS + SRE + DFI (n = 20)	63.6	100.0	80.0	100.0	69.2

Predictor	Pre-test probability	Probability ratio + (IC 95%)	Post-test probability +	Probability ratio – (IC95%)	Post-test probability -
MORTALITY
TRANSITIONAL SIRS	4%	0.46 (0.07-3.16)	2% (0-12%)	1.07 (0.96-1.20)	4% (4-5%)
PERSISTENT SIRS		16 (4.90-27)	41% (29-53%)	0.33 (0.17-0.64)	1% (1-3%)
BISAP ≥ 2		2.92 (2.40-3.55)	12% (10-14%)	0.14 (0.04-0.52)	1% (0-2%)
MULTIORGAN FAILURE
TRANSITIONAL SIRS	5%	1.18 (0.4-3.49)	6% (2-15%)	0.98 (0.83-1.15)	5% (4-6%)
PERSISTENT SIRS		13 (7.36-23)	40% (28-54%)	0.45 (0.28-0.72)	2% (1-4%)
BISAP ≥ 2		2.74 (2.18-3.45)	13% (11-16%)	0.24 (0.10 – 0.59)	1% (1-3%)

UEG's educational platform for gastroenterology education bringing recordings, courses, guidelines, and more together in one place.

Most popular content

Rate for better recommendations!