Introduction

Artificial intelligence (AI) systems in endoscopy are predominantly developed in expert centers using standardized, high-quality imaging. However, performance of these systems may decline due to heterogeneous imaging conditions found in daily clinical practice. A primary source of this variability is the diversity in post-processing enhancement settings used across different endoscopy units. These enhancement settings may improve image quality parameters such as color, contrast and sharpness. While often subtle to the human eye, these alterations can significantly impact the accuracy of AI predictions. A potential solution is to simulate the heterogeneity of these enhancement settings during the AI systems training phase using domain-specific data augmentation methods.

Aims & Methods

We aimed to evaluate the negative impact of post-processing enhancement settings on AI performance and explore domain-specific data augmentation as a method to mitigate performance loss.
We investigated two commonly used AI systems in endoscopy: a computer-aided detection (CADe) system for Barrett’s neoplasia¹ and a computer-aided diagnosis (CADx) system for the optical diagnosis of colorectal polyps². First, both systems were trained using their original datasets, which comprised limited variability of enhancement settings. These CAD systems were then tested across a wide range of test sets. Each test set comprised the same images but were displayed with a different enhancement setting. This represented the image heterogeneity of daily clinical practice. Finally, both CAD systems were retrained using domain-specific data augmentation. This involved the use of a proprietary software tool to precisely simulate the variability of the complete spectrum of available enhancement settings during training. The array of test sets was then reevaluated by these updated CAD systems.

Results

The CADe and CADx system trained using their original training set, displayed substantial performance variability across the different test sets. Depending on the test sets enhancement setting, the CADe system displayed sensitivity scores between 83% and 92%, a 9% difference. Specificity scores ranged between 84% and 91%, a 7% difference. Similarly, the CADx system reached differences of up to 7% (78-85%) for sensitivity and 18% (45-63%) for specificity. After retraining both CAD systems using domain-specific data augmentation, sensitivity and specificity variability was reduced to 2% (89-91%) and 1% (90-91%) for CADe and 2% (78-80%) and 8% (55-63%) for CADx.

Conclusion

The performance of current endoscopic AI systems can vary substantially depending on the post-processing enhancement setting of its input data. Domain-specific data augmentation can effectively reduce this variability to ensure consistent performance across different endoscopy units.

References

1. Fockens N, Jong MR, Jukema JB, Pouw RE, Duits LC, Van Munster SN, et al. A deep learning system for detection of early Barrett's neoplasia: a model development and validation study. The Lancet Digit Health. 2023;5:905-21.
2. Houwen BB, Hazewinkel Y, Giotis I, Vleugels JL, Mostafavi NS, van Putten P, et al. Real-time computer-aided diagnosis system for optical diagnosis of diminutive colorectal polyps including sessile serrated lesions: a prospective, multicenter study with benchmarking against screening endoscopists. Endoscopy. 2022;54(S 01):eP133.

Introduction

In recent years, there has been a surge in the development of artificial intelligence (AI) systems for endoscopic applications, such as Barrett’s neoplasia detection and polyp characterization. While these systems have shown promising results, the vast majority is trained and validated on small, retrospective, single-center or expert-acquired data sets. Current AI systems are particularly prone to a so-called domain-shift: a system only performs well on data it has been trained on. As real-world scenarios in the endoscopy suite can be complex and diverse, the risk of poor generalization of AI should not be underestimated.

Aims & Methods

In this study we aim to provide a quantitative analysis of this domain shift and propose several strategies to improve generalization to clinical practice. We trained three AI systems for commonly used applications in gastrointestinal endoscopy, using publicly available data sets with high quality standards. In order to identify the limitations of these systems, all test sets were artificially degraded in a stepwise manner with clinically prevalent artifacts: blur, inadequate exposure and color variability. These degradations represent the heterogeneity of image quality in daily clinical practice. Severity of degradation was clinically calibrated by a panel of expert endoscopists. Two levels of degradation were identified: 1) mild degradation, still considered as high quality data; 2) moderate degradation, considered as image quality that could still be encountered in daily clinical practice. The trained models were subsequently evaluated for their appropriate performance metrics on the gradually degraded test sets. After implementation of several training strategies to mitigate potential performance loss, the models were re-evaluated on the same test sets.

Results

All three models display a considerable decrease in performance when exposed to various types of clinically relevant image degradation. Mean loss was 7.4% and 16.6% for respective mild and moderate degradation. Implementation of specific training strategies to improve robustness resulted in substantial reduction of performance loss on all degraded test sets.

Conclusion

This study highlights the potential limitations of current AI systems in endoscopic applications which may lead to poor generalization. Specific training strategies can mitigate this risk and improve robustness of these systems. Development of robust and reliable AI systems that deliver consistent and accurate performance is crucial to ensure safe implementation into daily clinical practice.

Introduction

Artificial intelligence (AI) systems in endoscopy are predominantly developed in expert centers using high-quality images and may not perform as expected in community hospitals due to the heterogeneity of image quality. This can be attributed to the tendency of AI systems to perform well on familiar data but deteriorate quickly on data that slightly deviates from the training set. This phenomenon can be referred to as ‘domain shift’.

Aims & Methods

This study aimed to evaluate the performance drop of a computer-aided detection (CADe) system for Barrett’s neoplasia when exposed to the diverse imaging conditions found in community hospitals and assess various training strategies to mitigate this decline of performance.
First, a CADe system was developed using a high-quality training set collected by experts, consisting of 437 images from 173 neoplastic Barrett’s patients and 574 images from 200 non-dysplastic Barrett’s esophagus patients. Classification performance was tested on three test sets comprising different levels of image quality (high, moderate, and low). These quality levels represented the image quality heterogeneity of community hospitals. Each test set included 120 images from a the same cohort of 65 neoplastic and 55 non-dysplastic Barrett’s patients. We then applied three training strategies to improve the CADe systems robustness to image quality variability: 1) additional training data comprising lower quality images; 2) domain-specific pretraining; and 3) adjustments to the architecture of the CADe system.

Results

The CADe system trained solely on high-quality data scored an AUC of 83% on the high-quality test set, which significantly decreased to 80% (p=0.0039) and 71% (p=0.0001) on the moderate and low-quality sets, respectively. Implementation of the robustness-enhancing strategies significantly improved AUC scores on all test sets to 92% for high-quality (p=0.0039), 93% for moderate-quality (p=0.0006), and 85% for low-quality images (p=0.0001). These methods also significantly reduced the performance decline on moderate (+1% vs -3%; p=0.0043) and low-quality test sets (-7% vs -12%; p=0.0163).

Conclusion

CADe systems developed solely on high-quality training images acquired by experts may not be as effective when deployed in community hospitals due to image quality heterogeneity. Adopting robustness enhancing techniques can limit performance loss caused by this domain shift.

References

1. Finlayson SG, Subbaswamy A, Singh K, Bowers J, Kupke A, Zittrain J, et al. The Clinician and Dataset Shift in Artificial Intelligence. New England Journal of Medicine. 2021.

Introduction

The development of artificial intelligence (AI) for endoscopy has significantly increased, yet the aspect of endoscopist-AI interaction remains insufficiently explored. This interaction is crucial as it could substantially influence the AI system's performance in routine clinical practice.

Aims & Methods

This study aimed to evaluate two different graphical user interfaces (GUIs) for a computer aided detection (CADe) system for recognition of early Barrett’s neoplasia. We performed a comparative analysis of two GUI designs—the frequently used bounding box GUI and a novel heatmap GUI—using an extensively evaluated CADe system¹. Thirty-seven endoscopists from six countries participated. Each endoscopist assessed 70 Barrett's esophagus. Every video was analyzed by the CADe system and contained, at some point, a detection which was visualized by the GUI. The study was conducted in two phases, with each video initially shown using one of the GUIs, followed by a two-week washout period after which the same videos were displayed with the alternative GUI. Endoscopists were tasked with identifying potential neoplastic lesions, selecting biopsy sites, and stating their GUI preference.

Results

No significant differences were found for classification performance between the two GUIs (sensitivity 83% vs. 83%, p = 0.29; specificity 86% vs. 86%, p = 0.09). Both interfaces achieved similar scores in localization accuracy, with a median of 97%. Among the participants, 23 expressed a preference for the heatmap GUI, while 14 favored the bounding box.

Conclusion

Despite a general preference for the heatmap GUI among endoscopists, there were no statistically significant differences in performance outcomes between the two interfaces.

References

1. Fockens N, Jong MR, Jukema JB, Pouw RE, Duits LC, Van Munster SN, et al. A deep learning system for detection of early Barrett's neoplasia: a model development and validation study. The Lancet Digit Health. 2023;5:905-21.