Introduction

Computer-aided polyp detection (CADe) systems based on artificial intelligence are increasingly used in routine endoscopic colorectal cancer screening. These systems are regularly updated and occasionally provide adjustable detection thresholds that impact their performance. However, little is known about the effect of these adjustments on the system’s performance.

Aims & Methods

The objective of this study is to evaluate the performance of various CADe systems on a common benchmark dataset, and to compare the performance of different versions and configurations of the same system. The study employed a total of 101 colonoscopy videos, and each video containing visible polyps was manually annotated with a bounding box, resulting in a total of 129 705 polyp images. The performance of three different CADe systems was assessed: two versions of GI Genius, two detection types of EndoAID, and the freely available CADe system EndoMind. The evaluation included a comprehensive analysis of sensitivity, false positive rate, and time to first polyp detection. In addition, the quality of the bounding boxes in delimiting the polyp was also evaluated by calculating the intersection over the union.

Results

EndoAID (type A), the earlier version of GI Genius, and EndoMind detected all 93 polyps in at least one image. Both the later version of GI Genius and EndoAID (type B) missed one polyp. The mean per-frame sensitivity for each system was 50.63 % for the earlier version of GI Genius, 67.85 % for the later version of GI Genius, 65.60 % for EndoAID (type A), 52.95 % for EndoAID (type B), and 60. 22 % for EndoMind. Regarding the mean first detection time, the earlier version of GI Genius required 1 510 ms, the later version of GI Genius required 607 ms, EndoAID (type A) required 659 ms, EndoAID (type B) required 1 316 ms, and EndoMind required 1 083 ms. The CADe system that presented the fewest number of false positive frames was EndoAID (Type B) with a ratio of only 0.63 %. Lastly, EndoMind was the best performing system in terms of bounding box quality with an intersection over union value of 68.32 %.

Conclusion

This study provides a direct comparison of the performance of different CADe systems, as well as different versions and configurations of the same system. The results show that the performance of CADe systems can vary significantly depending on the system used, the version of the system, and the configuration of the system. By providing an objective evaluation of the strengths and weaknesses of CADe systems, this study can help clinicians and researchers select the most appropriate system for their specific needs.