Introduction
Automatic optical characterization (CADx) systems aim to support gastroenterologists in identifying the type of polyps found during a colonoscopy procedure (e.g. “adenomatous” or “non adenomatous”). It has the potential to decrease the risks, workloads, and costs of polypectomy and pathology tests. The majority of reported CADx systems provide a classification score for each polyp frame, followed by an aggregation of the scores to determine a final prediction of the polyp diagnosis. A common aggregation mechanism is a simple voting procedure. Other aggregation methods include the average or maximum score of the entire sequence. In all of the above, longer sequences should provide more information necessary to get an accurate prediction. However, the aggregation approach does not leverage complementary information (e.g. polyp geometry) that can be learned by jointly processing multiple views. In addition, views diversity and quality are not considered as part of the inference process. At the aggregation stage, only the per-frame score is taken into account, while the appearance of the polyp in each view is not considered. Joint processing of multiple frames, on the other hand, allows relative weighing of each frame with respect to the other available frames in the sequence, while considering their quality and diversity.
Aims & Methods
We propose a multiview learning model, in which multiple frames of a polyp are processed jointly as part of the training process, resulting in a single score for the entire sequence. We apply a convolutional neural network (CNN) to embed each frame into a vector representation. We then use an attention model (Transformer) to jointly process these representations and predict a single score for the entire sequence, rather than for each frame separately. The system is trained in an end-to-end manner, allowing the model to learn to predict a score based on multiple views with varying quality and diversity.
Here we test the performance of the proposed approach in predicting the endoscopic appearance-based polyp classification done by GIs. Predicting the histology results is left for future work. The polyps are annotated by expert GIs into adenomatous or non adenomatous. The test set consists of 565 polyps taken from 317 videos that were unanimously classified by 2 expert GIs, independently. We compare our suggested model to a baseline model which was trained to predict a score for each frame separately, followed by aggregation of the scores for each polyp.
Results
We compare the AUROC of the joint learning model ("Joint Learning") and 3 baseline aggregation schemes - averaging the per-frame scores ("Mean Score"), maximum score ("Maximum Score"), and majority voting ("Voting"). We use polyp frames sequences of varying lengths and evaluate the performance repeatedly on 10 random selections of frames from each sequence. We report the mean and STD of the AUROC for each sequence length. As seen in the table, the joint learning model outperforms all baseline aggregation schemes.
Aggregation Method \ Sequence Length (# Frames)
| 1 | 10 | 20 | 30 | 40 |
Joint Learning
| 0.913 ± 0.007
| 0.959 ± 0.003
| 0.962 ± 0.003
| 0.962 ± 0.002
| 0.961 ± 0.001
|
| Mean Score | 0.907 ± 0.013
| 0.952 ± 0.003
| 0.955 ± 0.003
| 0.955 ± 0.002
| 0.956 ± 0.002
|
| Maximum Score | 0.907 ± 0.015
| 0.954 ± 0.004
| 0.957 ± 0.001
| 0.956 ± 0.003
| 0.956 ± 0.002
|
| Voting | 0.831 ± 0.022
| 0.937 ± 0.007
| 0.945 ± 0.044
| 0.946 ± 0.002
| 0.948 ± 0.002
|
Conclusion
Joint learning of polyp appearance in multiple frames results in better performance for polyp classification in colonoscopy. The joint learning process allows the model to consider diversity, quality and complementary information from multiple views of a single polyp, rather than process each frames separately.
References
Hassan, Cesare, et al. "Artificial intelligence allows leaving-in-situ colorectal polyps." Clinical Gastroenterology and Hepatology 20.11 (2022): 2505-2513.
Mori, Yuichi, et al. "Benefits and challenges in implementation of artificial intelligence in colonoscopy: World Endoscopy Organization position statement." Digestive Endoscopy (2023).
Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems 30 (2017).
Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020).
Disclosure
The authors are affiliated with Verily (@verily.com) and Google (@google.com).
