Introduction

The metaverse consists of interconnected 3D virtual worlds that offer immersive experiences to users by stimulating a sense of presence, primarily through wearable headsets. Extended reality (XR), the core technology of the metaverse, combines virtual reality (VR), augmented reality (AR), and mixed reality (MR) to provide a shared online space where users can interact in both physical and virtual worlds.

Aims & Methods

We developed a holography-guided surgical navigation system based on the metaverse platform to improve spatial awareness in gastrointestinal (GI) and hepatobiliary-pancreatic (HPB) laparoscopic and robotic surgeries and investigated its usefulness.
We performed 40 endoscopic/laparoscopic GI and HPB surgeries using a metaverse-based holography-guided surgical navigation system. Over 20 Asian surgeons used a head-mounted display (HMD) during surgery. Holographic 3D guides of organs, lesions, and procedures were created based on individual patient CT/MRI images, providing a realistic 3D view in real space above the surgical field. Simultaneously, lifelike 3D avatars of surgeons were created, allowing for remote surgical planning and discussions in the metaverse.

Results

In all 40 surgeries, tumor lesions, surrounding organs, and blood vessels were accurately spatially reproduced as holographic images in a sterile space above the patient's abdomen, along with resection lines to guide surgical techniques. The system setup took less than 5 minutes on average, and no postoperative adverse events occurred in any patient. Positioning errors in superimposing the virtual models were within 2mm. Holographic avatars increased the efficiency and effectiveness of reviewing, brainstorming, and sharing virtual content. Holographic guidelines allowed surgeons to share their positions and hand movements, proving useful for hands-on training.

Conclusion

Our holography-guided surgical navigation system with the metaverse platform facilitated holographic telepresence through augmented reality, transforming the operating room into a 3D workspace. By allowing users to connect remotely and interact with lifelike avatars, our system reduced surgeon stress and unlocks new levels of computing beyond spatial limitations, regardless of any devices were used. We are currently evaluating its usefulness in gastroenterology by applying it to navigation guides for hepatic US-guided radiofrequency ablation (RFA) and endoscopic mucosal disection (ESD) of the gastrointestinal tract.

References

• Maki Sugimoto. Cloud XR (Extended Reality: Virtual Reality・Augmented Reality・Mixed Reality) and 5G networks for holographic medical image-guided surgery and telemedicine. Multidisciplinary Computational Anatomy - Principles and Clinical Application of MCA-based Medicine. pp 381-338, 2021. Editor: Hashizume M. Springer, Singapore. 2021. DOI https://doi.org/10.1007/978-981-16-4325-5_52
• Maki Sugimoto. Extended Reality (XR:VR/AR/MR), 3D Printing, Holography, A.I., Radiomics, and Online VR Tele-Medicine for Precision Surgery. Surgery and Operating Room Innovation. p65-70. Editor: Takenoshita S, Yasuhara H. Springer, Singapore. 2020. DOI https://doi.org/10.1007/978-981-15-8979-9_7
• Maki Sugimoto. Augmented Tangibility Surgical Navigation Using Spatial Interactive 3-D Hologram zSpace with OsiriX and Bio-Texture 3-D Organ Modeling. IEEE 2015 International Conference on Computer Application Technologies (CCATS 2015). 189-194, 2015. DOI:10.1109/CCATS.2015.53

Disclosure

Holoeyes Inc.; Board member

Introduction

In recent years, cold snare polypectomy (CSP) for small polyps has become popular, but it is considered inappropriate as a treatment for cancer because of the shallow depth of resection at the muscularis mucosa level and the risk of inadequate pathological evaluation. ¹² Therefore, it is important to diagnose advanced histology (AH) (high grade adenoma or adenocarcinoma) prior to treatment. Based on our retrospective review of colorectal neoplastic lesions smaller than 10 mm, the following six endoscopic findings were extracted as significant predictors of AH findings in the multivariate analysis : (1) surface redness, (2) white spots, using white light image, (3) heterogeneity in mucosal pattern, using non-magnified Narrow band imaging (NBI), (4) irregular vessel or surface pattern(equivalent to JNET Type 2B,3), using magnified NBI, (5) loss of lobulation, and (6) non-polypoid growth, using non-magnified chromoendoscopy using indigo carmine. ³

Aims & Methods

The aim of this study was to evaluate the usefulness of image-enhanced endoscopic findings in real-time prediction of AH for small colorectal polyps. Patients scheduled for colonoscopy were recruited for consent prior to the examination at 9 Japanese institutions, and the presence of the aforementioned AH-predictive endoscopic findings was noted for up to 5 lesions per patient when neoplastic polyps of 5-9 mm were detected. If one of the predictive findings was positive, endoscopic resection using the electric current method or surgical resection was applied. The primary endpoint was the negative predictive value (NPV) of the 6 AH-predicting findings using pathology as the gold standard. Secondary endpoints were NPV of AH prediction by modality combination, odds ratios of each predictors and percentage of non-CSP for T1 cancer. Taking account for practical value, the target threshold of NPV was set for 90%.

Results

A total of 1671 patients were enrolled in the study, of which 517 patients with 849 lesions (80 AH lesions) corresponding to 5-9 mm polyps were included in the analysis. The primary endpoint was 0.945 (95% confidence interval (CI): 0.923-0.961). Secondary endpoints were white light + NBI non-magnified/magnified 0.941 (95%CI: 0.920-0.958), white light + NBI non-magnified 0.934 (95%CI: 0.913-0.952), white light + chromoendoscopy 0.926 (95%CI: 0.904-0.944), and white light 0.922 (95%CI: 0.900-0.940). Odds ratio (95%CI) of each 6 findings were as followings; irregular vessel or surface pattern 5.00 (2.59-9.37), loss of lobulation 4.10(1.61-6.56), white spots 3.32(1.60-6.56), non-polypoid growth 3.24(1.13-8.22), heterogeneity 2.53(1.41-4.42), surface redness 1.41(0.56-3.14). All 6 T1 cancer had multiple predictive findings of AH, and the percentage of non-CSP performed was 100%.

Conclusion

Six predictive findings of AH combined with image-enhanced endoscopy were shown to be useful in determining the indication for CSP (UMIN 000040974).

References

1. Ito A, Suga T, Ota H, et al. Resection depth and layer of cold snare polypectomy versus endoscopic mucosal resection. J Gastroenterol 2018;53:1171-1178.
2. Uraoka T, Takizawa K, Tanaka S, et al. Guidelines for Colorectal Cold Polypectomy (supplement to "Guidelines for Colorectal Endoscopic Submucosal Dissection/Endoscopic Mucosal Resection"). Dig Endosc 2022;34:668-675.
3. Iwai T, Imai K, Hotta K, et al. Endoscopic prediction of advanced histology in diminutive and small colorectal polyps. J Gastroenterol Hepatol 2019;34:397-403.