Introduction

The metaverse consists of interconnected 3D virtual worlds that offer immersive experiences to users by stimulating a sense of presence, primarily through wearable headsets. Extended reality (XR), the core technology of the metaverse, combines virtual reality (VR), augmented reality (AR), and mixed reality (MR) to provide a shared online space where users can interact in both physical and virtual worlds.

Aims & Methods

We developed a holography-guided surgical navigation system based on the metaverse platform to improve spatial awareness in gastrointestinal (GI) and hepatobiliary-pancreatic (HPB) laparoscopic and robotic surgeries and investigated its usefulness.
We performed 40 endoscopic/laparoscopic GI and HPB surgeries using a metaverse-based holography-guided surgical navigation system. Over 20 Asian surgeons used a head-mounted display (HMD) during surgery. Holographic 3D guides of organs, lesions, and procedures were created based on individual patient CT/MRI images, providing a realistic 3D view in real space above the surgical field. Simultaneously, lifelike 3D avatars of surgeons were created, allowing for remote surgical planning and discussions in the metaverse.

Results

In all 40 surgeries, tumor lesions, surrounding organs, and blood vessels were accurately spatially reproduced as holographic images in a sterile space above the patient's abdomen, along with resection lines to guide surgical techniques. The system setup took less than 5 minutes on average, and no postoperative adverse events occurred in any patient. Positioning errors in superimposing the virtual models were within 2mm. Holographic avatars increased the efficiency and effectiveness of reviewing, brainstorming, and sharing virtual content. Holographic guidelines allowed surgeons to share their positions and hand movements, proving useful for hands-on training.

Conclusion

Our holography-guided surgical navigation system with the metaverse platform facilitated holographic telepresence through augmented reality, transforming the operating room into a 3D workspace. By allowing users to connect remotely and interact with lifelike avatars, our system reduced surgeon stress and unlocks new levels of computing beyond spatial limitations, regardless of any devices were used. We are currently evaluating its usefulness in gastroenterology by applying it to navigation guides for hepatic US-guided radiofrequency ablation (RFA) and endoscopic mucosal disection (ESD) of the gastrointestinal tract.

References

• Maki Sugimoto. Cloud XR (Extended Reality: Virtual Reality・Augmented Reality・Mixed Reality) and 5G networks for holographic medical image-guided surgery and telemedicine. Multidisciplinary Computational Anatomy - Principles and Clinical Application of MCA-based Medicine. pp 381-338, 2021. Editor: Hashizume M. Springer, Singapore. 2021. DOI https://doi.org/10.1007/978-981-16-4325-5_52
• Maki Sugimoto. Extended Reality (XR:VR/AR/MR), 3D Printing, Holography, A.I., Radiomics, and Online VR Tele-Medicine for Precision Surgery. Surgery and Operating Room Innovation. p65-70. Editor: Takenoshita S, Yasuhara H. Springer, Singapore. 2020. DOI https://doi.org/10.1007/978-981-15-8979-9_7
• Maki Sugimoto. Augmented Tangibility Surgical Navigation Using Spatial Interactive 3-D Hologram zSpace with OsiriX and Bio-Texture 3-D Organ Modeling. IEEE 2015 International Conference on Computer Application Technologies (CCATS 2015). 189-194, 2015. DOI:10.1109/CCATS.2015.53

Disclosure

Holoeyes Inc.; Board member