The Application of the Preoperative Image-Guided 3D Visualization Supported by Machine Learning to the Prediction of Organs Reconstruction During Pancreaticoduodenectomy via a Head-Mounted Displays

Abstract

Early pancreatic cancer diagnosis and therapy drastically increase the chances of survival. Tumor visualization using CT scan images is an important part of these processes. In this paper, we apply Mixed Reality (MR) and Artificial Intelligence, in particular, Machine Learning (ML) to prepare image-guided 3D models of pancreatic cancer in a population of oncology patients. Object detection was based on the convolution neural network, i.e. the You Only Look Once (YOLO) version 7 algorithm, while the semantic segmentation has been done with the 3D-UNET algorithm. Next, the 3D holographic visualization of this model as an interactive, MR object was performed using the Microsoft HoloLens2. The results indicated that the proposed MR and ML-based approach can precisely segment the pancreas along with suspected lesions, thus providing a reliable tool for diagnostics and surgical planning, especially when considering organ reconstruction during pancreaticoduodenectomy.