Artificial intelligence-based technology for enhancing the quality of simulation, navigation, and outcome prediction for hepatectomy

AI technology for 3D simulation

The intrahepatic vascular structure and accurate liver segmentation and volumetry must be identified to ensure precise and safe liver surgery[Table 1]. Three-dimensional simulation software has been applied to reconstruct intrahepatic structures and calculate future remnant liver volume^[22]. Previous studies using deep learning-based algorithms for the automatic extraction of portal veins and hepatic veins found that the deep learning model contributed to reducing the processing time^[3,4]. Chen et al. reported that with the use of the residual-dense-attention U-net model, a CNN, accurate segmentation of the liver and liver tumor on CT images could be obtained^[5]. Koitka et al. demonstrated that a CNN provided fully automated 3D volumetry of the right and left liver on CT images^[6]. Mojtahed et al. proposed a novel medical software (Hepatica) for performing automatic liver volumetry, followed by semiautomatic delineation of the Couinaud segments^[7]. CNN can automatically delineate the liver from a 3D T1-weighted magnetic resonance image and segment the volume corresponding to the liver. The new software could accurately delineate the liver and divide the volume into Couinaud segments. Lyu et al. used a novel approach to train convolutional networks for liver tumor segmentation using Couinaud segment annotation, which complies relatively better with the radiologists’ work practice and significantly reduces manual effort^[8]. The new method can use these annotations to estimate pseudo tumor masks as pixel-wise supervision for training a fully supervised tumor segmentation model. AI technology has contributed to the extraction of detailed and precise data on vascular vessels and liver segmental volumes.

AI technology for 3D and 4D printing

Three-dimensional printing is an innovative technology for tissue and organ fabrication, medical education, and preoperative planning. The use of 3D-printed liver models allows surgeons to obtain accurate information regarding vessel anatomy, the relationship between the tumor and vessels, and the parenchymal transection plane. Surgeons can freely handle the patient’s liver before surgery. In addition, 3D-printed liver models can be used to train new surgeons^[23,24]. Various materials such as polymers and hydrogels are used to fabricate the 3D-printed structure, and a complex creation process, such as the extrusion of feedstock material and building components layer by layer with dimensional accuracy, is needed. Meiabadi et al. reported that an artificial neural network-based method can enhance the accuracy of modeling for toughness, part thickness, and production cost-dependent variables^[25]. Rojek et al. showed the utility of AI-based design for 3D printing in saving materials and reducing waste^[26]. Recently, 4D printing has emerged, in which the fourth dimension of time is added to 3D printing, connecting the change of shape, properties, and functionality of the printed material over time following stimuli. An AI algorithm can be used to determine the best design of the toolpath and the stimuli-responsive material distribution, allowing precise shape control of the 4D-printed structure. AI technology can also ameliorate the design of 4D printing using a library of previous scans of the target region of interest and coupling it with incomplete anatomy scan data to reconstruct a patient-specific 4D-printing model. AI-based 4D printing can improve the form and function of the materials in shape- changing and shape memory^[27].

AI technology for AR and intraoperative ultrasound

Intraoperative navigation techniques, which began with intraoperative ultrasound, may help surgeons perform liver surgery. Recently, AR has been applied to assist the operator in minimally invasive surgery. Liver tumors and vascular and biliary structures reconstructed using preoperative CT images are projected on the liver surface during liver parenchymal transection^[28]. Adballah et al. used AR software during the laparoscopic resection of liver tumors^[29]. Pseudotumor was created in sheep cadaveric liver, and a virtual preoperative 3D model was reconstructed using CT imaging. When the tumor image and 1-cm peritumoral margins were projected onto the liver surface during AR laparoscopic liver resection, the resection margins were more accurate and had less variability than those obtained using standard ultrasonographic navigation. CNN has been used to obtain real-time semantic segmentation of the scene and improve the precision of the subsequent 3D enhancement for an in-vivo robot-assisted radical prostatectomy^[9]. Lin et al. proposed using a dual-modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by a CNN model^[10]. Structured light images are used to recover the depth maps of tissue surfaces using a fully convolutional network. The spectrographic and RGB images were jointly processed by a CNN-based super-resolution model to generate pixel-level dense hypercubes. By combining the depth maps and hypercubes using AR, surgeons can visualize the recovered 3D surfaces, narrow-band images, and oxygen saturation maps. Luo et al. evaluated the utility and accuracy of the proposed AR navigation system for performing liver resection by a stereoscopic laparoscope using five modules: hand-eye calibration, preoperative image segmentation, intraoperative liver surface reconstruction, image-to-patient registration, and AR navigation^[11]. An automatic CNN-based algorithm was used to segment the liver model using preoperative CT images. An unsupervised CNN framework was introduced to estimate the depth while reconstructing the intraoperative 3D model for registration. AI systems have also been applied in intraoperative ultrasound. Barash et al. developed an AI system to detect liver lesions in intraoperative ultrasound. The area under the curve (AUC) of the algorithm performance was 80.2%, and the overall classification accuracy was 74.6%. The algorithm was found to assist in identifying focal liver lesions in intraoperative ultrasound performed by the liver surgeon^[14].

AI technology for fluorescence imaging using ICG

ICG is mainly used as a fluorogenic reagent for fluorescence imaging-guided surgery. Protein-bound ICG emits fluorescence that peaks at approximately 840 nm when illuminated with near-infrared light (750 nm-810 nm)^[30]. Because it is hard for this wavelength to be absorbed by hemoglobin or water, structures that contain ICG can be visualized through human tissue thicknesses of up to 5 mm-10 mm using a near-infrared camera system. ICG fluorescence imaging is used in hepatobiliary surgery to visualize the anatomy of the bile duct, hepatic tumors, and hepatic segmental areas. Intravenous injection of ICG during surgery allows fluorescence images of the bile ducts to be obtained in the surgical field. Fluorescence cholangiography provides detailed information on the anatomy of the extrahepatic bile duct. At first, fluorescent images of the biliary tract are displayed on a monitor with standard spatial resolution images. Switching from standard images to fluorescence images is required^[31]. The high sensitivity of image sensors and advances in signal-processing technology have allowed for the application of fluorescent imaging in laparoscopic surgery^[32,33]. Recent advances in imaging technology have enabled the fusion of fluorescence and full-color visible images with high-resolution quality^[34] and allowed for the application of fluorescence imaging to laparoscopic liver surgery^[35]. In addition, deep learning-based algorithms have been applied to fuse fluorescence images with visible light images. The deep learning fusion method is based on CNNs and can achieve a good infrared and visible image fusion effect^[21]. Liu et al. used a CNN to obtain a weight map and used image pyramids to fuse infrared and visible images^[36]. Zhang et al. proposed an adaptive brightness fusion method using the deep learning fusion method to fuse intraoperative near-infrared fluorescence and visible images^[21]. Shen et al. applied a deep CNN to capture fluorescence imaging to determine glioma quickly and accurately in real-time during surgery. The developed deep CNN combined with the second near-infrared window fluorescence images can predict the pathological diagnosis while achieving an AUC of 0.945 during surgery^[37].

The CNN architecture has also been applied to fluorescence lifetime imaging microscopy (FLIM). FLIM is an imaging technique that uses the inherent properties of fluorescent dyes. It identifies different intensity patterns and the lifetime of autofluorescence between cancerous tissues, margins, and normal tissues^[38]. CNNs can reduce the acquisition time required to reconstruct pixel raw fluorescence data into intensity and lifetime images^[39]. Marden et al. reported that a CNN allows for accurate and rapid localization and visualization of aiming beam segmentation during FLIM acquisition^[40].