Abstract
Purpose
Three-dimensional (3D) surgical planning is widely accepted in liver surgery. Currently, the 3D reconstructions are usually presented as 3D PDF data on regular monitors. 3D-printed liver models are sometimes used for education and planning.
Methods
We developed an immersive virtual reality (VR) application that enables the presentation of preoperative 3D models. The 3D reconstructions are exported as STL files and easily imported into the application, which creates the virtual model automatically. The presentation is possible in “OpenVR”-ready VR headsets. To interact with the 3D liver model, VR controllers are used. Scaling is possible, as well as changing the opacity from invisible over transparent to fully opaque. In addition, the surgeon can draw potential resection lines on the surface of the liver. All these functions can be used in a single or multi-user mode.
Results
Five highly experienced HPB surgeons of our department evaluated the VR application after using it for the very first time and considered it helpful according to the “System Usability Scale” (SUS) with a score of 76.6%. Especially with the subitem “necessary learning effort,” it was shown that the application is easy to use.
Conclusion
We introduce an immersive, interactive presentation of medical volume data for preoperative 3D liver surgery planning. The application is easy to use and may have advantages over 3D PDF and 3D print in preoperative liver surgery planning. Prospective trials are needed to evaluate the optimal presentation mode of 3D liver models.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Background
In hepatobiliary surgery, three-dimensional (3D) models of the liver with its vascular structures are reported to be helpful in preoperative planning by suggesting a shorter operation time, a reduction of the resection scope, or the amount of bleeding [1,2,3]. To date, these 3D reconstructions are mostly presented as 3D PDFs and only rarely as 3D-printed models. Some publications describe the presentation in preoperative planning by means of a 3D-printed model [4, 5] that gives a good impression of tumor location especially in multifocal tumor burden and when vascular invasion with the necessity of reconstruction is suspected [6]. The potential drawback of 3D prints is a time delay for the printing process and higher costs [7]. Viewing the model on a regular screen as a 3D PDF has the limitation of depending on the viewer’s mental ability to transfer images into 3D structures, but enables editing functions of the model like the change of the transparency of the different structures [8, 9]. With the technical progress and increasing affordable availability of virtual reality (VR) devices, medical applications have been introduced using VR as well. Thus, VR also represents an interesting new alternative for the presentation of these preoperative 3D models in medicine [10]. With this article, we introduce an immersive VR tool for interactive preoperative hepatobiliary surgery planning.
Methods and workflow
To obtain a valid 3D visualization of the liver for surgical planning, a CT or MRI scan with contrasted hepatic artery and portal and hepatic veins must be available. The reconstruction can either be performed by an external provider or locally with (semi-)automatic segmentation software. In our department, semi-automatic, server-based software (Synapse 3D, FUJIFILM Europe GmbH, Düsseldorf) is utilized by trained members of our surgical team [11]. DICOM (Digital Imaging and Communications in Medicine) data can easily be transferred from the radiologic PACS System (Sectra AB, Linköping, Sweden) to synapse 3D. Transfer and reconstruction take about 2 h as previously described [11]. After completion of the reconstruction, the visualization is exported as stereolithography (STL) files, which are the standardized file format for 3D printing. The different structures (parenchyma, tumor, hepatic veins, portal vein, hepatic artery) are exported as separate files (Fig. 2).
Our application, which was developed using the game engine Unity (version 2019.2.14f1, Unity Technologies, San Francisco, CA, USA) enables an easy upload of these STL files (via drag and drop) and automatically generates the VR 3D model. The duration of the data transfer of the STL files and the calculation of the VR model takes no longer than 1 min. The application provides native support for all OpenVR compatible VR headsets, including the HTC Vive Pro (HTC Corporation, Taoyuan City 330, Taiwan), which is used in the current setup. The surgeon can view and edit the 3D model while wearing the VR headsets. To interact with the 3D liver model, both HTC Vive controllers can be used. Moving and rotating of the model are realized via a virtual “laser pointer.” By pointing towards the model and pressing a specific button, the model is attached to the intersection point and can be moved and rotated. To provide finer control over rotation, it is possible to use the touchpad of each controller: By pressing up/down or left/right, the model is rotated around the vertical or horizontal axis. Additionally, the model can be scaled by pressing and holding dedicated buttons on both controllers simultaneously; moving the controllers apart increases the model size and vice versa.
A menu that can be shown and hidden as required by pressing a certain button gives an overview of all available structures. By interacting with a slider, the opacity can be changed from invisible over semi-transparent to fully opaque. In addition, surgeons can draw potential resection lines on the liver surface. All these functions can be used in a single or multi-user mode, which enables an interactive preoperative planning and discussion by the complete surgical team (Fig. 1, Supplementary Video 1)
(MP4 195833 kb).
Brief clinical evaluation
Aside from the incremental feedback from the clinical development team, we asked five highly experienced HPB surgeons of our department to evaluate the application after using it for the first time. Neither of the five surgeons had any experience with VR technology before. The evaluation was carried out using the established “System Usability Scale” (SUS) [12]. Ten items on the system’s practicability and complexity as well as the user’s confidence to use and re-use it are rated on 5-point Likert scales. The evaluation using the Likert scale can be converted to a value of 0–100 by a fixed formula and would represent the best possible applicability at the maximum value of 100% (0–50%: not acceptable, 51–67%: poor, 68%: OK; 69–80%: good, 81–100%: excellent). The five evaluating surgeons from our department reached an average value of 76.6 %, which classified the application as rather easy to use, with 4 out of 5 surgeons giving a good to very good rating (80–95 %). One surgeon was not convinced by the application in its current form for daily clinical use, which resulted in a rating of 45%. Nevertheless, the participant emphasized the future potential. All 5 surgeons rated the learning effort to use the system as low (0.2 on the Likert scale 0–4), which positively highlights the system’s ease of use. Additionally, we discussed the potential use of the application and the competing demonstration modalities (PDF, printed model, and AR) (Table 1).
The demonstration of the system to the HPB experts was independent of a specific surgical procedure. Thus, we used cases that were available in our collection of 3D reconstructions that have been printed for surgical planning [6]. Thus, intraoperative feedback regarding the preoperative visualization in immersive VR from the surgeons was not collected and will be part of future investigations in a clinical trial.
Discussion
Until today, there is little literature about the application of VR as a representation modality of 3D models in liver surgery. Most publications describe the representation of such models in preoperative planning by means of 3D PDF or 3D prints [4, 8]. The term virtual resection planning is often misleading and usually refers to the ability to draw resection lines on a reconstructed liver with volume calculation. However, these applications are currently presented on 2D monitors. We have developed an immersive application that demonstrates 3D models in virtual reality using VR headsets and furthermore enables interaction with the model (Supplementary Video 1). An essential requirement for a VR application as a presentation modality for preoperative surgical planning is the correct presentation of the original data. This is ensured by this application (Fig. 2). In addition, this technology avoids the disadvantages of viewing the models on a 2D screen, and at the same time, it enables almost all advantages of 3D printing, especially moving and grasping the models. In VR, these movements can be carried out with controllers. The drawback of surface reflection, which can arise from the surface of the 3D print model at certain angles, is also avoided in VR. Furthermore, various interactions can be used to “show or hide” overlaps of pathologies and vessels of the liver in the VR application for better understanding. This is not possible with a 3D print. However, the lacking haptic interaction with the 3D model with the surgeon’s own hands but with the controllers can be a limitation of the VR application and will be a part of future research by combining 3D prints and VR [13] or with specialized VR gloves. One advantage of imaging in a 3D pdf format and viewing on a 2D screen—besides the broad availability and low costs—is currently the possibility to enable volume calculation, provided that a volume segmentation of the liver was initially performed and specially exported. This is not yet included in this application but part of current investigations. However, for an accurate and on-demand volume calculation directly in the immersive VR application, proper data format is DICOM instead of the currently used STL files.
The intraoperative use of the 3D visualization of preoperative image data is a field that has been widely investigated. Recent developments in augmented reality (AR) technologies enable the spectating of 3D reconstructions in a sterile field using speech or gesture commands [14, 15]. The live alignment of the preoperative data in the operating room during liver surgery with the currently available AR technology is lacking accuracy but will hopefully be possible with technological advances. The intraoperative use of this VR application is conceivable, but at the moment certainly limited due to the loss of sight of the actual surgical field. Another limitation is the initial costs for the VR equipment as well as the increased technical effort due to the different computer programs and their handling. The advantages of the more established 3D presentation modalities may be that a 3D-printed model can enable a more plastic preoperative patient information about their upcoming surgery [5]. A 3D PDF may be suitable when the presentation on a screen covers the need e.g. of a tumor board. However, virtual meetings and the presentation of 3D data in immersive VR are already possible and will probably gain wider acceptance and use [16]. Also, medical education and specialized HPB apprentices may profit from immersive virtual reality applications due to the interactive use and the possibility to interact with multiple participants and over distances. This is in concordance with recent literature that evaluated a similar VR application to be suited mostly for student and resident training [17]. To date, this is the main development goal of the presented application. However, all these possibilities have to be put into perspective since technological advances will make VR and AR applications more available and more affordable for a broad field of users. The integration of patient data into VR applications needs to be evaluated regarding the connection to hospital information systems (HIS) with the necessary data security ensured. Table 1 gives an overview of the advantages and disadvantages of the mentioned technologies as feedback from the evaluating HPB specialists. However, trials are needed to evaluate which presentation modality suits the different areas of application in surgery, e.g., preoperative planning or education.
In conclusion, we present an immersive VR application for preoperative 3D liver surgical planning. The application is easy to use and may have advantages over 3D PDF and 3D print in preoperative liver surgery planning. Prospective trials are needed to evaluate the optimal presentation mode of 3D liver reconstructions.
References
Mutter D, Dallemagne B, Bailey C, Soler L, Marescaux J (2009) 3D virtual reality and selective vascular control for laparoscopic left hepatic lobectomy. Surg Endosc 23(2):432–435. https://doi.org/10.1007/s00464-008-9931-y
Cai W, Fan Y, Hu H, Xiang N, Fang C, Jia F (2017) Postoperative liver volume was accurately predicted by a medical image three dimensional visualization system in hepatectomy for liver cancer. Surg Oncol 26(2):188–194. https://doi.org/10.1016/j.suronc.2017.03.006
Uchida M (2014) Recent advances in 3D computed tomography techniques for simulation and navigation in hepatobiliary pancreatic surgery. J Hepatobiliary Pancreat Sci 21(4):239–245. https://doi.org/10.1002/jhbp.82
Bangeas P, Tsioukas V, Papadopoulos VN, Tsoulfas G (2019) Role of innovative 3D printing models in the management of hepatobiliary malignancies. World J Hepatol 11(7):574–585. https://doi.org/10.4254/wjh.v11.i7.574
Lopez-Lopez V, Robles-Campos R, Garcia-Calderon D, Lang H, Cugat E, Jimenez-Galanes S, Fernandez-Cebrian JM, Sanchez-Turrion V, Fernandez-Fernandez JM, Barrera-Gomez MA, de la Cruz J, Lopez-Conesa A, Brusadin R, Gomez-Perez B, Parrilla-Paricio P (2020) Applicability of 3D-printed models in hepatobiliary surgey: results from "LIV3DPRINT" multicenter study. HPB (Oxford). https://doi.org/10.1016/j.hpb.2020.09.020
Huber T, Huettl F, Tripke V, Baumgart J, Lang H (2021) Experiences With Three-dimensional Printing in Complex Liver Surgery. Ann Surg 273(1):e26–e27. https://doi.org/10.1097/SLA.0000000000004348
Perica ER, Sun Z (2018) A systematic review of three-dimensional printing in liver disease. J Digit Imaging 31(5):692–701. https://doi.org/10.1007/s10278-018-0067-x
Yang T, Lin S, Xie Q, Ouyang W, Tan T, Li J, Chen Z, Yang J, Wu H, Pan J, Hu C, Zou Y (2019) Impact of 3D printing technology on the comprehension of surgical liver anatomy. Surg Endosc 33(2):411–417. https://doi.org/10.1007/s00464-018-6308-8
Newe A, Ganslandt T (2013) Simplified generation of biomedical 3D surface model data for embedding into 3D portable document format (PDF) files for publication and education. PLoS One 8(11):e79004. https://doi.org/10.1371/journal.pone.0079004
Pfeiffer M, Kenngott H, Preukschas A, Huber M, Bettscheider L, Muller-Stich B, Speidel S (2018) IMHOTEP: virtual reality framework for surgical applications. Int J Comput Assist Radiol Surg 13(5):741–748. https://doi.org/10.1007/s11548-018-1730-x
Paschold M, Huettl F, Kneist W, Boedecker C, Poplawski A, Huber T, Lang H (2020) Local, semi-automatic, three-dimensional liver reconstruction or external provider? An analysis of performance and time expense. Langenbeck's Arch Surg 405:173–179. https://doi.org/10.1007/s00423-020-01862-7
Borsci S, Federici S, Lauriola M (2009) On the dimensionality of the System Usability Scale: a test of alternative measurement models. Cogn Process 10(3):193–197. https://doi.org/10.1007/s10339-009-0268-9
Reinschluessel AV, Muender T, Uslar V, Weyhe D, Schenk A, Malaka R (2019) Tangible organs: introducing 3D printed organ models with VR to interact with medical 3D models. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems May 2019 Paper No.: LBW1816, pp 1–6. https://doi.org/10.1145/3290607.3313029
Ruger C, Feufel MA, Moosburner S, Ozbek C, Pratschke J, Sauer IM (2020) Ultrasound in augmented reality: a mixed-methods evaluation of head-mounted displays in image-guided interventions. Int J Comput Assist Radiol Surg 15(11):1895–1905. https://doi.org/10.1007/s11548-020-02236-6
Sauer IM, Queisner M, Tang P, Moosburner S, Hoepfner O, Horner R, Lohmann R, Pratschke J (2017) Mixed Reality in Visceral Surgery: Development of a Suitable Workflow and Evaluation of Intraoperative Use-cases. Ann Surg 266(5):706–712. https://doi.org/10.1097/SLA.0000000000002448
Bashkanov O, Saalfeld P, Gunasekaran H, Jabaraj M, Preim B, Huber T, Hüttl F, Kneist W, Hansen C (2019) VR multi-user conference room for surgery planning. 18 Jahrestagung der Deutschen Gesellschaft für Computer- und Roboterassistierte Chirurgie eV : 19-21 September 2019, Reutlingen Tagungsband (2019):S. 264-268
Kenngott HG, Pfeiffer M, Preukschas AA, Bettscheider L, Wise PA, Wagner M, Speidel S, Huber M, Nickel F, Mehrabi A, Muller-Stich BP (2021) IMHOTEP: cross-professional evaluation of a three-dimensional virtual reality system for interactive surgical operation planning, tumor board discussion and immersive training for complex liver surgery in a head-mounted display. Surg Endosc. https://doi.org/10.1007/s00464-020-08246-4
Acknowledgements
Open Access funding enabled and organized by Projekt DEAL.
Funding
Tobias Huber receives intramural funding from the University Medical Center Mainz and the Federal Ministry of Education and Research (BMBF: 16SV8059 “AVATAR”).
Tobias Huber and Werner Kneist receive funding by the Federal Ministry of Education and Research (BMBF: 16SV8059).
Christian Hansen and Bernhard Preim receive funding by the Federal Ministry of Education and Research (BMBF: 16SV8054).
Florentine Huettl and Markus Paschold received intramural funding from the University Medical Center Mainz.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.
Informed consent
Informed consent was obtained from the evaluating liver surgeons.
Conflict of interest
The authors declare no competing interests.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Boedecker, C., Huettl, F., Saalfeld, P. et al. Using virtual 3D-models in surgical planning: workflow of an immersive virtual reality application in liver surgery. Langenbecks Arch Surg 406, 911–915 (2021). https://doi.org/10.1007/s00423-021-02127-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00423-021-02127-7