Abstract
Introduction
Although robotic-assisted surgery is increasingly performed, objective assessment of technical skills is lacking. The aim of this study is to provide validity evidence for objective assessment of technical skills for robotic-assisted surgery.
Methods
An international multicenter study was conducted with participants from the academic hospitals Heidelberg University Hospital (Germany, Heidelberg) and the Amsterdam University Medical Centers (The Netherlands, Amsterdam). Trainees with distinctly different levels of robotic surgery experience were divided into three groups (novice, intermediate, expert) and enrolled in a training curriculum. Each trainee performed six trials of a standardized suturing task using the da Vinci Surgical System. Using the ForceSense system, five force-based parameters were analyzed, for objective assessment of tissue handling skills. Mann–Whitney U test and linear regression were used to analyze performance differences and the Wilcoxon signed-rank test to analyze skills progression.
Results
A total of 360 trials, performed by 60 participants, were analyzed. Significant differences between the novices, intermediates and experts were observed regarding the total completion time (41 s vs 29 s vs 22 s p = 0.003), mean non zero force (29 N vs 33 N vs 19 N p = 0.032), maximum impulse (40 Ns vs 31 Ns vs 20 Ns p = 0.001) and force volume (38 N3 vs 32 N3 vs 22 N3 p = 0.018). Furthermore, the experts showed better results in mean non-zero force (22 N vs 13 N p = 0.015), maximum impulse (24 Ns vs 17 Ns p = 0.043) and force volume (25 N3 vs 16 N3 p = 0.025) compared to the intermediates (p ≤ 0.05). Lastly, learning curve improvement was observed for the total task completion time, mean non-zero force, maximum impulse and force volume (p ≤ 0.05).
Conclusion
Construct validity for force-based assessment of tissue handling skills in robot-assisted surgery is established. It is advised to incorporate objective assessment and feedback in robot-assisted surgery training programs to determine technical proficiency and, potentially, to prevent tissue trauma.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Robotic-assisted surgery (RAS) requires an advanced technical skill set. Due to increased degrees of freedom, the learning curve for RAS is less steep than for laparoscopy and skills that need to be developed are eye-hand coordination, control of the camera and the third arm as primary surgeon, switching between instrument arms, bimanual dexterity, needle handling and suturing, depth perception and tissue manipulation. Therefore, it is important that sufficient time and attention is devoted to achieving and optimizing this skill set for performing RAS [1,2,3]. Especially as there is no haptic feedback and a reduced tissue feeling. This can be safely obtained by deliberate training in a non-clinical/simulation environment [4].
Previously, our research group validated and analyzed the effect and importance of objective force, motion and time feedback for laparoscopic simulation training [5,6,7]. There are various modalities for RAS training. The most frequent modality consists of virtual reality training, box training with wet lab (cadaver models) or dry lab (suture pads and biotissue) [8, 9]. The most widely used virtual reality trainer is the da Vinci Skills Simulator (dVSS, Mimic Technologies, Seattle WA, USA) assessing a number of objective parameters: total completion time, instrument collisions, time of excessive force, instruments out of view, economy of motion and master workspace [10]. While VR simulations can be very realistic and provide a good approximation of the tasks and procedures being trained, current simulators cannot naturally replicate the complexity and unpredictability of real-life situations. In particular, for example, the unpredictability and physics of suturing and knot tying.
Box training still often consists of real time or video assessment with subjective forms [11,12,13,14]. Despite, the increased risk of unintentional tissue damage due to the absence of haptic feedback in most robotic surgery systems, current training curricula with virtual reality (VR) and box training lack the ability to train and assess tissue manipulation and the application of forces. This complicates training safe tissue manipulation in RAS.
The aim of this study is to obtain construct validity evidence for an objective force and time measuring system in robotic surgery training and to analyze the learning curves.
Methods
Study design
In this international multicenter prospective study, participants from two academic hospitals were included: the Dept. of Surgery Heidelberg University Hospital (Germany, Heidelberg) and the Amsterdam University Medical Centers (The Netherlands, Amsterdam).
Participants
Participants were classified and divided into three groups based on their prior robotic surgery experience as operator: novices consisting of junior residents (0 cases and exposure to the system), intermediates consisting of senior residents and young surgeons (< 15 robotic-assisted surgery cases as console surgeon and > 25 robotic knots) and experts consisting of attendees/robot experts (> 15 robotic-assisted surgery cases as console surgeon and > 50 robotic knots) (Fig. 1).
Systems and hardware
The da Vinci Si Surgical System (Intuitive Surgical Inc., Sunnyvale, California USA) and a box trainer equipped with a suture pad were used for the robotic suturing task (Supplemental File A). The trials were performed using braided multifilament sutures: 12 cm Novosyn 2/0 HR26 (BBraun, Melsungen, Germany). The trials were assessed using the objective ForceSense measuring system (Medishield, Delft, The Netherlands). The ForceSense provided feedback regarding maximum force, mean force, max impulse, force volume and time (Table 1) [6, 15]. All trials were recorded and uploaded to an online database.
Protocol
All trainees received a brief instruction on the da Vinci Surgical System. The trainees were introduced to the technical aspects of the da Vinci Surgical System: ergonomics, user interface, controls, EndoWrist, camera, and clutching. Each trainee performed six separate repetitions of the interrupted robotic surgical suture and knot tying task. Furthermore, the participants received instructions and a video regarding the surgical suture and knot tying task. The suture consisted of a square knot and an additional loop to lock the suture. The camera and the suture were prepared in advance in a standardized fashion and after a countdown, the training task began. This was considered as one trial of the task. All six trials were performed consecutively on the same day.
Statistical analyses
Data from the online database was analyzed using IBM SPSS statistics 28 (SPSS Inc., Chicago, Illinois USA). Descriptive statistics and frequency measurements were performed to determine the means and standard deviation. GraphPad (Prism 9.0.0, San Diego, California USA) was used for boxplots of the outcomes. Shapiro–Wilk test was performed and the data was not normally distributed. Post hoc power analyses after the initial inclusions was performed (Supplemental file E).
Construct validity
A Kruskal–Wallis test was performed to determine whether significant differences were prevalent between the novices, intermediates and experts. Subsequently, Mann–Whitney U tests were used to analyze the differences between the groups individually.
To strengthen the construct validation analyses (in addition to comparing outcomes from the different experience groups) a linear regression was also performed. Linear regression analyzed the effect of robotic surgery experience on the different outcome measurements.
Learning curve
To determine overall progression of skills and the effectiveness of this short course, the Wilcoxon signed-rank test was conducted to compare the outcomes of the first and the last performed trial. An outcome with a p < 0.05 was regarded as statistically significant.
Results
A total of 360 repetitions, performed by 60 participants, were included for analyses. Nineteen of the participants were female and one of the participants had a dominant left hand.
Construct validation comparing novices, intermediates and experts
Significant differences between the novices, intermediates and experts were observed regarding the total completion time (41 s vs 29 s vs 22 s p = 0.003), mean non zero force (29 N vs 33 N vs 19 N p = 0.032), maximum impulse (40 Ns vs 31 Ns vs 20 Ns p = 0.001) and force volume (38 N3 vs 32 N3 vs 22 N3 p = 0.018) (Supplemental file B, Table B1). The intermediates and experts had significant better results compared to the novices for all parameters (Table 2 and Figs. 2, 3, 4) (Supplemental file B, Table B2–B3). Furthermore, the experts showed better results in mean non-zero force (22 N vs 13 N p = 0.015), maximum impulse (24 Ns vs 17 Ns p = 0.043) and force volume (25 N3 vs 16 N3 p = 0.025) compared to the intermediates (p ≤ 0.05) (Supplemental file B, Table B4).
Construct validation using linear regression analyses
The regression analysis indicate clear regression of the total task time (p ≤ 0.05) and partial regression for the maximum impulse (Trial 1 p ≤ 0.001, Trial 2 p = 0.050 and Trial 6 p = 0.007), maximum force (Trial 1 p = 0.029), force volume (Trial 1 p = 0.006 and Trial 2 p = 0.029) and mean non-zero force (Trial 1 p = 0.023) (Supplemental file C, Table C1–C5).
Learning curve analysis
Comparing the first and sixth trial, all groups were able to complete the trials faster: novices (164 s vs. 97 s p = 0.011), intermediates (103 s vs. 89 s p = 0.011) and experts (106 s vs. 72 s p = 0.002) (Table 3). Moreover, a decrease in the mean non-zero force was observed in the novice group (1.38 N vs. 1.22 N p = 0.040) and expert group (1.07 N vs. 1.00 N p = 0.017). Similarly, a decrease in max impulse was found in the novice group (27.01 Ns vs. 15.09 Ns p = 0.016) and expert group (14.74 Ns vs. 10.45 Ns p = 0.017). Lastly, the expert group were able to decrease the force volume (0.98 N3 vs. 0.66 N3 p = 0.019).
Discussion
This study showed construct validity evidence as the objective measuring system was able to show clear differences on objective force parameters and distinguish between novices, intermediates and experts on the da Vinci Surgical System. These difference are in line with our prior findings for laparoscopy [5, 7]. The system is not exclusively able to distinguish novices and experts but also intermediates and experts. The experts showed lower completion time and force parameter outcomes in almost all trials for all parameters.
Furthermore, all three groups improved in the completion time when comparing the first and last trial. The novices and experts improved in mean non-zero force, maximum impulse and force volume. This improvement of skill, based on objective measurements, and the learning curve in robotic tissue handling skill confirms and extends our prior findings [5,6,7, 16]. This is of interest in daily practice regarding robotic suturing and knot tying, since increased psychomotor skills and hand–eye coordination translate into better tissue manipulation skills [17, 18].
Previously, the importance of forces and tissue manipulation in different experience groups was described by our research team [5, 7]. For robotic surgery, the differences are even more divergent due to the difficult learning curve of tissue treatment in RAS due to the lack of haptic feedback. Furthermore, a significant higher number of suture breakage was observed in the novice and intermediate group. In terms of clinical relevance, differences in force measurements could potentially be indicative of differences in surgical performance or outcomes. For example, if one group of trainees consistently applies higher or lower forces during training, this could potentially translate to differences in surgical precision or the risk of complications during actual surgery.
The ability to plot learning curves allows us to use regression analysis combined with artificial intelligence and machine learning to create predictive software that can be used to provide custom training to the user including an estimation of the amount of time needed on the trainers. This can have a tremendous effect on cost related factors in a time that most education budgets are under pressure.
Although direct force feedback lacks in most robotic platforms and therefore ex vivo training is of paramount importance, this study is the first to analyze and report objective interaction force assessment in robotic surgery training [19]. Some studies have researched grip force but not the force exerted on the tissue [20]. Furthermore, studies have been conducted with excessive force and instrument collisions in the da Vinci Skills Simulator (dVSS) [21,22,23,24,25,26]. However, this is limited to a virtual reality environment and does not translate to force exerted on tissue by instruments.
A strength of this study is the large sample size of participants from two international academic hospitals and thus increasing the generalizability of the construct validation. The participants performed a standardized suture and knot tying task which can not only be applied in a broad spectrum of daily practice but also is representative for the assessment and validation of laparoscopic and general surgical skill [27, 28]. Another strength is the use of previously validated objective force and time and metrics that represent instrument and tissue handling skills in laparoscopic skills training [5, 7, 29]. Furthermore, all trials were recorded and participants, peers and supervisors can review the performances. A limitation is that no motion parameters could be measured, due to the discrepancies of trocar instrument and da Vinci instrument diameter. In future studies that include combined force, motion and time parameters, the diameter of the motion sensors should be adjusted.
Conclusion
This study showed that it is possible to distinguish between different skill levels in robot-assisted surgery. This provides validity evidence and relevance for using objective assessment of tissue handling skills during simulation training for robotic surgery. Moreover, repetitive assessment showed learning curves over time. To ensure and quantify technical competency at the end of training, we advise surgical trainers to incorporate force-based assessment in robotic surgery training systems.
References
Zhao B, Lam J, Hollandsworth HM, Lee AM, Lopez NE, Abbadessa B, Eisenstein S, Cosman BC, Ramamoorthy SL, Parry LA (2020) General surgery training in the era of robotic surgery: a qualitative analysis of perceptions from resident and attending surgeons. Surg Endosc 34:1712–1721. https://doi.org/10.1007/S00464-019-06954-0
Schreuder HWR, Wolswijk R, Zweemer RP, Schijven MP, Verheijen RHM (2012) Training and learning robotic surgery, time for a more structured approach: a systematic review. BJOG 119:137–149. https://doi.org/10.1111/J.1471-0528.2011.03139.X
Sridhar AN, Briggs TP, Kelly JD, Nathan S (2017) Training in robotic surgery—an overview. Curr Urol Rep. https://doi.org/10.1007/S11934-017-0710-Y
Azadi S, Green IC, Arnold A, Truong M, Potts J, Martino MA (2021) Robotic surgery: the impact of simulation and other innovative platforms on performance and training. J Minim Invasive Gynecol 28:490–495. https://doi.org/10.1016/J.JMIG.2020.12.001
Hardon SF, van Gastel LA, Horeman T, Daams F (2021) Assessment of technical skills based on learning curve analyses in laparoscopic surgery training. Surgery 170:831–840. https://doi.org/10.1016/J.SURG.2021.04.024
Horeman T, Dankelman J, Jansen FW, Van Den Dobbelsteen JJ (2014) Assessment of laparoscopic skills based on force and motion parameters. IEEE Trans Biomed Eng 61:805–813. https://doi.org/10.1109/TBME.2013.2290052
Rahimi AM, Hardon SF, Uluç E, Bonjer HJ, Daams F (2022) Prediction of laparoscopic skills: objective learning curve analysis. Surg Endosc. https://doi.org/10.1007/S00464-022-09473-7
Puliatti S, Mazzone E, Dell’Oglio P (2020) Training in robot-assisted surgery. Curr Opin Urol 30:65–72. https://doi.org/10.1097/MOU.0000000000000687
Chen R, Rodrigues Armijo P, Krause C, Siu KC, Oleynikov D (2020) A comprehensive review of robotic surgery curriculum and training for residents, fellows, and postgraduate surgical education. Surg Endosc 34(1):361–367. https://doi.org/10.1007/S00464-019-06775-1
MScore - Mimic Simulation, (n.d.). https://mimicsimulation.com/mscore/ (accessed June 13, 2022).
Liang MI, McCann GA, Rath KS, Backes FJ, Cansino C, Salani R (2014) Training the next generation of robotic surgeons using guided mentorship: a randomized controlled trial. J Minim Invasive Gynecol 21:1075–1079. https://doi.org/10.1016/J.JMIG.2014.05.011
Kim HJ, Choi GS, Park JS, Park SY (2014) Comparison of surgical skills in laparoscopic and robotic tasks between experienced surgeons and novices in laparoscopic surgery: an experimental study. Ann Coloproctol 30:71–76. https://doi.org/10.3393/AC.2014.30.2.71
Holst D, Kowalewski TM, White LW, Brand TC, Harper JD, Sorenson MD, Kirsch S, Lendvay TS (2015) Crowd-sourced assessment of technical skills: an adjunct to urology resident surgical simulation training. J Endourol 29:604–609. https://doi.org/10.1089/END.2014.0616
Guni A, Raison N, Challacombe B, Khan S, Dasgupta P, Ahmed K (2018) Development of a technical checklist for the assessment of suturing in robotic surgery. Surg Endosc 32:4402–4407. https://doi.org/10.1007/S00464-018-6407-6
Horeman T, Rodrigues SP, Willem Jansen F, Dankelman J, Van Den Dobbelsteen JJ (2012) Force parameters for skills assessment in laparoscopy. IEEE Trans Haptics 5(4):312–322. https://doi.org/10.1109/TOH.2011.60
Willuth E, Hardon SF, Lang F, Haney CM, Felinska EA, Kowalewski KF, Müller-Stich BP, Horeman T, Nickel F (2021) Robotic-assisted cholecystectomy is superior to laparoscopic cholecystectomy in the initial training for surgical novices in an ex vivo porcine model: a randomized crossover study. Surg Endosc. https://doi.org/10.1007/s00464-021-08373-6
Agha RA, Fowler AJ (2015) The role and validity of surgical simulation. Int Surg 100:350–357. https://doi.org/10.9738/INTSURG-D-14-00004.1
Jones D, Jaffer A, Nodeh AA, Biyani CS, Culmer P (2018) Analysis of mechanical forces used during laparoscopic training procedures. J Endourol 32:529–533. https://doi.org/10.1089/end.2017.0894
Diez SP, Borghesan G, Joyeux L, Meuleman C, Deprest J, Stoyanov D, Ourselin S, Vercauteren T, Reynaerts D, Poorten EBV (2019) Evaluation of haptic feedback on bimanually teleoperated laparoscopy for endometriosis surgery. IEEE Trans Biomed Eng 66:1207–1221. https://doi.org/10.1109/TBME.2018.2870542
Judkins TN, Oleynikov D, Stergiou N (2008) Objective evaluation of expert performance during human robotic surgical procedures. J Robot Surg 1:307–312. https://doi.org/10.1007/S11701-007-0067-1
Lerner MA, Ayalew M, Peine WJ, Sundaram CP (2010) Does training on a virtual reality robotic simulator improve performance on the da Vinci® surgical system? J Endourol 24:467–472. https://doi.org/10.1089/END.2009.0190
Liss MA, Abdelshehid C, Quach S, Lusch A, Graversen J, Landman J, McDougall EM (2012) Validation, correlation, and comparison of the da Vinci Trainer ™ and the da Vinci surgical skills simulator™ using the Mimic™ software for urologic robotic surgical education. J Endourol 26:1629–1634. https://doi.org/10.1089/END.2012.0328
Moglia A, Perrone V, Ferrari V, Morelli L, Boggi U, Ferrari M, Mosca F, Cuschieri A (2017) Influence of videogames and musical instruments on performances at a simulator for robotic surgery. Minim Invasive Ther Allied Technol 26:129–134. https://doi.org/10.1080/13645706.2016.1267018
Wang F, Zhang C, Guo F, Sheng X, Ji J, Xu Y, Cao Z, Lyu J, Lu X, Yang B (2021) The application of virtual reality training for anastomosis during robot-assisted radical prostatectomy, Asian. J Urol 8:204–208. https://doi.org/10.1016/J.AJUR.2019.11.005
Cumpanas AA, Bardan R, Ferician O, Latcu SC, Lazar OF, Duta C (2020) The impact of tiredness on virtual reality robotic surgical skills. Wideochirurgia I Inne Tech Maloinwazyjne 15:298–304. https://doi.org/10.5114/WIITM.2020.93201
Berges AJ, Vedula SS, Malpani A, Chen CCG (2022) Virtual reality simulation has weak correlation with overall trainee robot-assisted laparoscopic hysterectomy performance. J Minim Invasive Gynecol 29:507–518. https://doi.org/10.1016/J.JMIG.2021.12.002
Emken JL, McDougall EM, Clayman RV (2004) Training and assessment of laparoscopic skills. JSLS 8:195–199
Mori T, Hatano N, Maruyama S, Atomi Y (1998) Significance of “hands-on training” in laparoscopic surgery. Surg Endosc 12:256–260. https://doi.org/10.1007/s004649900646
Horeman T, Van Delft F, Blikkendaal MD, Dankelman J, Van Den Dobbelsteen JJ, Jansen FW (2014) Learning from visual force feedback in box trainers: tissue manipulation in laparoscopic surgery. Surg Endosc. https://doi.org/10.1007/s00464-014-3425-x
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Disclosures
A. Masie Rahimi, Sem F. Hardon, E. Willuth, F. Lang, Caelan M. Haney, Eleni A. Felinska, Karl-Friedrich Kowalewski, Beat P. Müller-Stich, Tim Horeman, F. Nickel and Freek Daams have no conflict of interest or financial ties to disclose.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Rahimi, A.M., Hardon, S.F., Willuth, E. et al. Force-based assessment of tissue handling skills in simulation training for robot-assisted surgery. Surg Endosc 37, 4414–4420 (2023). https://doi.org/10.1007/s00464-023-09905-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00464-023-09905-y