Abstract
Background
In the Netherlands, each year a three-day international multidisciplinary trauma masterclass is organized to provide the knowledge and skills needed to care for critically injured trauma patients. This study was designed to longitudinally evaluate the effect of the course on participant’s self-assessment of their own ability and confidence to perform general and specific skills.
Methods
Between 2013 and 2016, all participants were invited to complete a questionnaire before and during follow-up. Participants were asked to self-assess their level of confidence to perform general skills (communication, teamwork, leadership) and specific skills. Mean scores were calculated, and mixed models were used to evaluate correlation.
Results
We asked 265 participants to participate. Response rate was 64% for the pre-questionnaire, 63% for the post-questionnaire and for 3 months, 1 year and 2 years, respectively, 40%, 30%, 20%. The surgical group showed a statistically significant increase in self-assessed confidence for general skills (3.82–4.20) and specific technical skills (3.01–3.83; p < 0.001). In the anesthetic group, self-assessed confidence increased significantly in general skills (3.72–4.26) and specific technical skills (3.33–4.08; p < 0.001). For both groups statistical significance remained during follow-up.
Conclusions
This study demonstrated a sustained positive effect of a dedicated multidisciplinary trauma training curriculum on participant’s self-assessed confidence to perform both general and specific technical skills necessary for the care of injured patients. Given the known association between confidence and competence, these findings provide evidence that dedicated trauma training curricula can provide positive lasting results.
Level of evidence
This is a basic science paper and therefore does not require a level of evidence.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Background
Each year a masterclass on trauma care for surgeons or final-year residents in surgery under the auspices of the International Association for Trauma Surgery and Intensive Care (IATSIC) and the Dutch Trauma Society (NVT) is organized in the Netherlands. The course consists of a Definitive Surgical Trauma Care (DSTC™) course and a Definitive Anesthetic Trauma Care (DATC™) course. In the Netherlands, both DSTC™ and DATC™ are combined and integrated into a local multidisciplinair variation of the masterclass, in accordance with the core format of the DSTC™ and DATC™ curriculum (DSATC). This masterclass is based on well-established principals and objectives of trauma management described by the IATSIC and is intended to provide surgeons, anesthesiologists and scrub nurses with knowledge and skills needed to safely care for the critically injured trauma patient. Despite the fact that severe traumatic injury with an injury severity score >16 affects only 5% of the Dutch trauma patient population in contrast to 22% in the USA, participants must be skilled in all facets of trauma care [1, 2]. This course offers the possibility to practice cases that are rarely seen, such as a prehospital emergency thoracotomy (Netherlands n = 33 in 5 years) [3] (London n = 71 in 15 years) [4] (Japan n = 91 in 5 years) [5].
The three-day course consists of a theoretical part with key lectures, case presentations and case discussions and hands-on workshops with operative procedures on fresh human cadavers and a live porcine model. Apart from technical skills, crew resource management principles such as communication within the team, leadership and team work are practiced in the workshops. The live porcine operative laboratory includes all members of the team (surgeons, anesthetists, operative and anesthesia nurses, and a staffed laboratory), realistically reinforcing the concepts of crew resource management.
Information on short- and long-term learning effects of a definitive trauma course is scarce, and available data only regard the change in theoretical knowledge [6,7,8,9,10]. A change in technical skills, communication and team skills has not been documented most likely because these are more difficult to assess objectively. Smaller studies have already shown that a short trauma masterclass resulted in significant improvement in participants’ knowledge [11, 12]. However, because there is no standardization, it is difficult to draw a conclusion about effectiveness [13]. Therefore, we have started research on evaluating the DSTC™ and DATC™ with focus on self-assessment of learners’ confidence in their ability to perform the desired skills as a result of this trauma training curriculum. As part of this research, we investigated the change in and retention of technical skills and non-technical skills by means of a self-reporting questionnaire regarding skills to manage the critically injured trauma patient.
Methods
All data were collected from an online questionnaire (Appendix) that was sent to the participants (in masterclasses organized annually from 2013 to 2016) before the course started, directly after and at subsequent intervals of 3 months, 1, 2, 3, 4 and 5 years after the course. Each course consisted on average of 24 surgical participants, 24 anesthetic participants and 12 scrub nurses. Scrub nurses were also included because this group was intimately involved in performing the technical and non-technical skills and were also part of the dedicated trauma team. All participants were requested to anonymously complete the questionnaire. The follow-up questionnaire was sent by e-mail with one reminder 4 weeks later. The questionnaire consisted of six items regarding general skills (for instance; non-technical skills such as questions related to leadership and communication within the team, and general technical skills such as triage and treatment of mass casualties) and 22 DSTC™ or 18 DATC™ items, regarding specific technical skills. An example of a question regarding general skills was: “How competent do you feel with being the leader of the team?”. An example of a question regarding specific technical skills was: “Do you feel competent in handling a patient with a pelvic trauma?”. The items were derived from the course content and objectives. Each item was scored on a 1–5 Likert scale, with a Likert response of 1 indicating completely incompetent and a response of 5 indicating fully competent [14]. The following participant characteristics were collected for descriptive purposes: gender, training phase (resident or consultant), hospital setting (academic or peripheral hospital) and the number of years of experience as a consultant. The questionnaire remained identical during follow-up before (pre), at the end (post) and 3 months, 1 year and 2 years after the masterclass (follow-up).
We compared pre- and post-scores with post- and follow-up scores. The changes in these scores were compared between DSTC™ and DATC™ participants. The general and specific competencies were analyzed separately and compared between DSTC™ and DATC™ participants.
Statistics
All respondents were labeled with a random number which remained the same during the whole follow-up period. Data from each questionnaire were automatically collected and annually arranged in a data file.
For the comparison between the two groups, medians and interquartile ranges (IQR) of total mean Likert scale scores were calculated. Each questionnaire consisted of a general and specific part with the variables pooled for each subgroup. Total scores were analyzed per profession. A mixed model using factor time with random factor participants was used to deal with the correlation of the total score within a participant group. Comparisons between time points were adjusted for multiple testing using the method of Sidak [15]. The significance of each correlation coefficient from the mixed-model analyses as well as the random effect for predicting each outcome measure was assessed at p ≤ 0.05. All analyses were carried out using SPSS software (version 25, SPSS, Inc., Chicago, IL).
Results
Descriptive analysis
This study included questionnaire results obtained from participants in Dutch DSATC masterclasses conducted between 2013 and 2016. This made it possible to analyze four complete cohorts of participants with a two-year follow-up. The response rate of the 265 participants who were sent invitations to fill in the questionnaire is summarized in Table 1. A total of 214 individual participants were identified. A total of 101 were in the surgical group (47%), 61 were in the anesthetic group (29%) and 52 (24%) were scrub nurses. Fifty-nine percent (n = 126) of the participants were male. The descriptive statistics of the participants are summarized in Table 2.
Statistical analysis
Overall analysis contained the surgical, the anesthetic and the scrub-nurse subgroups (Fig. 1). This analysis was further divided into general skills (communication, teamwork) for both groups and leadership for the surgical group. In addition to general skills, specific skills were also analyzed. For a detailed overview of the results, see supplementary materials (Tables 1–5).
General skills
For surgical and anesthetic participants, the total mean scores for self-assessed general skills significantly increased and remained above the pre-course level during 2 years of follow-up (Fig. 1). For scrub nurses, there was an significant increase after 3 months.
Specific skills
For all the subgroups, the total mean scores for specific skills increased significantly (Fig. 2). For surgical and anesthetic participants, these scores remained statistically significant above the pre-course level at interval follow-up. For scrub nurses, significance was lost after 2 years. Specific skills in the anesthetic group contained more variables in comparison with the surgical group but also increased significantly. Sub-analysis showed an increase in self-assessed confidence on communication and teamwork skills. These scores also remained above the pre-course level, but showed a decrease after 3 months.
Sub-analysis
Additional analysis where performed for (1) course year (2013, 2014, 2015, 2016) and subdivided into residents and consultants, (2) experience level (resident, specialist with <10 years’ experience, specialist with >10 years’ experience), (3) gender and (4) type of hospital (academic, peripheral, other institutions).
Course year
No major differences were found between the course years. Residents showed statistical increase in general skills in the course year 2014 and for specific skills in all years. Consultants showed significant increase in general skills in 2013 and for specific skills in all 4 years.
Experience level
For both residents and consultants, self-assessed confidence in both general and specific skills raised significantly. The pre-course level of experienced participants was significantly higher in comparison with residents. A significant raise was found for specific skills, but lost significance after 1 year.
Gender
Gender was further divided in surgical general and specific skills, anesthetic general and specific skills and scrub nurses general and specific skills. There were no significant differences in baseline scores based on gender. Male participants in both surgical and anesthetic groups had significant increases in self-assessed confidence in both general and specific skills. Female surgical participants had significant improvement in specific skills over baseline, while female anesthetic providers had significant increases in both general and specific skills.
Type of hospital
Besides statistical significant increase in both general and specific skills, no differences were found between academic versus peripheral hospitals. For both groups, these scores remained significantly above the pre-course level at two-year follow-up.
Discussion
The overall self-assessed confidence in trauma care competency improved for both general and specific skills, as a result of attending the combined masterclass DSTC™ or DATC™. In the follow-up period, we found that (with exception of the scrub nurses) the improvements in self-assessed confidence over baseline were maintained for the duration of the study. When we extrapolate our data to, respectively, 3, 4 and 5 years, we see a downward trend in self-assessed confidence in performing general and specific skills for cases that are rarely seen in the Netherlands. In accordance with the Advanced Trauma Life Support® (ATLS®), we would therefore advise that the course should be repeated every 4 years. Increase in self-perceived competency is likely when participants have gained experience in their practice [16, 17]. Several participants indeed reported the use of knowledge and skills learned during the trauma course in actual trauma cases, which was confirmed by instructors working in the same institution. However, despite practical experience, we found participants with more than 10 years of experience still had a significant increase in their self-assessed skills over baseline, underlining the usefulness of this masterclass even for experienced consultants. The foregoing is possible the result of the fact that a part of the cases practiced is not often seen in the Netherlands. This in combination with extrapolation of our data makes it useful to timely repeat this course.
Studies on self-assessment have been performed in the past, but not on this scale [13, 18, 19]. In our study, the self-assessment of confidence in specific surgical skills increased more than in specific anesthetic skills, which may be attributed to the fact that the used anesthetic cases are more resembling common anesthetic practice. Other contributing might be the larger group of surgical participants and loss of follow-up after 3 months. In addition, improvements over baseline in the self-assessed confidence of surgical participants were maintained for the 2 years of the study. In contrast, anesthetic participants were noted to have a trend to decline in self-assessed confidence after 3 months of follow-up, but this decline was to a level still significantly above the pre-course baseline. We found that residents had a lower beginning level of self-assessed confidence when compared to consultants, likely explained by the result of work experience. We found no statistical significant differences for participants from academic versus peripheral hospitals or by gender. While the course was largely the same during the study period, attempts were made each year to improve the course based on participant and faculty feedback. Participants rated the course as very good during the years. In spite of minor adjustments in the course and changing some of the faculty year to year, there were no differences found in participants’ self-assessment based on the year of the course.
The increase in self-assessed competency we saw with the live porcine laboratory was an expected finding and has been reported by others [6, 8]. This can be attributed to the realistic nature of a live tissue laboratory and the ability to replicate the actual skills required in practice. The unique addition of other members of the team (anesthetist and operating room nurses) to our course may have contributed to the improvements seen in self-assessed crew resource management and team communication skills. Other simulation-based critical care courses, with emphasis on team performance, also had a positive effect on general and specific surgical competencies in surgical trainees [20].
Residents seem to benefit more from a masterclass than senior consultants and show a steeper learning curve [21, 22]. Similar results were found after a laparoscopic training course, in which trainees had comparable performance data at the end as surgeons [23]. The possible lack of variability and novelty in the anesthetic part of the exercises compared to the surgical part may also explain the difference in specific competence increase. Notably, reported technical and non-technical competences were rated relatively high before the masterclass in the anesthetic group.
Self-reporting is an important tool to uncover strengths and weaknesses and enable further development effectively. Physicians, however, the least experienced in particular, often fail to recognize their impairments and overrate knowledge and skills [18]. Concerning technical skills, some studies show inaccurate self-reports of physicians when comparing their self-reports to objective measurements (like for instance the Objective Structured Assessment of Technical Skills) [24, 25] while other studies show accurate self-reports [26] and an improvement in accuracy with increasing experience [18, 27, 28]. Concerning non-technical skills, surgeons proved not to be very accurate in their self-reports [18, 26]. Novices tend to overrate while seniors tend to underestimate their non-technical skills [26]. On the other hand, senior residents tend to an unwarranted confidence in skills which was not comparable with reality [29].
Our study aimed to minimize socially desirable answers by making the questionnaires anonymous particularly because most participants and instructors were acquainted. This approach was also used in a comparable study regarding advanced trauma operative management [30]. However, as the data were anonymous, data could only be viewed in terms of means (medians of total means) and no comparison at the individual level could be made. This also hampered the analysis of the three-month follow-up results, particularly because response rate had dropped and the study lacked information on participant’s experience with trauma cases after the course. Results from an ongoing German study, with less participants but similar outcomes are pending [19].
Strength and limitations
With 214 individual respondents, this study is the largest determining self-assessed confidence for skills as a result of a trauma course. Research started in 2013 and therefore only data from four course years are obtained. The faculty board had remained largely the same, the course format changed only minimally, which makes it possible to compare data from different years.
The study has few shortcomings. Our conclusion that self-assessed skills were maintained up to 2 years is based upon a response of 20% selection of all participants. It is not unlikely that there was a response bias in this group when responders represent those who reported sustained self-assessed confidence and those who did not respond. This limitation could be addressed by increasing the response rate, for example, by making the evaluation mandatory. Though al course participants were closely monitored by the faculty and discussed each day at the faculty meeting, it is not clear whether the self-assessed improvements in general technical and specific non-technical skills are the result of attending this trauma masterclass or may be a feature of ongoing clinical training and gained experience. To access this problem, it is needed to compare a group of non-attenders with attenders of the masterclass. Therefore, the study group is planning a future study with focus on expert-based observations at subsequent intervals. This in combination with the assessed self-confidence in performing technical and non-technical skills should demonstrate the added value of this masterclass.
The self-reported questionnaire was not validated as such. It was based on questionnaires used by Gaarder et al. [9] in a comparable Norwegian trauma course and adjusted to the content of both DSTC™ and DATC™.
The original goal was to achieve a five-year follow-up. However, the response for 3, 4 and 5 years after the course was so low that these data were not included in the analysis <11% (see Table 1) Possible explanation is that e-mail addresses are outdated or that participants have started working elsewhere. Another common idea is that participants simply are no longer able to adequately self-assess themselves. In a future follow-up study, these all can be prevented by having the faculty or an independent expert assess these technical and non-technical skills with validated scoring lists. This study only looked at technical and non-technical skills. The effect on participants’ self-assessed knowledge that is gathered by attending lectures, case discussions and practical sessions is not evaluated as such and is an important topic for further research.
This is the largest study to date to evaluate the long-term self-assessed confidence to perform skills following a trauma masterclass. We demonstrated that this three-day multidisciplinary trauma curriculum resulted in sustained self-assessed confidence and was of benefit to all participants, regardless of specialty, seniority or experience. A result of these findings increased emphasis on crew resource management was implemented in especially the practical sessions of this multidisciplinary masterclass. Dedicated training curricula such as the DSTC™ and DATC™ course contribute positively to self-assessed confidence to perform the skills taught. Studies such as this can assist in determining the retention of skills and help determine the interval between re-training. In order to retain advanced skills for cases that are not widely seen in the Netherlands, it is advised to timely refresh these skills. In accordance with ATLS®, this is advised after 4 years. The complex question of whether self-assessed confidence translates into improved outcomes could not be addressed in this study and is an important target of future study.
Abbreviations
- ATLS® :
-
Advanced Trauma Life Support
- DATC™:
-
Definitive Anesthetic Trauma Care
- DSATC:
-
Definitive Surgical and Anesthetic Trauma Care developed in the Netherlands
- DSTC™:
-
Definitive Surgical Trauma Care
- IATSIC:
-
International Association for Trauma Surgery and Intensive Care
- NVT:
-
Dutch Trauma Society
References
DiMaggio C, Ayoung-Chee P, Shinseki M, Wilson C, Marshall G, Lee DC et al (2016) Traumatic injury in the United States: in-patient epidemiology 2000–2011. Injury 47(7):1393–1403
Landelijk Netwerk Acute Zorg (2018) Traumazorg in beeld, landelijke traumaregistratie 2013–2017. https://www.lnaz.nl/cms/18.335_LNAZ_LTR_Rapportage-2013-2017.pdf
Van Vledder MG, Van Waes OJF, Kooij FO, Peters JH, Van Lieshout EMM, Verhofstad MHJ (2017) Out of hospital thoracotomy for cardiac arrest after penetrating thoracic trauma. Injury 48(9):1865–1869
Davies GE, Lockey DJ (2011) Thirteen survivors of prehospital thoracotomy for penetrating trauma: a prehospital physician-performed resuscitation procedure that can yield good results. J Trauma Acute Care Surg 70(5):E75–E78
Matsumoto H, Mashiko K, Hara Y, Kutsukata N, Sakamoto Y, Takei K et al (2009) Role of resuscitative emergency field thoracotomy in the Japanese helicopter emergency medical service system. Resuscitation 80(11):1270–1274
Jacobs LM, Burns KJ, Luk SS, Marshall WT III (2005) Follow-up survey of participants attending the Advanced Trauma Operative Management (ATOM) Course. J Trauma 58(6):1140–1143
Hansen KS, Uggen PE, Brattebo G, Wisborg T (2007) Training operating room teams in damage control surgery for trauma: a followup study of the Norwegian model. J Am Coll Surg 205(5):712–716
Jacobs LM, Burns KJ, Kaban JM, Gross RI, Cortes V, Brautigam RT et al (2003) Development and evaluation of the advanced trauma operative management course. J Trauma 55(3):471–479 discussion 9
Gaarder C, Naess PA, Buanes T, Pillgram-Larsen J (2005) Advanced surgical trauma care training with a live porcine model. Injury 36(6):718–724
Jacobs LM, Lorenzo C, Brautigam RT (2001) Definitive Surgical Trauma Care live porcine session: a technique for training in trauma surgery. Conn Med 65(5):265–268
Pringle K, Mackey JM, Modi P, Janeway H, Romero T, Meynard F et al (2015) A short trauma course for physicians in a resource-limited setting: Is low-cost simulation effective? Injury 46(9):1796–1800
Kuhls DA, Risucci DA, Bowyer MW, Luchette FA (2013) Advanced surgical skills for exposure in trauma: a new surgical skills cadaver course for surgery residents and fellows. J Trauma Acute Care Surg 74(2):664–670
Mackenzie CF, Tisherman SA, Shackelford S, Sevdalis N, Elster E, Bowyer MW (2019) Efficacy of trauma surgery technical skills training courses. J Surg Educ 76(3):832–843
Streiner DL, Norman GR, Cairney J (2015) Health measurement scales: a practical guide to their development and use, 5th edn. Oxford University Press. https://doi.org/10.1093/med/9780199685219.001.0001
Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat 6(2):65–70
Billings ME, Curtis JR, Engelberg RA (2009) Medicine residents’ self-perceived competence in end-of-life care. Acad Med J Assoc Am Med Coll 84(11):1533–1539
Shackelford S, Garofalo E, Shalin V, Pugh K, Chen H, Pasley J et al (2015) Development and validation of trauma surgical skills metrics: preliminary assessment of performance after training. J Trauma Acute Care Surg 79(1):105–110
Moorthy K, Munz Y, Adams S, Pandey V, Darzi A (2006) Self-assessment of performance among surgical trainees during simulated procedures in a simulated operating theater. Am J Surg 192(1):114–118
Back DA, Waldmann K, Hauer T, Huschitt N, Bowyer MW, Wesemann U et al (2017) Concept and evaluation of the German War Surgery Course—Einsatzchirurgie-Kurs der Bundeswehr. J R Army Med Corps 163(3):206–210
Doumouras AG, Keshet I, Nathens AB, Ahmed N, Hicks CM (2012) A crisis of faith? A review of simulation in teaching team-based, crisis management skills to surgical trainees. J Surg Educ 69(3):274–281
Langhan TS, Rigby IJ, Walker IW, Howes D, Donnon T, Lord JA (2009) Simulation-based training in critical resuscitation procedures improves residents’ competence. CJEM 11(6):535–539
Mackenzie CF, Bowyer MW, Henry S, Tisherman SA, Puche A, Chen H et al (2018) Cadaver-based trauma procedural skills training: skills retention 30 months after training among practicing surgeons in comparison to experts or more recently trained residents. J Am Coll Surg 227(2):270–279
Aggarwal R, Hance J, Undre S, Ratnasothy J, Moorthy K, Chang A et al (2006) Training junior operative residents in laparoscopic suturing skills is feasible and efficacious. Surgery 139(6):729–734
Pandey VA, Wolfe JH, Black SA, Cairols M, Liapis CD, Bergqvist D (2008) Self-assessment of technical skill in surgery: the need for expert feedback. Ann R Coll Surg Engl 90(4):286–290
Brewster LP, Risucci DA, Joehl RJ, Littooy FN, Temeck BK, Blair PG et al (2008) Comparison of resident self-assessments with trained faculty and standardized patient assessments of clinical and technical skills in a structured educational module. Am J Surg 195(1):1–4
Arora S, Miskovic D, Hull L, Moorthy K, Aggarwal R, Johannsson H et al (2011) Self vs expert assessment of technical and non-technical skills in high fidelity simulation. Am J Surg 202(4):500–506
Rosen L, Jacobson N, Weinberg A, Ascher-Walsh C (2019) Resident simulation training improves operative time of the retropubic midurethral sling procedure for stress incontinence. Int Urogynecol J 30(8):1359–1363. https://doi.org/10.1007/s00192-018-3744-x
Ward M, MacRae H, Schlachta C, Mamazza J, Poulin E, Reznick R et al (2003) Resident self-assessment of operative performance. Am J Surg 185(6):521–524
Bowyer MW, Shackelford SA, Garofalo E, Pugh K, Mackenzie CF (2015) Perception does not equal reality for resident vascular trauma skills. J Surg Res 198(2):280–288
Jacobs L, Burns K, Luk S, Hull S (2010) Advanced trauma operative management course: participant survey. World J Surg 34(1):164–168. https://doi.org/10.1007/s00268-009-0276-z
Funding
No funding was utilized for this work.
Author information
Authors and Affiliations
Contributions
All authors have been actively involved in the drafting and critical revision of the manuscript and contributed to the acquisition and analysis of data and writing of this article.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that there is no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Appendix: Questionnaire ‘Assessing a multitraumatized patient’
Appendix: Questionnaire ‘Assessing a multitraumatized patient’
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Tan, E.C.T.H., Rijnhout, T.W.H., Rensink, M. et al. Self-assessment of Skills by Surgeons and Anesthesiologists After a Trauma Surgery Masterclass. World J Surg 44, 124–133 (2020). https://doi.org/10.1007/s00268-019-05174-w
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00268-019-05174-w