Background

Throughout the United States (U.S.), mass casualty incidents (MCIs) are increasing in both number and scope [1, 2]. MCIs may be natural or man-made, although incidents involving shootings or explosions are becoming increasingly common [2,3,4]. As MCIs have increased, there has been a reciprocal focus on training first responders for these types of events [4,5,6]. MCIs, particularly those involving shootings or explosions, require complex decision making since they involve: security threats, life-threatening injuries, challenging triage and transport decisions, coordination among response personnel and interoperability across diverse agencies [7]. Most MCI training efforts to date can be classified into three paradigms: 1) live, large-scale simulations of mass casualty incidents, cast in temporary settings, that involve the use of manikins and actors; [8,9,10,11] 2) tabletop drills that resemble board games; [12] and 3) lectures or presentations on mass casualty response, sometimes followed by facilitated discussions about treating specific patient injuries [13]. These types of simulation platforms provide first responders with methods for increasing knowledge and confidence but are unlikely to provide the sustained practice they need to be able to deliver successful triage and treatment of patients while under the duress of a mass casualty incident [14].

More recently, with the advent of higher-end gaming computers and computer interfaces, virtual reality (VR) systems provide alternative platforms for training individuals in skills required to respond to complex scenarios such as MCIs [15, 16]. Compared to traditional mass casualty incident training methods, VR systems can be programmed to portray the chaotic environment associated with MCIs [17]. Scenarios can be customized to be progressively more challenging through manipulation of noise, debris, and other distractions. Additionally, scenarios can be made more difficult through the addition of more patients who suffer higher degrees of injury acuity. These systems are portable, reusable and allow for continuous practice opportunities for first responders to gradually improve their skills in responding to MCIs [17].

When faced with an MCI, first responders are required to perform two complex tasks. First, they must effectively diagnose the entire scene, which involves sorting and triaging casualty victims into priority groups for transport to hospitals or trauma care centers. Second, they must be able to diagnose and treat those individual patients who are most at risk of losing their life or limb if they are not treated quickly. Both these skills involve clinical reasoning, which is best learned through repetitive, deliberate practice with performance feedback from an instructor [18].

One triage protocol that has been shown to be effective is the Sort, Assess, Lifesaving interventions, and Treatment and/or Transport (SALT) Protocol [19]. This protocol was proposed by a Center for Disease Control (CDC) sponsored working group to standardize triage methods across the U.S. The SALT Protocol is evidence-informed and endorsed by numerous professional organizations including: the American College of Emergency Physicians, American College of Surgeons Committee on Trauma, National Association of EMS Physicians, and National Disaster Life Support Education Consortium [19]. SALT has several advantages over other protocols. First, it can be applied to both pediatric and adult casualties. Second, it saves time by using voice commands to perform an initial global sort. Third, it eliminates the requirement of having to count respirations or check capillary refill, which are challenging tasks in chaotic environments. Finally, SALT allows for more rapid application of life saving interventions (LSIs) such as tourniquets [13].

The primary purpose of this project was to evaluate the use of a VR platform for training first responders in the skills required for disaster response, specifically skills related to SALT triage and treatment of an MCI. We were guided in our design of this VR simulation platform by the foundational elements of Brain-based Learning Theory, which includes the principle that learning complex clinical tasks is maximized when learners can experience immersion in realistic settings [20, 21]. Because little is known about whether learners would find the VR platform to be an acceptable alternative to traditional disaster training platforms, our objective was to gauge learners’ reactions to this fully immersive VR experience with triaging and treating patients in response to a virtual MCI. A secondary purpose was to determine whether prior experience using VR systems was associated with the participant’s perceptions about the quality of the VR training encounter.

Methods

Study design, setting and participants

This study was a prospective observational study of learner impressions of a fully automated, fully immersive VR system for training and assessing personnel to effectively respond to an MCI [17]. The system uses a commercially available laptop gaming computer and a Meta Quest 2 VR headset, (Reality Labs, Meta Platforms, Menlo Park, CA. https://www.meta.com/quest/products/quest-2/) and was designed for portability and practical use by agencies responsible for training first responders. We used the system as part of a short program to train learners in implementing the SALT Triage protocol [22].

Training was delivered to Emergency Medical Service (EMS) clinicians: paramedics and EMTs, from agencies throughout Central Ohio. EMS training sessions were held at the agency’s training facilities or in their service bays. We also held SALT Triage training sessions for future and current emergency medicine physicians: medical students, emergency medicine (EM) residents, and EM physicians. Physicians who are boarded in emergency medicine are commonly involved in the training of EMS professionals in disaster medicine skills. Medical learner training was conducted in medical school classrooms. Study participants completed consent forms and were screened for susceptibility to motion sickness (e.g. vertigo, dizziness), a potential side effect of VR. This study was approved by The Ohio State University Institutional Review Board (Columbus, OH, USA).

SALT triage training

The SALT Triage training was delivered in four parts. First, learners were introduced to the protocol through a brief lecture. Most learners already knew how to deliver life-saving treatments such as application of a tourniquet, so for those groups, we focused attention on the global sort technique and the definitions of each level of triage. More novice learners were also taught how to perform life-saving treatments. Second, each learner entered the VR simulator and participated in an orientation of the system. Third, the learners were provided the opportunity to perform SALT Triage on a simulation scenario that contained eleven patients, all of whom required triage categorization, and more than half required life-saving treatments. Finally, after the learner’s VR encounter, they received feedback through a debriefing session that was guided by a printed report of their performance. During debriefing, learners were offered the opportunity to explain their reasoning behind the decisions they made during the encounter.

MCI-VR simulator

The VR simulator scenario involved the recreation of a terrorist bombing of an underground subway station (Fig. 1). The system is fully described elsewhere but will be briefly summarized here [17]. Participants entered the MCI scene by donning a VR Head-Mounted Display (HMD) and hand-held controllers. While participants could physically walk around the virtual space, the controllers made navigation more efficient through the “teleport” (point to the spot they want to go and click) or “strafe/glide” (small movement) features, which moved them to where they wanted to go without walking. This means that the physical space for the encounter did not have to be the full size of the virtual subway platform, which if converted to actual physical space would be approximately 60 by 30 feet. For safety, participants were provided with a guardian (a research staff member) to ensure that they did not physically venture into physical obstacles.

Fig. 1
figure 1

Paramedic participant using the mass casualty incident-virtual reality system

Full size image

In VR, participants were equipped with a virtual medical kit. They used the controllers to grab tools and triage tags from the kit, or to take a pulse on a patient. Virtual patient injuries were those commonly suffered by individuals exposed to an explosion. Subsequently, learners could implement various lifesaving interventions (LSIs) including opening an airway of someone struggling to breathe, controlling major hemorrhage with a tourniquet or wound packing, or needle decompressing the chest of a victim who is suffering from a tension pneumothorax. Following completion of the encounter, the participants removed the headset and moved on to the debriefing session.

The MCI-VR System recorded and tabulated essentially every action the learner performed during the simulation encounter to produce a performance report. Faculty evaluators used this performance report to provide feedback to the learner through a post encounter debriefing on their performance.

Measures

Post-encounter surveys were designed by the researchers primarily to assess learner satisfaction with their VR experience. In addition to items that evaluated features of the system, we asked participants for an overall judgement of its training potential and whether the system encounter supported their learning needs. To gauge whether participants needed prior VR experience to benefit from the encounter, we used ownership of a VR system as a proxy for experience. We asked two open-ended questions about the system’s strengths (effective features of the system), and things we might do to improve it. Members of the VR development team used feedback from the second open-ended item to fine tune the MCI-VR system during Beta testing. The survey also included requests for personal characteristics such as: professional role, level of training, and prior triage experience (Appendix A).

Analysis

We described our participant characteristics using frequencies and percentages and compared professional groups (EMS vs. Medicine) using Chi Squares (Fisher’s Exact Tests). To assess whether participants’ evaluations of the MCI-VR system features were associated with prior VR experience, we compared those who owned a VR system, to those who did not, using independent t-tests. For items related to how well the MCI-VR encounter contributed to the participants’ learning, and opinions about the utility of the MCI-VR system, we compared EMS to medical professionals using independent t-tests. For statistically significant results, we calculated associated effect sizes: either Cohen’s D for mean statistics or Cramer’s V for Chi Squares. Statistical analyses were performed with IBM-SPSS, Version 29.0.1 (IBM Corp. Released 2022. IBM SPSS Statistics for Windows, Version 29.0.1 Armonk, NY: IBM Corp). To control rates of Type-1 error, we used the Bonferroni method to calculate family-wise levels of significance by dividing the desired P = 0.05 level by the number of comparisons within in each set (k/0.05) [23].

We used summative content analysis to code and reduce open-ended survey responses to themes [24,25,26]. First, one author (ARP) drafted a preliminary coding dictionary based on anticipated survey responses and common responses received in program evaluations. Second, responses to open-ended questions were entered into an Excel workbook (Excel, Version 2211, Redmond, WA, USA). Third, other authors (ARP, DPW, NEK) used word units (i.e., words, word stems, and words phrases with common semantic meanings) to assign codes to each response from the coding dictionary. Comments about strengths and suggestions for improving the VR system were coded separately. Coding assignments were performed iteratively as: 1) the coding dictionary was refined, 2) codes that shared similar underlying concepts were combined, and 3) codes that contained fewer than ten responses were eliminated. Excel continued to be used to process responses into codes until the dataset was effectively reduced to the strongest themes based on the number of times responses appeared.

Results

Between March 2022 and May 2023 we trained a total of 536 participants using our SALT Training Program which included encounters with the MCI-VR simulation system. Of the individuals trained, 83% (447 of 536) completed post-encounter surveys, however, we excluded 73 individuals who completed less than half of the survey (13.6%; 73 of 536). Subsequently, our final post-encounter survey response was 70% (375 of 536).

Participant characteristics

Eighty-six percent (86%, 324 of 375) of respondents were EMS professionals (Basic EMTs or Paramedics), and 61% (230 of 374) of respondents considered themselves to be seasoned first responders. One quarter of our subjects (25%, 92 of 374) indicated enthusiasm for computer games, while 17% (64 of 374) reported owning a VR system (Table 1).

Table 1 Participant characteristics. Frequencies and percentages of participant demographic characteristics for N = 324 EMS participants and N = 51 medical participants; with comparisons between groups made with Chi Square (Fisher’s Exact Tests) and Cramer’s V effect sizes (es)
Full size table

To gauge respondents prior triage training and experience, we asked them to identify specific courses they had taken, and the number of large-scale disaster drills in which they had participated. Seventy percent (70%, 262 of 374) of respondents had completed some form of triage training, while only 33% (124 of 374) had been trained specifically in SALT Triage. Fifty-five percent (55%, 206 of 374) had completed a formal disaster response training program; and of that group, the mean number of disaster drills in which they participated was 2.3 drills (Std. Dev. = 0.90) (Table 1).

EMS professionals differed significantly from medical learners regarding training and professional identity as a first responder. Significantly more EMS professionals than medical learners considered themselves to be “seasoned first responders” (P ≤ 0.001, es = 0.44). Participants who identified as seasoned first responders were also significantly more likely to have: completed some form of triage training other than SALT (P ≤ 0.001, es = 0.29); completed a formal disaster response training course (P ≤ 0.001, es = 0.29); and participated in significantly more large-scale disaster drills than those who did not think they were seasoned first responders (P ≤ 0.04, es = 0.22). The effect sizes associated with these differences are considered intermediate in size (Table 1).

Evaluation of VR features

Nearly all participants indicated that the VR orientation helped them to master the use of controllers to navigate the virtual space, (96%, 359 of 374) and subsequently felt sufficiently prepared to enter the MCI-VR subway station scenario (93%, 348 of 374). Almost all participants identified the MCI-VR simulator (95%, 354 of 374) and the patients within as realistic (91%, 340 of 374). They also reported that patients responded to their commands during the encounter (85%, 319 of 374). Participants agreed that the medical kit contained all the tools they needed to field treat patients (91%, 317 of 374), and that tools were easy to use (90%, 335 of 374). We observed no differences in evaluation responses from participants who own VR equipment and those who do not (Table 2).

Table 2 VR system evaluation. Means (Mn), standard deviations (SD), frequencies and percentages from participant evaluations of the VR system features comparing N = 64 Virtual Reality System Owners to N = 310 Non-Owners. Features were evaluated using a 5-point Likert Response Set: (1 = Strongly Disagree, 2 = Disagree, 3 = Disagree/Agree Equally (D/A =), 4 = Agree, 5 = Strongly Agree)
Full size table

Despite the large numbers of participants who found the orientation helpful, 18% percent (66 of 374) of subjects said that VR navigation (a skill they learn during orientation) was challenging. Furthermore, 26% (98 of 374) believed they needed more time to acclimate to the VR environment before engaging the MCI-VR subway station (Table 2).

Contribution to learning

While the instructional program designed for use with the MCI-VR system was not specifically intended to be a summative performance assessment, we asked participants about their views regarding the accuracy of the system generated score and whether their encounter performance was a valid assessment of their first responder skills. Both EMS and medical professionals responded similarly to these items, with about three-quarters of participants saying that the score was accurate (77%, 288 of 374) and that it was a valid assessment of their skills (74%, 276 of 374) (Table 3). However, EMS professionals rated the value of feedback and practice with the system significantly lower for making them more effective as a first responder (P ≤ 0.001, es = 0.55). The Cohen’s D effect sizes for these differences in mean ratings were considered intermediate in size.

Table 3 Contribution to learning. Means (Mn), standard deviations (SD), frequencies and percentages, along with p-values for independent t-tests (with Cohen’s d effect sizes (es)), that compared EMS (N = 323) to Medical (N = 51) professionals on how well the VR system contributed to their learning
Full size table

Future potential of MCI-VR for training first responders

Participant’s general response to the MCI-VR Simulation encounter was positive, with nearly all stating that they would recommend this experience to other first responders (96%, 360 of 374) and assigning it a letter grade of “A” for excellent or “B” for good. (93%, 349 of 374) (Table 4). Despite these positive ratings, only 74% (277 of 374) viewed the MCI-VR training as effective as live training.

Table 4 Perceived utility of the VR system for the future. Means (Mn), standard deviations (SD), frequencies and percentages along with independent t-test results (with Cohen’s d effect sizes (es)), that compared EMS (N = 323) to Medical (N = 51) professionals on the value or utility of the VR system for training in the future
Full size table

Summative content analysis

Three-quarters of participants provided comments about strengths of the MCI-VR System (75%, 279 of 347), while fewer than half (49%, 184 of 347) provided suggestions for improving the program.

Strengths of the MCI-VR system

Five key themes about strengths of the MCI-VR program were identified by the content analysis: realism, interactivity, and the opportunities to practice triage and treatment skills, decision making, and managing multiple patients (Table 5). The most common comment was about the realism of the simulation, some specifically about the realism of the virtual patients: “I thought the patient responses/panic even when they were “minimal” was a realistic portrayal of the emotional influence that patients have on providers despite the severity of injuries.” Several participants recognized the value of the MCI-VR system, including some who appreciated the opportunity to practice without having to risk harm to “real” patients: “Able to make mistakes without hurting real people.” A number of participants mentioned the interactivity of the MCI-VR system and the patients within as being key to learning and practicing triage and treatment skills: “The physical aspect of interacting with patients, treating and assigning them tags was really helpful,” “This was a good refresher and helped remember what equipment is needed.” Participants also remarked about the importance of experience with triage decision making. Specifically, the simulation pushed participants to “manage an overwhelming situation and critically think while staying focused.” Finally, several participants recognized that one key advantage of the VR simulation experience was the opportunity to manage multiple patients collectively instead of one at a time like in tabletop exercises: “I enjoyed having to manage multiple patients in rapid sequence. This gave me the ability to go from one patient to the next. Other trainings like this present one standardized patient at a time and those patients may not be trained to act or behave appropriately when injured in that manner.”

Table 5 Top themes and illustrative comments derived from the thematic analysis of participant comments about the STRENGTHS of the MCIVR simulation
Full size table

Suggestions for improving the MCI-VR system

Most suggestions for improvement had to do with modifying the system by either improving or adding features (Table 6). Many suggestions, particularly those involving patient interactivity, or injury visibility, were used by programmers to make improvements to the system over the period in which these evaluations were conducted. Some features requested by participants were already built into the system but were not yet made available during their one-time encounter. These include the ability to increase the difficulty of the scenario by increasing the levels of chaos: “Building on this to add more stressors or distractions would be great,” and “Make patient status dynamic during the exercise.” Other suggestions involved possible features for the future, such as adding tools to the medical kit (chest seals, trauma sheers, or a stethoscope), or building virtual sets that resemble environments within the participant’s jurisdiction. Some participants recognized the value in being able to have more than one participant in the MCI-VR environment at the same time: “Make it multiplayer so crews can work in concert.” A substantial number of participants experienced system glitches, such as the system freezing or the VR headset losing connectivity to the laptop computer on which the program runs. Glitches occurred regularly at the beginning of Beta testing but subsided over time. Comments about glitches also declined as problems were fixed. Finally, there were a few participants who felt the need for more time in the orientation to become more adept at using the controllers: “I needed a little more time familiarizing myself with the controls”.

Table 6 Top themes and illustrative comments derived from the thematic analysis of participant SUGGESTIONS FOR IMPROVING the MCI-VR System
Full size table

Discussion

Participants of the SALT Training with the MCI-VR System would recommend this experience to others and graded the experience as an “A” for excellent or “B” for good. The system was equally received as an effective training experience by both prehospital and hospital care providers, regardless of prior experience with triage or disaster training. Both groups credited the feedback from the activity for improving their first responder skills and believed that continued practice with the system would make them more effective first responders. While some participants commented on the need for more time to acclimate to the VR environment, we found no statistical differences in ratings of the system between those who owned VR systems and those who did not. Participants rated the system’s realism and interactivity highly, and reinforced these ratings with similar comments about how these were strengths of the system.

Three-quarters of our participants considered the MCI-VR training to be as effective as live, large-scale exercises, which may be more realistic, but are expensive, resource intense, requiring substantial planning and amounts of space. This finding suggests that VR for disaster response training is a viable option and could provide EMS agencies more flexibility in offering their first responders opportunities to learn and practice triage and treatment skills in realistic recreations of mass casualty incidents.

Virtual reality technology has gradually improved over the past ten years to become more immersive and realistic. Earlier versions of VR technology for training disaster response skills, such as Second Life, resembled computer versions of tabletop drills by presenting content on 2D computer screens. Participants interacted with these versions by controlling avatars through keyboards and mouse controls [29, 30]. VR became more immersive with the use of computer automated virtual environments (CAVEs), which used wall projection and liquid crystal glasses to create simulated disaster environments [31, 32]. CAVEs immersed subjects in the simulation rather than watching an avatar on a computer screen. The major drawback to these CAVE systems was that they employed expensive technology that limited their use to institutions with simulation centers and large simulation budgets [33].

Recent advances in VR technology now allow for the creation of realistic, highly immersive experiences at a much lower cost than these earlier systems. Our simulation used a MetaQuest 2 VR head-mounted display with controllers that allowed for individual tracking, tactile pulse checks, and highly immersive simulations. As such, VR simulations that were previously restricted to very expensive systems in large simulation centers can now be created at significantly less cost and be made accessible to a wider variety of organizations from university or medical school training programs to local fire, rescue, and EMS agencies. Moreover, assessments can be automated and built into the simulation, further increasing the utility of newer VR approaches.

Our VR simulation illustrates the practicality of this technology for disaster and mass casualty training. The MCI-VR system was well received by a variety of learners at different levels of training and did not rely on prior computer gaming experience, nor prior experience with disaster training or drills. Literature on optimal ways to prepare learners for VR environments was limited, so we designed our own brief orientation to help learners transition. The responses of our learners made clear that for most, a brief (approximately 10 min) guided tutorial and orientation was an effective way to prepare them for the actual simulation. This tutorial allowed the most novice VR learners to focus on the triage and treatment of patients without having to worry about the “knobology” of controlling the VR system.

One aspect of the training praised by learners was the value of the expert debriefing. Although the system generated a score report that could be used by the learner, we chose to provide formative feedback by reviewing these reports with participants in person. Our customizable system provided a structure that could leverage simulation-based mastery learning to gradually improve a learner’s mastery of triage and treatment skills over time [34]. Furthermore, this system provides a living laboratory for studying and evaluating triage protocol efficacy and the performance of learners from various backgrounds.

Limitations

This project examined participants’ evaluation of SALT triage training using the MCI-VR system, which is analogous to their satisfaction with the experience. Subsequently, we are not able to conclude that the training improved participant triage and treatment performance. Nor are we able to conclude that patient outcomes would improve because of improved first responder skills. Future research will be needed to attain these higher-level objectives. Although we screened for motion sickness prior to entry into the MCI-VR system, we did not ask participants about the impact of the system on them, either physiologically or psychologically, after the encounter [35]. While we assume from their positive evaluations that participants responded to the simulation similarly to a real-life incident, we were not able to verify this assumption.

Future research

Evaluation feedback has been used to improve the MCI-VR simulator including the expansion of the tools available in the medical kit and responsiveness of the virtual patients. Respondents have encouraged further development of the system, suggesting the inclusion of other disaster scenarios, and increasing the capacity of participants in the system for developing teamwork and communication. We are currently analyzing system generated data to investigate participant’s performance: such as time to hemorrhage control, and accuracy and efficiency in triaging and treating the scene. Further, we are planning subgroup analyses to compare the performances of first responders from varied training backgrounds (medical students, residents, paramedics and EMTs). Finally, we plan to leverage the VR system for studying human factors such as stress and their relationship to first responder performance.

Conclusions

Our VR simulator is an automated, customizable, fully immersive virtual reality system for training and assessing personnel in the proper response to a mass casualty incident. Participants perceived the encounter as realistic and effective for training, regardless of their prior level of training or their experience using virtual reality. The MCI-VR simulator was demonstrated to be a viable training alternative to current methods of training disaster response.