Can Remote Virtual Simulation Improve Practice-Based Training? Presence and Performance in Incident Commander Education

Practice-based training in vocational education must be better supported for individuals to be prepared to work directly after completing the education (Clayton & Harris, 2018). Several organizations, particularly in health, architecture, product development, and emergency management, utilize virtual simulation (VS) for such training. According to a meta-review examining 2,582 papers, the effectiveness of virtual training is often comparable to the effectiveness of live simulation (LS) training but sets additional requirements on users and settings (Kaplan et al., 2020). Apart from a few examples from health and laboratory studies, the literature has not explored practice-based training in remote settings (Heradio et al., 2016; Vaughan, Dubey, Wainwright, & Middleton, 2016). Current VS training in remote settings is case-based, focusing on certain scenarios and discussions, manipulating documents, or watching and sharing pictures and videos. Although this training is valuable for some exercises, it may not be sufficient to prepare students for work requiring technical and non-technical competencies and the skills to manage emergency situations. There is a significant difference between discussing what should be done in an imagined emergency situation and performing in a simulated incident. During LS at a fire academy training ground, incident commander (IC) students must apply knowledge and skills while being situated in a believable, relevant context that resembles an incident in society. In numerous organizations, exercises in LS are considered “the gold standard.”

Virtual reality (VR) environments (incorporating computer-simulated objects, avatars, and interaction properties and involvement) (Lombard & Ditton, 1997) allow high presence in the virtual environment. Presence refers to the perceived sense of “being there,” the extent to which people “experience the virtual environments as more the presenting reality than the real world” around them (Slater, Usoh, & Steed, 1994, p.130). Presence often can be associated with improved performance (Slater & Wilbur, 1997) and greater learning goal achievement (Hoffmann, Meisen, & Jeschke, 2016; Young, Stehle, Walsh, & Tiri, 2020). The relation between the chosen VR technology, presence, and learning influences the experience and performance (Roberts, Heldal, Otto, & Wolff, 2006; Schroeder et al., 2001). More immersive technologies may contribute to higher presence, but they do not necessarily have positive impacts on learning (Makransky, Terkildsen, & Mayer, 2019). Non-immersive technologies may hinder smooth collaboration because of the limited workplace and “fragmentation” of the work area due to screen sizes and windows, among other aspects (Hindmarsh, Fraser, Heath, Benford, & Greenhalgh, 1998). Fragmentation is different in immersive VR, the experience can be still influenced by disturbances from technical devices; for instance, cables, 3D glasses, or a lack of understanding of others’ situations (Heldal et al., 2005; Slater, Brogni, & Steed, 2003).

While progressing from investigating technologies in laboratories to their real-life implementation, it is crucial to assess the expected benefits of the technologies. The process of choosing VR technologies and applications, establishing them, and using them for training practitioners has seldom been studied in practical settings. The fidelity in the simulation must be sufficient to achieve a presence comparable with experiences acquired in real environment (Pillai, Schmidt, & Richir, 2013). Visual photorealism, particularly in immersive technologies, is associated with higher costs; however, it may not be associated with higher training effectiveness (Stevens & Kincaid, 2015). The organizations using LS with real objects on training grounds or in simulation centers interested in VS and remote virtual simulation (RVS) may lack knowledge of which technologies and levels or aspects of photorealism are needed for effective training (Frøland, Heldal, Sjøholt, & Ersvær, 2020; Heldal, Fomin, & Wijkmark, 2018; Radianti, Majchrzak, Fromm, & Wohlgenannt, 2020). While advanced technologies may be used at dedicated training centers, remote simulations require accessible, affordable, intuitive, and reliable technologies (Di Natale, Repetto, Riva, & Villani, 2020).

During the COVID-19 pandemic, it has proven difficult to conduct practice-based training and assessment activities while maintaining social distancing. Consequently, it has become challenging to educate and assess the performance of Fire Safety ICs. Using innovative solutions such as VR has become increasingly relevant to bridge educational gaps (Hammar Wijkmark, Heldal, Fankvist, & Metallinou, 2020; Jnr, 2020; Yiasemidou, Tomlinson, Chetter, & Shenkar, 2021). Due to the pandemic, VR training is now most valuable when conducted remotely. However, introducing RVS for training can be difficult, particularly for safety-critical situations requiring practical competencies and skills.

This article investigates the introduction of RVS for practice-based training and final examination of ICs in Sweden. Data were collected during the practical implementation of RVS at the Swedish Civil Contingencies Agency (Myndigheten för Samhällsskydd och Beredskap, MSB), the organization responsible for educating firefighters and ICs. Practice-based IC training in Sweden is typically undertaken on the training ground in LS settings and has been supplemented by on-site VS since 2017. RVS-based training was considered a necessary step after VS due to the COVID-19 pandemic throughout 2020. The process of implementation was investigated through data collected from observations, questionnaires, and interviews with students and instructors, as well as the students’ performance assessed by instructors. The approach was guided via an action research strategy (Baskerville, 1999) focusing on three cycles: (1) a pilot study to examine the feasibility of using RVS for training and assessment in April 2020, (2) the implementation of RVS for the examination of one class in April and May 2020 (RVS1), and (3) an improved implementation of RVS for the examination of one class in November and December 2020 (RVS2).

The overall aim of this article is examining the role of RVS for practice-based training for experiencing training values. This will be achieved via answering the following research questions:

• RQ 1 How is presence experienced in RVS for practice-based training?

• RQ 2 How is the collaboration between students and instructors experienced in RVS?

• RQ 3 What are the most important aspects influencing presence in relation to the used technical settings?

• RQ 4 What are the main added values and limitations of RVS?

The results will provide a better understanding of the utilization of RVS and VS technologies to support practical training situations. It informs practitioners organizing educational modules of remote practice-based training and research on different training modes. According to our present knowledge this article provides an account of the first remote examination regarding practical training for IC qualification on an international basis.

This article is structured as follows: Section 2 describes basic concepts from VR and presence literature and their influence on understanding the practical training for ICs. Section 3 outlines the study design and used technologies. Section 4 presents the results and addresses the research questions. Section 5 discusses the findings and compares the results of RVS training and assessment with results from previous studies using LS and VS in the same context. The article ends with conclusions and suggestions for future research in Section 6.

This study was conducted in the Swedish context, in relation to resources, technical settings, educational structure, competency, and economic and organizational preconditions. The results of this study may not be transferrable to other settings, such as locations with lower bandwidth or different educational structures. An overview of the first steps (Hammar Wijkmark, Heldal, et al., 2020) and a discussion of the cognitive aspects of such examinations (Hammar Wijkmark, Metallinou, & Heldal, 2021) were presented earlier.

This section outlines the basic concepts and definitions used in this research and the educational context for acquiring IC qualification. It also includes relevant research from the VR research domain, with a focus on presence, performance, and learning related to emergency management education. How IC students and their instructors experience practice-based training in RVS and the current main training formats (LS- and VS-based training) is presented here.

As there are different definitions of the terms Virtual Reality (VR) environment, presence, and immersion in research, the definitions and basic concepts used in the present article must be specified. Virtual Simulation (VS) is in this work synonymous with VR, a three-dimensional model of the real world, abstract objects, or data where the user can control motion and orientation and interact according to specified rules. Numerous VR definitions distinguish presence from immersion, namely the experienced involvement from the technical properties needed to produce a surrounding experience (Schroeder, Heldal, & Tromp, 2006; Slater et al., 1994). Accordingly, the technologies examined in this article are non-immersive virtual environments used in a mixed reality setting (see Section 3.2). By practice-based training in remote locations, this article refers to activities that allow participants to communicate, share a virtual representation of environments, interact, and cooperate remotely. Cooperation refers to more than sharing pictures, videos, and other digital materials; it allows participants co-manipulate representations, share viewpoints, and interact, much like in computer games. This enables participants to gain skills and competencies in preparation for related real-life work situations, which is an important part of practice-based training. Today's computer games, including many serious games (games supporting learning or work, and not leisure) often use elements to better support surrounding, immersive experiences. However, immersiveness alone is not enough, serious games often lack pedagogical strategy and evidence of learning outcomes; as such, they are seldom fully integrated into education (Gorbanev et al., 2018; Yu, Gao, & Wang, 2021).

The ICs in the first (lowest) level of the command chain are often the first officers to arrive at the scene of an incident and are thus responsible for decisions regarding the first mitigating actions and, if necessary, requesting further resources. The IC is expected to assess the situation, gather and interpret information, and determine, communicate, and implement a plan to effectively and safely organize rescue resources and mitigate the consequences of an incident. They communicate with the Fire Service team, other command levels, other emergency management actors on the scene, bystanders, and the media. Firefighters who progress to become ICs may have several years of experience from hundreds of incidents (full-time personnel in urban areas) or very limited experience from real incidents (part-time personnel in rural areas). New ICs lack experience; thus, gaining some experience of acting as the commander in a simulated emergency is crucial.

The final examination has previously always been conducted in LS, namely requiring students and instructors to be at the same location.

In LS, scenarios unfold at a physical training ground using physical buildings, vehicles, people, fire, and smoke often built and/or arranged for firefighter training; for instance, to allow the extinguishing of fires and extrication from crashed vehicles. The student is physically present in the same geographical environment as the dedicated buildings, objects, equipment, other students, instructors, actors, and technical personnel. Interaction with others takes place face-to-face or via radio, much like a real incident. Although safety is the first priority, LS poses some risk for injuries and, when using real fire, exposure to carcinogenic particles (Wingfors, Nyholm, Magnusson, & Wijkmark, 2018). The development of a scenario in these settings is limited by safety and environmental regulations; for example, fires cannot spread to other buildings, smoke cannot represent plastics or rubber burning, and gas bottles cannot explode. The buildings are built to stand for several fires per year, that is, fireproof buildings that do not resemble any buildings in society or the fire behavior and cues of a real incident. The consequences of wrong decisions cannot be simulated in a way that represent the real-life situations. This limits the training value for the IC role due to missing more detailed cues. These elements must often be added to training (helping students to imagine these) and some aspects must be subtracted by verbal information given by the instructor. The instructor/assessor can observe IC students perform in a realistic incident context, but it is challenging and often impossible to provide LS training in sufficiently diverse situations to meet new and dynamic learning needs, such as the simulation of fires including new building materials, chemicals, or electric vehicles. Real, physical grounds cannot be used; for example, harbors cannot be closed to practice large ship fires (Jansen, 2014). Complex incidents, large fire explosions involving many firefighters (Chittaro & Sioni, 2015) cannot be easily set up and repeated hundreds of times to allow for the training of many in the same manner (Lamb, Farrow, Olymbios, Launder, & Greatbatch, 2020). Notwithstanding these shortcomings, the prevailing view among instructors is that LS involving physical objects and interaction is the only adequate method to train and assess practical competencies necessary for ICs.

While VS technology was purchased by MSB in 2011, it was not used before 2017 (Heldal et al., 2018). There are many possible explanations regarding this delay, for example, lack of experience, digital incompetence, medium-level managerial support, and the existing, successful, LS training. Other studies have also investigated the slow implementation processes for using VS tools in emergency management education and identified necessary performance requirements. Examples are the requirements for high fidelity representations (Williams-Bell, Murphy, Kapralos, Hogue, & Weckman, 2015), the need for additional competencies for instructors (Alklind Taylor, 2014), and concerns regarding students’ learning incorrect or incomplete actions (Frank, 2014) or more accessible training situations (Backlund, Heldal, Engstrøm, Johannesson, & Lebram, 2013).

RVS is not only an option for distance training and assessment during COVID-19-related restrictions but also of great interest to organizations providing distance education where the student and instructors can participate from remote geographical locations. Remote collaboration in VS requires non-problematic, quick social interaction via technology and an ability to perceive, define, and approach common goals. Students and teachers must collaborate in an educational context to clarify goals, methodologies, and roles. However, in remote interaction, the rich communication of social cues, such as non-verbal communication, can be curtailed, thus impacting interaction since some cues are lost. When using RVS, it must be accepted that some social cues are filtered out by the medium (Heldal, 2007), and that technical limitations hinder the transmission of some interactions (Frank, 2014; McMahan, Bowman, Zielinski, & Brady, 2012). A given member of the group does not necessarily know at the start the goals of others or how they solve problems. Following how the participants view the environment, solve a problem (with instructors also playing their role), and perceive the technology (windows, devices, and what others notice in the virtual environment) can also be difficult (Bowman, Johnson, & Hodges, 2001; Heldal et al., 2005; Hindmarsh et al., 1998). The experienced success of performing tasks depends on the complexity of the tasks, and also on personal characteristics and technical competencies.

Before the pandemic, in March 2020, there were no plans to use VS remotely or implement it in any MSB final examinations. The authors of this article planned to follow a final LS examination of one IC class and perform a feasibility study for a corresponding RVS class. After ceasing all on-site training (LS and VS) and being urged by the need to examine one class of IC students, MSB decided to test RVS for IC final examinations.

Before VS became a technology mature enough to adopt, LS was the only realistic practice-based training available where the student could perform in an incident context. The hesitation about adopting VS has been focused on the training transfer to real-life settings, although as we argued earlier, the effectiveness of LS exercises can also be questioned. The transfer of knowledge from LS/VS/RVS to real life incidents is complex and therefore can be impossible to measure due to the time lag between training and knowledge implementation (often several years) and the dynamic, unpredictable nature of real incidents.

In Experiential Learning Theory, learning occurs in four steps: experiencing, reflecting, conceptualizing, and acting/actively experimenting (Kolb, 1984). From the IC students' perspective, the experience would include “taking on the role as the IC” in expected and unexpected incident scenarios. The students must convince instructors that they perceived the situations correctly and had the skills and competencies to handle them. They must demonstrate realistic responses and make sound decisions. Reflecting upon the experiences in the scenario is often done verbally, including feedback from the instructor. This leads to conceptualization, which triggers new or improved ways of acting. By this, the IC-student realizes that “knowledge is created through the transformation of experience” (Kolb, 1984, p. 90). Accordingly, “sense-luscious” authentic experiences that flood the senses are the best for learning, as argued by Zull (2002), implying that not only the surroundings but also the perception of being present in the situation are essential for learning (Han, 2020).

Presence refers to the user's ability to focus on the virtual representations and actions, the experience of “being there” in the simulation, rather than on the surrounding physical environment (Slater et al., 1994). In numerous studies, a common assumption is that experiencing a high degree of presence in a VS can result in improved performance (e.g., Abich, Parker, Murphy, & Eudy, 2021; Monteiro, Melo, Valente, Vasconcelos-Raposo, & Bessa, 2020; Schroeder et al., 2001). Although the literature is not conclusive on whether there is a causal relationship between presence and positive training transfer (to real-life performance), it can be posited that a sufficient level of fidelity, namely a degree of realism, is required for effective training (Salas, Bowers, & Rhodenizer, 1998, Stevens and Kincaid, 2015). Additionally, it is not easy to determine what degree and aspect of realism are most important in VS to achieve presence and be able to transfer learning from training to work settings. For example, at first glance, instructors may believe that visual photorealism and physical human interaction are necessary for high experiences and training transfer. In fact, the training transfer depends more on how the simulation is used (Heldal, 2018; Makransky et al., 2019; Salas et al., 1998) and not necessarily only on the type of technologies applied. How much financial investment an organization can allocate to VS and RVS in general and its influence costs and learning outcomes would require further research.

It is also important to identify what aspects of the virtual environment or technical interaction reduce presence, such as when the experienced presence is disturbed by strange representations, clumsy devices, or unintuitive interactions (Slater et al., 2003). The psychological fidelity, positive or negative stress, and arousal associated with a real-life incident can be difficult to replicate in VS, RVS, and LS training. Stress is part of the natural emergency response context and is also present, although to a lesser extent in LS, VS, and RVS training where the IC students should demonstrate and apply their command skills. The experience of stress is believed to enhance skill retention and the transfer of training from the simulated experience to the real world (Mayer & Volanth, 1985; Williams, 1980).

The present study involves a researcher actively engaged in implementing virtual technologies and improving education at MSB, thus following an action research approach (Baskerville, 1999). One researcher followed the whole implementation process of VS and RVS, while the other two researchers followed some parts of the process.

MSB, the agency responsible for IC education, ceased all on-site training (LS and VS) due to COVID-19 restrictions on gatherings of people. The need to graduate one class of IC students enabled the rapid adaptation of RVS. With this decision, the researchers organized the study in the following three cycles (also, see Figure 3):

• – (Cycle 1) a pilot study (to test the RVS-format and demonstrate usability to MSB management). The pilot study included eight experienced ICs from different fire and rescue services, testing “RVS-examination” from the premises of the fire station they worked at. Two experienced instructors/assessors arranged the simulation. This resulted in designing/calibrating the practice-based training settings by running five scenarios.

• – (Cycle 2) implementing an RVS examination for one class (20 IC students) with four instructors (RVS1, May 2020), running five scenarios per student.

• – (Cycle 3) implementing an RVS examination for a second class (23 IC students), with five instructors (RVS2, December 2020), running three scenarios per student.

Based on the experiences of the first RVS examination (May 2020), the scenarios were adjusted, and their number was reduced from five to three.

Data from the evaluation of the three abovementioned cycles aim to improve practice-based training at MSB and provide insights into the process of implementing virtual technologies in education. For each cycle, the steps of diagnosis, action planning, implementation, evaluation, and learning were followed.

In the pilot study, simulations lasted for one hour for each participant (experienced ICs acting as students). Performing the final examination in RVS lasted for four hours for each IC student. The RVS1 and RVS2 evaluations were based on observations of students and instructors, questionnaires answered by students, and interviews with instructors. Additionally, the assessment of students, performed by the instructors, provided information regarding achieved learning goals. The first author observed the instructors during sessions and interviewed them after the exam had finished. Each student completed one pre-exam and one post-exam questionnaire. The instructors graded each student (1–5, for the benefit of the research project, where 1 was ``no pass'') and provided information on the students’ performance. The students received a ``pass/no pass'' result.

For investigating presence, we applied the presence questionnaire (Slater, Sadagic, Usoh, & Schroeder, 2000; later modified by Schroeder et al., 2001) and added specific questions on the experience of the simulated environment and objects, problem-solving via tasks, and social communication and cooperation (Heldal, 2007). These three aspects were also formulated by Hontvedt and Øvergård (2020) for investigating simulation fidelity focusing on observations and questions.

The students were provided with information about the study and then decided whether or not to participate. The term real-like, used in the questions, was explained in the introduction to the questionnaires as relating to the resemblance of (R)VS training to experiences acquired in the physical realm, through LS training or the handling of real incidents. Each participant completed a consent form for participation, a pre-exam background questionnaire about individual interests and patterns for using technology, and a post-exam extended presence questionnaire. The instructors were the same individuals for both RVS1 and RVS2 (four instructors), with the addition of a fifth instructor in RVS2. All instructors were interviewed. Additionally, data were collected through participatory observation. Observations regarding student-participants’ and instructors’ activities or answers to questions were labeled according to their class (RVS1 or RVS2) and identification number: RVSn-pX/iX. For example, participant 3 in the second class was marked as RVS2-p3, and the second instructor working with the first IC class as RVS1-i2.

Five scenarios (A, B, C, D, and E) were designed and built in a 3D simulation tool (https://www.xvrsim.com/en/). This tool includes several prepared virtual environments (e.g., countryside, city) in which dynamic scenarios could be built using objects (e.g., vehicles, avatars, fire, and smoke) from a library to prepare events, triggers, or other functions. The scenarios were designed based on the course objectives (briefly described in Figure 1) and had corresponding levels of difficulty as defined by instructors for previous LS examinations. They included authored storylines mirroring actual, real-life incidents of similar dynamic events and involved civilians as a typical incident may have.

The five scenarios in RVS1 were:

• A.

  A road traffic collision. A farmer driving a pick-up has lightly collided with a truck while avoiding a collision with a deer. He is transporting a tank of herbicide that cannot be found in the decision support tool. This tank has a leakage caused by the incident.

• B.

  A car fire, threatening to spread to a building close by. The family is safe outside.

• C.

  A third-floor apartment is on fire. Initially, it is unknown if anyone is inside. After a while, a friend of the apartment's owner approaches the IC and explains that a cat is in the apartment and the owner is abroad.

• D.

  A collision including three vehicles has occurred under an overpass caused by loose timber on the road.

• E.

  Some young people have broken into a warehouse and started two fires before leaving. There are caravans, vehicles, and welding gas inside the warehouse.

These scenarios were used to provide relevant, realistic incident contexts in which the students could perform, that is, to adopt the IC role and convince instructors of their skills and competency. The scenarios provided the context for performing as the IC (Figure 1); in other words, the scenarios were not designed to assess specific technical skills and tactics in a specific situation. Skills and competencies were assessed and documented using an assessment tool formed according to the course objectives (Figure 1).

Four instructors were present during the RVS examinations. They alternately assessed the students but since they were also performing counterplay, this provided the possibility to discuss evaluations in a way that had not been possible in LS, where only one assessor is appointed. The instructors expressed that their ability to perform high quality assessment was enhanced.

After completing the scenarios and receiving feedback, some of the students spontaneously described their experienced presence: “This is more realistic than other methods for exercises” (RVS1-p14); “this was great, it works great remotely” (RVS1-p3); and “this is beyond my expectation. Interesting scenarios, the environment you built, giving orders works great [the firefighter avatars carry out the orders], and it feels like you are at the incident scene. This is the best substitute for being on site” (RVS1-p5). Many of the participants had not had high expectations that the experiences would be believable. Several ICs expressed surprise in their comments: “Overall, a great surprise. You do not have to pretend; all you see is what it is. Not like in the training ground” (RVS1-p16); “I think this worked well … there must be more of this in the course, especially remotely. It was gold [probably: great], as close to real as it can get. And I did not have to drive 2000 km to the college [for the examination]” (RVS1-p1); “This software is great. For sure I cannot blame the software for my mistakes” (RVS2-p7); and “this scenario could not have been done in the training ground” (RVS2-p5). A total of 90% of the students in RVS1 and 100% of students in RVS2 stated that they would like to participate in similar RVS training again. A total of 10% (2 students in RVS1) responded “neutral” and explained, “I had a hard time interpreting a realistic picture of all impressions” (RVS1-p4) and “I was not comfortable in the situation. It is a good supplement, but I would have needed more real training [in LS before]. The scenarios were good, and I would have liked to train more without the pressure of examination” (RVS1-p9). Their answers illustrate possible insecurity regarding being situated in the RVS and adopting the IC role (observations from RVS1).

We investigated whether experienced presence relates to performance, and whether experience from incident scenes as a firefighter would lead to higher presence and/or improved performance in RVS. Presence was investigated by the question: “Think of a previous LS session in which you experienced high presence. Compared to that, to what extent did you experience high presence in RVS?”; spatial presence by: “To what extent did you feel that you were in the virtual environment?”; co-presence by: “To what extent did you feel that you were in the same environment as others you met?”; and perceived easiness to resolve the task by: “How easy was it to solve the task, that is, to command the incident as you intended?” The performance measure is the average of summative grades (one for each performed scenario) given by the assessor (on a scale from 1 to 5, where 1 = fail, 2 = pass lowest level, 3 = pass medium level, 4 = pass with high level, and 5 = pass with excellence) for the benefit of the research project. The results are presented in Table 1. Although no strong correlation is observed, there may be a lower co-presence and performance for students with less experience of real-fire incidents as firefighters.

As Figure 4 illustrates, in the RVS setting, the IC students focused on the screen, seeing only the avatars of counterplayers, and hearing real human voices (the instructors) through their headset and/or digital radio. The students talked directly to avatars on the screen using the headset, and also communicated via radio, as firefighters often do during incidents. No additional technology was required, that is, no buttons on the gamepad or screen. No other people were present in the room with the student. According to the observations, the IC students focused on the screen during the whole session, with only some interruptions during the first scenario due to students being unaccustomed to moving their avatars in the environment using the key arrows.

In VS on-site (see Figure 2), (which MSB had used since 2017) the student focused on the screen while moving in and observing the virtual environment; however, when approaching or approached by the avatars of firefighters or bystanders, the instructor physically approached the student while performing counterplay. This drew the student's attention to the physical room and the person in it (the instructor acting as the avatar). Students had provided positive feedback on this aspect of the roleplay when asked about experiencing “Complementing the avatars on the screen with face-to-face human roleplay” (Hammar Wijkmark & Heldal, 2020). In transitioning from VS to RVS, a vital question in MSB concerned the potential consequences of the loss of visual, face-to-face interaction. However, the participants in the present study (RVS during 2020) did not report a need for this interaction. The instructors appreciated the ability to see the IC student's face (front view of) compared with LS (when this was possible only during face-to-face roleplay). Seeing the face in combination with a good audio connection and including the option to follow the student's view allowed instructors to further understand the student's perception (e.g., what cues s/he noticed and reacted to). The instructors orally communicated via headsets and radio. They could act in their appointed avatar roles, moving their avatar using the gamepad and speaking according to the role of the corresponding avatar. They could select and change which avatar they operated within the storyline and how they acted. They could also adjust their actions, if necessary, when responding to the student's commands. This required multitasking and collaboration between instructors to achieve realistic timing and dynamically change the pace and development of the incident. According to Hindmarsh, collaboration in this setting could encounter several fragmentation problems (disruptions in interaction) for actors using the virtual spaces and objects (Hindmarsh et al., 1998) and address impediments to social interaction (Heldal et al., 2005). There are several hypotheses regarding the negative effects of fragmentation of applied technologies were not experienced in this study. Some instructors suggested that the constant, high-quality audio connection and access to the student's view made it possible to make predictions and be more prepared. This allowed a good orchestration of counterplay and collaboration between the instructors. Indeed, orchestrating practice-based training and maintaining a well-prepared storyline in RVS are essential and undoubtedly influence the overall experience.

As presented by Hammar Wijkmark, Metallinou et al. (2021), all instructors conducting the RVS examination were convinced that the students were presented with similar challenges to those presented in LS examination and performed as they would in LS settings. They also perceived the students’ movement in the virtual environment and their communication with the avatars to be easy and unproblematic, which is consistent with the students' comments. The instructors stated that they were able to assess the students based on the established criteria for achieved learning objectives. One instructor explained the values of the virtual environment as “Everything that relates to the situation awareness, the development of the incident, like the spread of the fire and the extent of the damage, is possible to include in the virtual environment, which makes it extremely effective for assessment” (RVS1-i1).

It was observed that the screen showing the student's face was not widely used by the instructors, while the screen showing what the student was looking at (student's avatar view) was in use most of the time. Although the instructors considered both views as beneficial, they could extract useful information from knowing what the student focused on as to whether the student was consciously “reading” cues. Access to both views cannot be achieved in LS where the instructor cannot be certain about what the student is observing. Another instructor explained: “I see and hear the student all the time. I can more easily assess communication and the orders given. I can see the exact picture of what the student is looking at each moment. It can sometimes be difficult to determine what the student is focusing on in⁹ a live exercise in the field” (RVS1-i2).

The instructors played different roles through avatars. One instructor stated: “To have the opportunity to play the IC student's team [firefighters] makes it possible to ask questions if orders are unclear. Also, later during the scenario, one can [with the firefighter avatar] walk up to the IC and ask a related question to assess to what extent he or she has an understanding of the situation at hand” (RVS1-i1). The difference this makes for assessment, compared with LS training, was highly appreciated by the instructors. In LS, a firefighter team consists of firefighter students who, having participated in previous exercises in the same buildings, automatically perform actions the way they have learned to do, without seeking clarification if an order is unclear.

One instructor with extensive experience of both LS and VS who participated in all RVS scenarios highlighted the importance of the introduction (the welcoming of the IC student) to the RVS session. If welcomes were not handled well, it might affect the student's state of mind and negatively prime the whole experience (c.f. Heldal et al., 2005). Upon starting the session, the instructor video-called the student, presented the other instructors, showed a view of the instructor “control room,” and asked some informal questions about, for example, the fire and rescue service the student is allocated to, thus reducing exam nervousness and “technical fear” of the RVS format.

Training is necessary for practicing interaction in the same way as the ICs do during real incidents. Organizing practice-based training includes several stakeholders involved in emergency management education, objects, instruments, and technology, i.e. buildings, firetrucks, radios, etc. (Hammar Wijkmark, Heldal, et al., 2020). The following section investigates the role of the applied graphical representations for practice-based training regarding the context with essential objects for the incidents, the possibility of solving problems through tasks, and interaction with participants (Heldal, 2007).

Regarding the students’ performance self-assessment, 70% of participants in RVS1 and 74% in RVS2 responded “yes” to the question: “Do you think that you managed the task as well as you would have managed the corresponding real task?” Students commented, “It feels like this was quite similar to reality” (RVS1-p13), “believable incidents” (RVS2-p6), and “I do not think that the mistakes I made would have been less in a real setting” (RVS2-p11). The students who believed they would have performed better in a corresponding real incident explained: “In a real incident I would have seen the details in the incident better and easier and I would have seen where the people are and also seen the events and damages easier” (RVS1-p17); “Real world and this are two different things. This is good as complementary but does not replace reality” (RVS2-p13); “I was extra nervous because this was an examination” (RVS2-p3). Overall, many IC students appreciated to not need driving around 2000 km for the examination (e.g., RVS1-p1) and the possibility (e.g., RVS2-p20, RVS2-p5) to be able to use a scenario that would be impossible on the training ground.

The IC student and all others at the scene met in the virtual environment, which does not support hand gestures or pointing to explain directions. Regarding whether this was experienced as a hindrance, 40% stated “yes” and one student explained, “I am used to making gestures and pointing, that was not possible” (RVS2-p9). It was observed in RVS2 that verbal descriptions quickly compensated this; for instance: “look at the window on the second floor,” “he is over there on the right-hand side of the building.”

To the question: “Please describe aspects that you found pleasant in the task,” 50% reported that they appreciated the good counterplay; the voice acting done by instructors enhanced the realism of the situations.

The most appreciated added value of RVS was that the IC final exams could be conducted despite COVID-19 restrictions on gatherings of people. RVS in this study supported student performance and assessment at the same level or higher than VS on-site and LS. Since distance education is a high priority for MSB, RVS will be used to increase training volume remotely for several learning objectives. Training regarding interpersonal interaction within the team, namely leadership, including non-verbal communication, may be better when involving real people.

RVS is conducted in a medium that does not involve all senses. Sight and hearing are the most critical paths to gather information. In the physical settings of LS, additional senses would be activated, such as smell and touch. Although this is limited in LS, the effect of lacking this in RVS should be investigated further. Additionally, gestures cannot be used in RVS and must be compensated for by verbal communication.

Graduating from MSB IC courses has previously required a final examination in LS. Although this involves three days at the training ground, one week is scheduled for additional lectures. This involves costs to travel and time away from students’ ordinary jobs. For these two IC classes, a traveling distance of 37,354 km and its corresponding CO₂ impact were evaded. This corresponds to nearly 24 times the length of Sweden or 93% of a complete lap of Earth. The most common method of travel is by car. A calculation using the average mileage traveled by all IC final year students (280) estimates 242,335 km of traveling, which is equal to six laps around the globe. This traveling can be avoided by applying RVS.

The cost of one LS scenario used for an IC course was calculated in a previous study, accounting for all resources and materials required. The cost of one one-hour LS scenario is approximately 300 EUR (350 USD) (Heldal, 2018). Each student conducts two scenarios, with a total cost of roughly 600 EUR (700 USD), while being physically at the training ground for two days, since the activities also include observation of other students' performance. The cost for the RVS final examination, including all resources and materials, was calculated at 875 EUR (998 USD) per student, where the student participated from home for four hours. This cost divided by five scenarios (RVS1) gives a cost of 175 EUR (205 USD) per scenario and divided by three scenarios (RVS2) gives a cost of 290 EUR (340 USD) per scenario. The cost for two hours would be 430 EUR (or 505 USD, calculated in SEK and converted). When calculating and comparing the actual cost of a two-hour practice-based training session for one student, the RVS shows 28% less cost than LS.

Health is also a major concern regarding LS. In a previous study, it was shown that the levels of eight carcinogenic polycyclic aromatic hydrocarbons (PAH) metabolites were five to eight times higher in urine samples collected from students 20 hours after a regular smoke diving exercise compared to samples collected before the exercise (Wingfors et al., 2018). It was suggested that dermal exposure was the major route of exposure for firefighters. However, IC students and other individuals present at an LS training including real fire who are not wearing respiratory protective equipment expose themselves to a risk of inhalation exposure. This risk is not present in VS and RVS.

The action research method enabling this study involved one researcher with experience of emergency management training and a deep understanding of the training objectives and preconditions of the organization and its stakeholders, who was actively involved in the implementation process for about 10 years and in close collaboration with the instructors. Studying the implementation of the technology in context required extensive knowledge of the stakeholders and the organization's current needs.

The methods used for studying the instructors’ experiences of RVS were interviews and participant observation. The instructors highlighted in the interviews that it was of great value for them to be able to observe what (in the virtual environment) the student (through an avatar) was observing. The observing researcher also confirmed that the instructors were continuously using the screen showing the student's view. Therefore, we believe that this result is trustworthy and that this finding could inform future examinations performed after the pandemic. Instructors having experienced this advantage may be motivated to perform future exams in the RVS format to ensure their access to the information inherent in sharing the students' avatars view.

The experiences of the students were studied through questionnaires and observations. Additionally, for the benefit of the research project, the instructors made detailed notes and performed graded evaluations to provide further information about the students’ performances. This information, presented in Section 4.1.2, indicates that previous experiences of incidents as a firefighter may affect the level of performance and the experienced presence in RVS. The participating instructors, who had extensive instructor experience, were convinced that the performance was not considerably affected by the RVS format; in other words, the students’ performances corresponded to their anticipated performance in LS.

Earlier observations made between 2017 and 2019, indicated that students appreciated a form for “live counterplay,” with instructors physically approaching the student during role-play during VS on-site (see Section 2.1.2 or Hammar Wijkmark & Heldal, 2020). RVS, lacking this aspect, did cause some concern at MSB before its implementation. However, the results from RVS (acquired in the present study, 2020) indicate a higher presence in the simulated environment compared with VS on site. It would be interesting to explore whether physical on-site counterplay reduces the sense of presence in a virtual environment by splitting students’ attention between the screen and the person physically in the room.

The two reported classes participated in RVS training at different times during the pandemic. The pandemic began in spring 2020 when many people were less accustomed to digital meetings. During the months that followed, many have had numerous digital meetings and cooperated remotely. This may have influenced how the groups perceived and performed in the virtual environment. Certainly, the team of instructors, which was the same as the previous team with the addition of one new instructor who joined in the fall of 2020, had also gained experience both using the technology (handling the fragmentation of different screens and understanding the collaborative setting) and handling the counterplay, orchestration, and collaboration. Those improvements may have positively affected the students’ presence and may also explain the increased presence reported by students in RVS2 compared with RVS1. It would be interesting to explore the value of counterplay orchestration as an extension and connection to the scenario design, namely the importance of not only designing and building virtual scenarios but how to perform the counterplay to enhance presence and learning.

It takes time to develop the knowledge and skills to design, build, and run RVS training and assessment at this level. In this case, the competency was present in the organization due to the recent experiences of implementing VS (since 2017) and motivation to implement RVS. If organizations do not perceive the effort necessary for building and retaining competence in scenario design and play, and thus allocate insufficient resources to instructors, the quality of simulations and scenarios may be negatively affected. Training professional RVS instructors is time-consuming, however, this is a prerequisite for high-quality training experiences and acceptance of RVS. If RVS is viewed as ``just another tool in the toolbox,'' comparable with LS and used as LS with the same planning, objectives, design of learning activity, and assessment method, the added values of RVS (and VS) may be undermined.

A key question behind this study is: Is the presence and performance experienced in RVS adequate to train for the IC role? If the answer is yes (which the results of this study support), aspects that improve or hinder presence and performance must be further investigated. According to the definition of presence we used (Slater et al., 1994) the presence in a LS, a physical place should be 100%, but it is not. As we have shown, though LS provides 100% spatial presence, since it occurs in a real place (Slater et al., 1994), the odd appearance of the training buildings and the limited situational cues they can support may disturb engagement or limit high experiences, necessary for practicing ``the role'' of IC. It is possible to define more realistic scenarios in [R]VS, where the instructors are not limited by the available objects, thus they can populate the scene with all important objects, avatars and situations necessary to create situations that allow the students to reach the learning objectives. However, [R]VS does not automatically provide believable and engaging roleplay influencing high presence, this depends on the competence of those who are constructing the scenarios (Heldal, 2016). RVS presence also depends on different aspects regarding technology, counterplay, visual and audial impressions; it is essential to determine what these aspects are and how they influence experiences.

One of the main advantages of this study was that it provided an intervention and simultaneously studied practice-based training in context, near the practitioners and during a longer period, involving researchers who could influence settings, technologies and scenarios (Baskerville, 1999). To have domain knowledge, experiences with the technologies and constructing settings scenario is necessary for such studies. Knowledge about the state-of-the-art in research is also essential for developing practice-based learning spaces in [R]VS.

The overall goal of the work presented in this article was to investigate the role of RVS for practice-based training for IC students and instructors with previous experience from incidents in the society, and on training grounds (LS). This study indicates that the students experienced presence in RVS comparable with previously experienced presence in LS, and their presence was slightly higher than in earlier used VS performed on site. While the instructors positively experienced a new virtual learning space, some were more skeptical about using RVS due to the demands of setting up such training and their responsibility for this. However, all recognized the possibilities to assess practical training remotely as the primary value of RVS.

In RVS, face-to-face human interaction is replaced with avatars with authentic human voices. This study shows that progressing from VS to RVS, with no face-to-face human interaction, does not reduce the level of presence experienced in terms of the feeling of being in the learning scenario. The results also indicate that students with more firefighter experience feel a higher level of presence in RVS compared with their colleagues with less experience from the field. The studied RVS examination was not negatively influenced by the technology used or by technology fragmentation aspects; the technical setup supported natural communication via talking directly to avatars and via digital radio, as ICs communicate at incident scenes. A highly appreciated factor enabling presence was the well-performed live counterplay with human voices. However, these aspects require further examination in new scenarios.

During assessments, the instructors continuously observed the students, hearing everything they said and observing everything they observed. This suggests the potential for RVS to provide an enhanced assessment tool compared to LS and VS. The positive experiences of RVS led MSB to the strategic decision to implement RVS in all IC courses from 2021, replacing previous VS sessions and increasing RVS use in other courses.

One additional concrete value of this study is the avoidance of travel, which for the same training using LS would amount to almost one lap around Earth and 435 hours of traveling by car. If we regard travel time as an inefficient activity, avoiding the travel will free up 54 eight-hour sessions for other more productive or pleasant activities. Additional benefits are reduced costs, and no carcinogenic particle exposure. More research is needed to understand how graphical representations, scenario design, role-playing and the relation among these influence presence and learning.