Abstract

Scientific knowledge is still limited about the effect of commercial virtual reality content, such as experiences developed for advertising purposes, on individual emotional experience. In addition, even though correlations between emotional responses and perceived sense of presence in virtual reality have often been reported, the relationship remains unclear. Some studies have suggested an important effect of ease of interaction on both emotions and the sense of presence, but only a few studies have scientifically explored this topic. Within this context, this study aimed to: (a) test the effect of inducing positive emotions of a commercial virtual experience developed for the promotion of an urban renewal project, (b) investigate the relationship between positive emotions and the perceived sense of presence, and (c) explore the association between the ease of interaction of the virtual experience with positive emotions and the sense of presence reported by the users. Sixty-one participants were recruited from visitors to the 2017 Milan Design Week “Fuorisalone” event. A survey was administered before and after the experience to collect information about users' demographics, positive emotions, sense of presence, and the ease of interaction with the virtual content. Results give evidence that: (a) the commercial virtual reality experience was able to induce positive emotions; (b) the positive emotions reported by users were associated with the sense of presence experienced in the virtual environment, with a directional effect from emotion to sense of presence; and (c) the easier the interaction, the more the sense of presence and positive emotions were reported by users.

1 Introduction

Virtual reality has already proven its potential in many application domains, such as the military (Pallavicini et al., 2016; Rizzo et al., 2011), medicine (Willaert et al., 2012; Zendejas et al., 2013), and mental health (Freeman et al., 2017). Since 2015, thanks to the launch of several commercial headsets on the market such as PlayStation VR (Sony Interactive Entertainment), HTC Vive (HTC), and Oculus Go (Oculus), and their increasingly affordable cost, virtual reality has stepped outside the boundaries of research laboratories and landed in new application fields, including marketing and entertainment (Walker et al., 2016; Muzellec, Lynn, & Lambkin, 2012; Stott, 2016).

One of the main drivers of virtual reality's success in all these fields can be attributed to the fact that, in comparison with traditional media such as photography or video, this technology can have a greater affective impact on the user (Gorini et al., 2010; Villani et al., 2012). In contrast to what happens while watching a video or playing a video game on a desktop display, users become part of the content itself, having the ability to interact with the virtual environment in a more or less realistic and complex way, thanks to the integration and use of trackers and special controllers (Pallavicini, Pepe, & Minissi, 2019). Because of these unique characteristics, virtual reality offers incomparable opportunities, allowing people to experience what has been technically defined as a “sense of presence,” the sensation of truly being inside the virtual environment (e.g., Botella et al., 2009; Riva, 2009; Riva et al., 2004).

Numerous studies have shown that ad hoc-developed virtual reality content can effectively elicit different emotions in individuals, including positive emotions such as joy (e.g., Felnhofer et al., 2015; Herrero et al., 2014). Interestingly, recent studies have reported preliminary evidence supporting an increase in positive emotions (i.e., joy and surprise) in the users after experiencing commercial virtual reality content, specifically video games (e.g., Pallavicini et al., 2019; Shelstad, Smith, & Chaparro, 2017). Far less is known about the effect on the emotional experience on the users of other kinds of virtual reality content, such as experiences developed for advertising purposes. In particular, very few studies have so far investigated such new kinds of content and their effect on users' emotional responses.

Correlations between emotional responses and perceived sense of presence in virtual reality environments have often been reported (e.g., Alsina-Jurnet et al., 2011; Riva et al., 2007). Nonetheless, even if presence is often regarded as a necessary mediator that allows real emotions to be activated by a virtual environment (Felnhofer et al., 2015; Slater, 2004), the relationship remains unclear, as results emerging from the literature appear to be mixed.

Recently, numerous studies and theoretical approaches have suggested that the specific virtual reality system's characteristics, including the ease of interaction with the system, serve a fundamental role in how effectively virtual environments can elicit emotions and sense of presence (e.g., Diemer et al., 2015; Lombard & Ditton, 1997; Marsh et al., 2001). Previous studies had identified disturbances in interaction modalities as potentially limiting the users' overall emotional experience and the perceived sense of presence in the virtual environment (Marsh et al., 2001; Pallavicini et al., 2013). However, only a few studies have scientifically explored this topic.

Therefore, this study aims to explore the effectiveness of a commercial virtual experience (i.e., the Urban Up Milan Virtual Experience, developed for the promotion of an urban renewal project and presented during an on-site event) in eliciting positive emotions in users, to investigate the link between emotional responses (i.e., self-reported intensity of positive emotions) and sense of presence experienced in the virtual content, as well as the effect of the virtual reality system's ease of interaction on emotional responses and on the perceived sense of presence. The purpose is to provide data and information concerning these topics, hence contributing to the scientific knowledge in this field.

2 Background

2.1 Virtual Reality as an “Affective Medium”

One of the main reasons for the unprecedented success of virtual reality is its greater affective impact on the user, compared to traditional, less immersive media such as photos and videos (e.g., Gorini et al., 2010; Villani et al., 2012). Thanks to its unique characteristics of immersion, defined as “the quantifiable description of a technology, which includes the extent to which the computer displays are extensive, surrounding, inclusive, vivid and matching” (Slater et al., 1996), and interactivity (e.g., Baños et al., 2008; Felnhofer et al., 2015), virtual reality is able to induce moderate to intense emotional, behavioral and physiological responses, and can be defined as an “affective medium” (Riva et al., 2007).

Before moving on to a discussion of the affective responses to virtual reality, it is important to clarify the terms affect, emotion, and mood. Even though these terms are often used interchangeably (Jeon, 2017), some distinct characteristics meaningfully distinguish the different types of affective states (Batson, Shaw, & Oleson, 1992; Beedie, Terry, & Lane, 2005). In particular, while researchers agree that the term affect refers to the subjective feelings of any type of affective experience (Forgas, 1995; Frijda & Scherer, 2009; Hudlicka, 2003), emotions and mood can be differentiated on the basis of intensity, duration, and cause (Beedie et al., 2005). While emotion is characterized by being short-term (lasting minutes), intense, and caused by a specific cue (Ekman & Davidson, 1994), mood states are considered rather long-term, lower intensity, with nonspecific causes (Bodenhausen, Sheppard, & Kramer, 1994; Forgas, 1995). Furthermore, there are two approaches commonly used in emotion classification: the discrete and the dimensional models. The discrete approach posits the existence of a small set of basic emotions. For example, in the model developed by Ekman, six basic emotions are proposed (i.e., anger, disgust, fear, joy, sadness, and surprise) (Ekman, 1992). Dimensional models instead consider emotions as multidimensional spaces, where each dimension represents property common to all emotions (Jeon, 2017). For example, the Circumplex Model discerns emotions on the basis of two dimensions: valence (positive and negative emotions) and arousal (i.e., the intensity of the emotion) (Russell, 1980).

Many studies have shown the ability of ad hoc virtual reality experiences developed by researchers to elicit different emotions through the interaction with their content. In particular, studies have demonstrated that virtual reality is generally capable of eliciting more intense emotional responses compared with less immersive display devices (i.e., screen or desktop displays) (e.g., Estupiñán et al., 2014; Pallavicini et al., 2018, 2019), to the point that, in some cases, such emotional responses are of comparable intensity to what we perceive in real life (Gorini et al., 2010; Villani & Riva, 2012). For example, it has been demonstrated that virtual food is as effective as real food, and more effective than photographs of food, in inducing anxiety in patients suffering from bulimia or anorexia (Gorini et al., 2010). Furthermore, an immersive virtual reality job simulation was found to be more effective than a real world simulation in eliciting a perceived anxiety state in individuals (Villani et al., 2012). This ability of virtual reality to elicit emotions has been used extensively for clinical purposes, for example, as a new medium for exposure therapy or for relaxation in the treatment of anxiety disorders (Maples-Keller et al., 2017; Serino et al., 2014).

Virtual reality has been shown not only that it can induce emotional and behavioral responses similar to (or higher than) those that occur in the real world, but also that it can drive very intense feelings that are difficult to recreate in a laboratory setting, such as awe and wonder (Chirico et al., 2017, 2018; Quesnel et al., 2017). According to Quesnel and colleagues (2017), a relevant sense of wonder might be reached if the virtual content is responsive and personalized. Regarding awe, a recent study has shown how immersive videos significantly enhanced the self-reported intensity of awe and led to higher parasympathetic activation compared with the same videos presented in 2D (Chirico et al., 2017). Moreover, virtual environments designed to induce awe caused significantly higher levels of awe than a neutral virtual environment did, and induced significantly more positive than negative affects (Chirico et al., 2018).

Furthermore, immersive virtual environments created ad hoc for emotional induction were effective in eliciting positive emotions, both in healthy individuals (e.g., Felnhofer et al., 2015) and in people suffering from different conditions, including chronic pain (Herrero et al., 2014), anxiety (Gorini & Riva, 2008; Pallavicini et al, 2009; Repetto et al., 2011, Maples-Keller et al., 2017), and stress disorders (Gaggioli et al., 2014; Serino et al., 2014). For example, a virtual environment specifically designed for chronic pain patients proved to be effective in increasing the intensity of positive emotions including joy and surprise (Herrero et al., 2014).

Similar results have also been observed from commercial virtual reality content. In fact, recent studies have reported a greater increase in positive emotions after playing a virtual reality game than were seen in less immersive (i.e., desktop) displays (Pallavicini et al., 2018, 2019; Shelstad et al., 2017). Specifically, after a virtual reality gaming experience (i.e., a first-person shooter game, and a survival horror game), statistically greater increases in enjoyment (Shelstad et al., 2017), happiness (Pallavicini et al., 2018, 2019), and surprise (Pallavicini et al., 2018) have been reported.

Nonetheless, less is known about the effect on emotional experience of other kinds of virtual reality commercial content, including those developed for advertising purposes. Although virtual reality has already been used in the marketing business, usually through on-site promotional experiences during events or fairs, by renowned brands (e.g., Adidas, McDonald's, National Geographic, and Absolut Vodka) (Altarteer et al., 2013; Alvares, 2016; Stott, 2016), and it is more and more often used for the promotion of cultural heritage (Carrozzino & Bergamasco, 2010; Garau & Ilardi, 2014; Pantano & Corvello, 2014) as well as for tourist destinations (Guttentag, 2010; Huang et al., 2016), knowledge about the effects of this kind of content on users' emotional experience is still limited.

2.2 Presence and Emotional Responses in Virtual Environments

Presence represents a crucial factor to be considered when dealing with human experience inside virtual systems, and it is often regarded as a measure of quality in order to assess, develop, and optimize virtual content (Bystrom et al., 1999; Riva et al., 2011). One of the well-known and adopted definitions of presence in scientific literature states that it is the “perceptual illusion of non-mediation” (Lombard & Ditton, 1997), that is, the perception to interact with a virtual stimulus or environment directly and without the support of any technology (Lee, 2004; Schubert, Friedmann, & Regenbrecht, 2001). Most researchers agree that presence is a multidimensional construct (Kalawsky, 2000), composed of several factors (Lee, 2004; Schubert et al., 2001; Witmer & Singer, 1998). Among the different classifications of presence (e.g., Heeter, 1992; IJsselsteijn et al., 2000), one of the most frequently adopted in literature distinguishes between three principal elements (Schubert, Friedmann, & Regenbrecht, 2001): spatial presence, the feeling that one is physically in the virtual space; involvement, the extent to which one keeps attention focused on the virtual stimulus and ignores competing incongruent information; and realness, or the extent to which the virtual stimulus coincides with expectations of the real stimulus.

From the beginning, virtual reality researchers linked the concept of presence to the emotion-processing theory (Foa & Kozak, 1986), considering presence as the precondition for emotional responses in virtual environments (Parsons & Rizzo, 2008), a necessary mediator that allows real emotions to be activated by a virtual environment (e.g., Felnhofer et al., 2014, 2015; Slater, 2004). According to this theoretical assumption, presence levels are not connected to the type of emotion (Slater, 2004). Supporting this hypothesis, for example, no difference has been reported in the levels of presence experienced in five different virtual park scenarios, eliciting respectively joy, sadness, boredom, anger, and anxiety (Felnhofer et al., 2015).

However, the relationship between the sense of presence and the emotions experienced in virtual reality is still not clear, as described above, and the link between the sense of presence and the emotional response has been identified as a critical topic to investigate (Villani, Lucchetta, Preziosa, & Riva, 2009). Other research, in fact, has reported that presence level may instead be related to the intensity of the experienced emotional state (e.g., Baños et al., 2008; Riva et al., 2007; Robillard et al., 2003). For instance, Riva and colleagues (Riva et al., 2007), in one of the very first studies conducted on the topic, found a difference between their emotional parks (i.e., “anxious” and “relaxing” virtual environment) and a neutral park, with the latter showing significantly lower levels of presence than was seen in the emotional scenarios. The significant contributions of emotions felt in virtual environments to the subjective feeling of presence have been reported, in particular regarding anxiety, both in healthy individuals (Baños et al., 2004), and in people suffering from phobias (Price et al., 2007). In addition, in a study investigating how stereoscopy (i.e., the illusion of depth and 3D imaging) can affect the sense of presence and the intensity of the emotional experience (i.e., joy, sadness, anxiety, and relaxation) that users feel in two virtual environments (i.e., sadness versus neutral), the emotional content of the virtual environment had an impact on the sense of presence: participants reporting a higher sense of presence also experienced more intense positive emotions (i.e., joy and relaxation) (Baños et al., 2008). Interestingly, some authors reported that relevant emotional background information also enhanced presence, indicating a causal influence of emotions on presence (Bouchard et al., 2008; Gorini et al., 2011).

The importance of emotions, especially arousal, on the sense of presence is also underlined by a recent theoretical approach, using the interoceptive attribution model (Diemer et al., 2015). According to this model, individuals judge the degree of presence they feel in virtual reality based on two fundamental elements: (1) the immersion of the VR system, defined as “what the technology delivers from an objective point of view” (Slater, 2003) (i.e., the more advanced and sophisticated the virtual reality devices is in term of immersion, the higher the level of presence users will experience); and (2) the individual's cognitive judgment of the degree of emotional arousal he or she feels. As arousal is a particularly strong indicator of emotional involvement, arousing emotions should lead people to rate their presence experiences as more intense than calm or neutral emotional states (Diemer et al., 2015). The fundamental role of emotions, especially of arousal, on the sense of presence is also underlined by a previous theoretical model. In particular, according to the arousal theory on presence (Freeman et al., 2005), arousal represents a “call to action,” leading to alertness, which in turn leads to higher presence ratings in virtual environments.

The fact that the relationship between emotions and presence in virtual reality environments is still an open research question could be explained by the fact that the different theoretical positions have not been rigorously tested. In particular, studies conducted so far on this subject have used correlational statistical analysis (e.g., Bouchard et al., 2008; Felnhofer et al., 2015; Riva et al., 2007); therefore, they have given no exact indication about the link between emotion and presence.

2.3 The Impact of Ease of Interaction of the Virtual Reality System on the Sense of Presence and Emotional Experience

Many studies in the human computing interaction (HCI) field have long emphasized the importance of the design of interactive user interfaces, with particular attention toward the ease of interaction with the content, to the point that the interaction can be described as “transparent” (Winograd & Flores, 1987) or “cognitively imperceptible” (Norman, 1992). Fluid interactions are considered fundamental factors of design models for the creation of technologies that promote emotions by engagement (Hancock et al., 2005; Hassenzahl & Tractinsky, 2006; Thüring & Mahlke, 2007). For instance, the conceptual framework defined as “emotional design” (Norman, 2004), which is largely adopted in the context of developing interactive content like video games (Baharom et al., 2014), and many other technological products (Desmet et al., 2007; Triberti et al., 2017), highlights that an effective design is composed of a modality of interaction perceived as simple and intuitive, and by the involvement of users' affective/visceral level through the use of physical features such as sight, touch, and sound.

In the case of virtual reality, such considerations are even more significant because of the advanced level of interaction with the content (Slater et al., 2009). One of the traditional definitions of presence states that it is “the perceptual illusion of non-mediation” (Lombard & Ditton, 1997), emphasizing the necessity for virtual reality systems to be characterized by immediate and easy interaction modalities. In support of the importance of the ease of interaction in virtual environments, previous studies reported that breakdowns or other interaction issues can gravely compromise the effectiveness of the virtual experience from an emotional point of view (Marsh et al., 2001; Pallavicini et al., 2013) and the perceived sense of presence (Pallavicini et al., 2013; Slater & Steed, 2000). As reported by Marsh and colleagues (2001), a model of effective interaction with a virtual reality system must necessarily avoid the occurrence of breakdown in interaction in order to maintain the illusion of presence, along a continuum of engagement. As identified by this framework, generally all interaction with virtual reality content falls in one of two main groups: navigation and exploration (i.e., the user movement and scanning in the virtual environment), or object manipulation (i.e., the identification and selection of objects in the virtual environment) (Marsh et al., 2001). Moreover, a previous study demonstrated that technological breakdowns in exploration (i.e., the head tracking was randomly reversed for 20 sec during the experience) significantly reduce the ability of virtual reality to induce both emotions and a satisfactory sense of presence, to the point that they are even lower than the ones elicited by less immersive media, such as audio and video (Pallavicini et al., 2013).

Even if fluid interactions do seem to correlate with emotional responses in virtual reality, and the sense of presence seems to be closely related to those emotions, only a few studies have scientifically explored this topic.

2.4 Overview of the Study

Within the context described above, this study has three main objectives: (1) to test the effectiveness of a commercial virtual experience developed for the promotion of a urban renewal project, the Urban Up Milan Virtual Experience, in inducing positive emotions; (2) to investigate the relationship between emotional responses (i.e., self-reported intensity of positive emotions) and the perceived sense of presence; and (3) to explore the effect of ease of interaction on positive emotions and the perceived sense of presence.

The virtual experience was developed by the Unipol Group in collaboration with Proxima Milano (Proxima VR) for the promotion of buildings constructed or renovated by the group itself in the city of Milan, and it has been tested “on site” (i.e., out of the laboratory) during the four days of the 2017 Milan Design Week Fuorisalone event, an interior design fair that takes place yearly in the city.

The main hypotheses explored by the study (see also Figure 1) were:

  • (H1) The virtual reality experience will induce positive emotions (i.e., joy, surprise, curiosity, and enjoyment) in users;

  • (H2) The positive emotions that will be reported by users are associated to the sense ofpresence experienced in the virtual environment:

    • (H2.1) the stronger positive emotions will be experienced by the users, the greater sense of presence will be reported by them (Model A);

    • (H2.2) the greater sense of presence will be reported by individuals, the stronger positive emotions will be experienced by them (Model B);

  • (H3) The ease of interaction in the virtualenvironment will be associated with the positiveemotions and the sense of presence experienced bythe users.

Figure 1.

Screenshot of the content of the Urban Up Milan Virtual Experience.

Figure 1.

Screenshot of the content of the Urban Up Milan Virtual Experience.

3 Materials and Methods

3.1 Participants

There were 61 participants: 32 females (52.5%) and 29 males (47.5%); age range (years old): 18–24 = 16 (26.2%); 25–34 = 8 (13.1%); 35–50 = 18 (29.5%); 51–70 = 17 (27.9%); over 70 = 2 (3.3%); previous experience with virtual reality: yes = 32 (52.5%); no = 29 (47.5%). Participants were recruited in April 2017 from visitors to the 2017 Milan Design Week Fuorisalone event in the city of Milan, Italy. No economic rewards were provided during the study. Before participating, all subjects were given written information about the study and were required to give written consent in order to be included. The study received ethical approval by the Ethical Committee of the University of Milano-Bicocca, and it was conducted according to APA norms of conduct (APA, 2010).

3.2 Psychometric Assessment

Before and after the virtual reality experience, self-report questionnaires were administered to participants using a tablet (iPad, 2nd generation, Apple). The following questions were included:

  • Demographics: before starting the experience, participants were asked to indicate:

    • Gender and age range: participants were asked to indicate their gender (“female or male”) and their age range (“18–24”; “25–34”; “35–50”; “51–70”; “over 70”);

    • Previous experience with virtual reality: participants were asked whether they had previous experiences with virtual reality (“yes” or “no”);

  • Positive emotions: in order to assess differences in individuals' emotional state, the Visual Analogue Scale (VAS)—a horizontal line, 10 cm in length, anchored by word descriptors at each end—was given before and after the experience; participants were asked to rate from 0 to 10 their perception of their inner state with reference to four different affects: Happiness (VAS-HP), Enjoyment (VAS-EN), Surprise (VAS-SUR), and Curiosity (VAS-CUR). The aggregate score of positive emotions reported excellent reliability (α= .811; Cronbach, 1951) and good inter-item correlation (r = .530).

  • Sense of presence: after the experience, participants were asked to answer the UCL Presence Questionnaire (Slater, Usoh, & Steed, 1994); respondents were required to rate three questions on a 1–7 point Likert scale: Q1—“Rate your sense of being in the virtual environment”; Q2—“To what extent were there times during the experience when the virtual environment was reality for you?”; Q3—“When you think back to the experience, do you think of the virtual environment more as an images that you saw or more as somewhere that you visited?”; Cronbach's reliability for the UCL measure was .732.

  • Ease of interaction: after the virtual experience participants were asked to rate the ease of interaction on a 10-point scale (from “not at all” to “very easy”).

3.3 Virtual Reality Content: Urban Up Milan Virtual Experience

Urban Up Milan Virtual Experience is a virtual reality experience related to several future buildings in the city of Milan. It was developed by the VFX&XR studio Proxima Milano for Unipol Group to promote the buildings constructed or renovated by the group itself in the city of Milan. It was presented during the 2017 Milan Design Week Fuorisalone event, an interior design fair that takes place in the city yearly. This virtual experience is a 3D CGI reconstruction of five sites (i.e., Torre Unipol, De Castilia 23, Torre Galfa, The Big, and Ca' Litta) on which the Unipol Group is working in the context of the urban renewal project (i.e., Urban Up Unipol Project Cities), with the aim of promoting some of the most important buildings of Italian heritage. It has been optimized for the HTC Vive, and allows its users to completely immerse themselves in the five buildings, and to navigate through them, moving from one room to another using HTC controllers and an HTC tracker (linked to a branded object that users can hold in their hand).

3.4 Procedure

Visitors to the space dedicated to the Urban Up Milan Virtual Experience at the 2017 Milan Design Week Fuorisalone event were asked whether they would like to participate in a study concerning the user experience of the virtual reality content. Upon their consent, they were given a questionnaire on a tablet (iPad, 2nd generation) before the virtual experience. The questionnaire included demographic questions (i.e., age range, gender), and questions assessing the users' knowledge of virtual reality technology. In order to assess differences in individuals' emotional state, a shortened version of the Visual Analogue Scale (VAS) was given to the participants, assessing happiness (VAS-HP), enjoyment (VAS-EN), surprise (VAS-SUR), and curiosity (VAS-CUR). Questionnaire completion took participants no more than five minutes to complete, being specifically created for an on-site evaluation, and the included questions were accurately selected in expectation of a consistent turnout.

Then, participants were accommodated in a dedicated space, an area of about 3 × 3 meters, and were asked to wear an HTC Vive, linked to an Omen X by HP desktop PC, with which they experienced the virtual content. The HTC Vive was connected to an Omen X by HP desktop PC through a 5-m cable with an HMDI connection, a USB 2 connection, and power. Participants were asked to wear the HMD and were given the HTC Vive controllers. The audio level was set to 45 for all participants. Subjectively, tracking appeared stable when using this configuration, and the virtual experience was playable with no visible tracker artifacts. All measurements were taken in an 8 × 5-m room with a 3.2-m high ceiling lighted by fluorescent lighting, with no reflective surfaces and no exposure to natural lighting. In the center of this room, a 3 × 3-m grid (i.e., the virtual reality area) was drawn on the floor using string and chalk, with grid lines drawn 1 m apart. After a brief training about the HTC Vive controls, participants, through HTC controllers and a tracker, could select and explore the virtual buildings. Users were free to walk outside and inside the buildings, interact with the environment, select videos, and activate interactive graphics within the virtual environment. The experience lasted about five minutes, during which users were given the ability to explore two buildings of their choice. After the virtual experience, participants were asked to answer the UCL Presence Questionnaire (Slater, Usoh, & Steed, 1994), and to assess the ease of interaction of the virtual experience. The VAS was then given to the participants in order to assess differences in emotional activation after the experience (VAS-HP, VAS-EN, VAS-SUR, VAS-CUR).

3.5 Statistical Analyses and Experimental Design

Statistical2 analyses were conducted during two successive steps. First, a preprocessing phase was set in order to test if variables were normally distributed (skewness and kurtosis values) and to detect the presence of both missing values and multi-variate outliers (Mahalanobis' distance was set to be equal to .001). As a result, two (3.2%) multivariate outliers were omitted from the analysis whereas all other variables under study were normally distributed with all skewness values within the [−2; +2] range (George & Mallery, 2010). Main descriptive and zero-order correlations were subsequently computed.

Next, the network of relationships between positive emotion and presence was analyzed via structural equation modelling (SEM) for repeated measures (Skrondal & Rabe-Hesketh, 2004). This analytical strategy was based on testing a given hypothesized set of both direct and indirect effects among variables and evaluating the extent to which the conceptual model fit with empirical data (i.e., between reproduced and empirical covariance matrix; Kline, 2015). The statistical and practical significance of the model could then be assessed by mean of goodness of fit indexes. In the present study, the following absolute and relative goodness of fit indexes were adopted: χ2 (normed chi-square) [a nonstatistically significant χ2 value and NC values of under 2.0 indicate good fit (Hair et al., 2010)]; root mean square error of approximation (RMSEA), normed fit index (NFI), non-normed fit index (NNFI), and comparative fit index (CFI). Thresholds for good model fit were: RMSEA <.07 (Schermelleh-Engel, Moosbrugger, & Müller, 2003), NFI >.95, NNFI >.95 (Marsh & Hau, 1996), CFI >.95 (Hu & Bentler, 1999).

In order to address the first two main hypotheses of the present study (HP1 and HP2), the model was set with positive emotions (pre- and post-experience) and sense of presence as endogenous variables. Standardized indirect and direct effect were presented. Since the direction of the association between emotion and sense of presence had not been addressed unequivocally by previous studies (see, for instance, Baños et al., 2004 or Riva et al., 2007), we tested two structurally equivalent models: Model A was estimated with a direct effect of emotions on sense of presence and Model B with a reversed direct effect from sense of presence to emotions. The two models were finally evaluated following the Akaike information criterion (AIC; Bozdogan, 1987). The maximum likelihood (ML) method was used to estimate coefficients. Following Tofighi & Mackinnon's (2016) recommendations, confidence intervals for indirect effects were computed via a parametric Monte Carlo simulation (200 bootstrap samples were used). The conceptual model is summarized in Figure 2. Given the size of the sample, especially considering the risk of Type 1 error, Bonferroni's correction was applied, and a consequence p-value for acceptance of statistical tests was set to .025 (Cabin & Mitchell, 2000).

Figure 2.

Research model (Hypothesis 2).

Figure 2.

Research model (Hypothesis 2).

Finally, in order to address HP3, two different regression models were computed. A first equation was set with sense of presence as the target variable and ease of interaction as an antecedent variable. Then, a second regression equation was estimated with pre- post-differences in self-reported scores on positive emotion as the target variable and ease of interaction as an antecedent. The role of antecedents in relation to both sense of presence and emotions was evaluated on the basis of explained variance, standardized beta weights, and general statistical significance.

4 Results

4.1 Descriptive Statistics

Table 1 summarizes main descriptive statistics of all variables under study.

Table 1.
Main Descriptives of Variables under Study
VariablesMeanSD
Happiness (Pre) 6.66 1.72 
Curiosity (Pre) 5.48 2.01 
Surprise (Pre) 7.84 1.52 
Enjoyment (Pre) 6.92 1.96 
Happiness (Post) 8.03 1.61 
Curiosity (Post) 7.9 1.81 
Surprise (Post) 8.23 1.74 
Enjoyment (Post) 8.18 1.84 
Ease of interaction 8.46 1.73 
Sense of presence 14.3 4.38 
VariablesMeanSD
Happiness (Pre) 6.66 1.72 
Curiosity (Pre) 5.48 2.01 
Surprise (Pre) 7.84 1.52 
Enjoyment (Pre) 6.92 1.96 
Happiness (Post) 8.03 1.61 
Curiosity (Post) 7.9 1.81 
Surprise (Post) 8.23 1.74 
Enjoyment (Post) 8.18 1.84 
Ease of interaction 8.46 1.73 
Sense of presence 14.3 4.38 

4.2 Results of Structural Equation Model

The structural path of the model was tested using Analysis of Moment Structure (Arbuckle, 2014). The overall fit of Model A reported acceptable goodness of fit indexes: chi-square (42) = 56.68, p = .063; RMSEA = .077, C.I. 90% [.001 −.111], pclose = .275, NFI = .861, NNFI = .958, CFI = .960, AIC = 126.78. Data conceptually and statistically supported the effects among the variables under study. The only violation of the selected cut-off point for model fit was in the correspondence to NFI, however this index tends to underperform in relatively small samples (Hoyle, 1995). The associations among variables were evaluated by decomposing the total standardized effects yielded by each in direct and indirect effects (values are reported in Figure 3). All estimates were in the expected directions and within admissible boundaries (e.g. no Heywood case was found; Kolenikov & Bollen, 2012) (see Figure 3 for standardized direct effects).

Figure 3.

Results of the structural equational model (Hypothesis 2). Standardized direct effects are reported. The one not-statistically significant value (.04) is in italics.

Figure 3.

Results of the structural equational model (Hypothesis 2). Standardized direct effects are reported. The one not-statistically significant value (.04) is in italics.

With regard to HP1 (i.e., the virtual reality experience will induce positive emotions in users), the model revealed a direct positive and statistically significant effect (B = .71, C.I. 95th = .370 – 1.21, p = .019) of the virtual experience on all positive emotions, meaning that experiencing the virtual content is associated (i.e., indirect effect for all emotions were reported in brackets) with higher scores on surprise (B = .65), happiness (B = .70), enjoyment (B = .71) and curiosity (B = .64).

With regard to HP2, Model A supported the direct path from emotion to sense of presence (HP 2.1). The effect was positive and statistically significant (B = .35, C.I. 95th = 0.099 – 0.630, p = .010). The analysis of Model B (HP2.2) revealed that the AIC value (AIC = 128.18) was lower if compared to model A (AIC = 126.78) suggesting that the best fitting model included a direct directional effect from emotion to sense of presence. In addition to the evaluation of the AIC values, Model B also reported generally lower goodness-of-fit indexes: chi-square (42) = 58.78, p = .050; RMSEA = .080, C.I. 90% [.004 - .127], pclose = .169, NFI = .858, NNFI = .940, CFI = .954, AIC = 128.18.

Finally, the regression equation was computed in order to test whether and to what extent ease of interaction scores were associated with sense of presence. The result of the equation suggested a statistically significant association between the two variables: F (1,60) = 7.77, p = .007). Corrected explained variance (R2) was equal to .12. In this case, the standardized weight of interaction on sense of presence was medium, statistically significant and positive (β= .341, p = .007), meaning that the easier the interaction, the more the sense of presence. In a similar fashion, the second regression equation tests whether and to what extent ease of interaction scores were associated with positive emotions, supported the model statistically: F (1,60) = 8.04, p = .006). In this regard, the standardized beta weight (β= .341, p = .007) suggested that the association between ease of interaction and positive emotions after the experience was positive, medium, and statistically significant. All in all, the result supported the acceptance of the last hypothesis.

5 Discussion

As hypothesized in the first main hypothesis of this study, data showed a statistically significant improvement in the users' positive emotions after the virtual experience. In particular, the analysis of comparison index (AIC) along with other goodness-of-fit indexes revealed a direct positive and statistically significant effect of the virtual experience on positive emotions, meaning that experiencing the virtual content was associated with higher scores on enjoyment, surprise, happiness, and curiosity reported by the users. Such ability of virtual reality content to elicit positive emotions has been widely demonstrated in virtual environments designed ad hoc by researchers for emotional induction, both in healthy individuals (e.g., Baños et al., 2014; Felnhofer et al., 2015), and in patients suffering from different disorders such as anxiety (e.g., Herrero et al., 2014; Maples-Keller et al., 2017; Repetto et al., 2011). This feature has been adopted extensively to improve individuals' well-being, since, as stated by the broaden-and-build model (Fredrickson, 2001), positive emotions provide an organism with non-specific action tendencies that can lead to adaptive behavior, such as interacting more with others, or engaging in creative challenges (Diener, 2000; Fredrickson, 2000). Therefore, as demonstrated by the positive technology approach (Riva et al., 2012), technology, including virtual reality, can be an effective tool for improving the quality of people's personal experiences (Herrero et al., 2014; Riva et al., 2016).

What emerges from this study, interestingly, is in line with what has been observed recently with regard to another genre of commercial virtual reality content, in particular, video games (Pallavicini et al., 2018, 2019; Shelstad, Smith, & Chaparro, 2017): not only virtual content specifically created for emotional induction, but also commercial virtual content—in this case, a virtual experience developed for the promotion of an urban renewal project and proposed during an on-site event—can be effective in inducing positive emotions in individuals.

Positive emotions appear to be strongly linked to the use of technological products and the overall level of user satisfaction with them, as underlined by the “emotional design” (Norman, 2004) of a conceptual framework, that is largely adopted in the context of developing interactive technologies (Baharom et al., 2014; Desmet et al., 2007; Triberti et al., 2017). If the results of the present exploratory study are confirmed by future research, they might represent positive evidence for the adoption of virtual reality technology in rather recent fields of application, such as for advertising purposes. Virtual reality technology could represent a tool possessing enormous potential, offering the possibility to create unforgettable experiences, while giving an emotional connection to engaged users (Greenwald, 2017; Xu, Park, & Baek, 2011), and creating a solid association in the customers' minds between the brand's values and their own experience (Mandelbaum, 2015; Pharr, 2011). Nonetheless, scientific studies exploring the capacity of this new kind of commercial virtual content to elicit positive emotions in users are still fairly limited and need future follow up. For instance, it would be interesting to compare the effect of promotional commercial virtual content in inducing positive emotions in users in comparison with non-immersive media, with the objective of assessing and comparing the effect of both instruments for this aim. Since virtual reality presents higher costs and longer development time than traditional advertising media, deepening the scientific knowledge about the effects of this medium on the emotional response of the users would help to justify its adoption.

Secondly, given that the link between the sense of presence and the emotional response in virtual reality is still not clear, making it a critical topic to investigate (Villani et al., 2009), the second objective of this study was to explore the network of relationships among positive emotion and presence. The analysis of two structurally equation models (Model A versus Model B) was included in order to cross-validate and test the direct effect in both directions; while Model A hypothesizes an effect of positive emotions on presence, Model B hypothesizes the opposite (i.e., an effect of presence on positive emotions). The analysis of comparison index (AIC) along with other goodness-of-fit indexes revealed that the model with the direct effect from emotions to sense of presence (Model A) was the best fitting solution.

These findings may be regarded as a strong aspect of this study as they offer interesting insights on the scientific debate about the relationship between emotions and presence. In particular, results of this study appear to support the theoretical assumption that presence experience is related to the intensity of the emotional responses in the virtual environment, as suggested by previous research (e.g., Baños et al., 2008; Bouchard et al., 2008; Riva et al., 2007), and are in line with what has been hypothesized by arousal theory (Freeman et al., 2005), and later by the interoceptive attribution model (Diemer et al., 2015). On the other hand, what emerged from this study seems to disagree with the theoretical approach that considers presence a prerequisite for feeling emotions in virtual environments, suggesting a direct effect of presence on emotions (e.g., Felnhofer et al., 2015; Slater, 2004). In previous studies (e.g., Baños et al., 2008; Bouchard et al., 2008; Riva et al., 2007), different from what has been carried out in ours, specific statistical methods were not used in order to analyze direct and indirect effects among emotions and presence; therefore, they do not support any causal relationship. What emerged from our study, despite being exploratory in nature, may deepen the scientific knowledge about this complex relationship, suggesting a direct effect of emotions on sense of presence. Future studies are necessary in order to verify such findings, for instance, through the experimental manipulation of the level of emotional intensity of the virtual content (e.g., neutral, low, high).

It is important to note that this discussion is based on data collected on emotional responses with self-reported measures. In the future, in order to investigate this subject more deeply, it would be useful to adopt physiological measures as well (Bowman et al., 2002; Chanel et al., 2006). For instance, future studies should assess skin conductance response and heart rate, indexes that are easy to record with mobile sensors, and that are strongly linked to arousal responses (e.g., Boucsein, 1992). If what has been observed in this study about the fundamental role of emotions on the sense of presence experienced in virtual environments is confirmed by future research, it would not only be an important element within a still open theoretical debate, but could also offer useful insights for the design of virtual reality content. In particular, it would become essential for designers of virtual environments not to focus on the search for technological solutions capable of guaranteeing realistic virtual environments or the integration of different sensory feedback, but instead to focus on the elements of virtual content able to elicit emotions in users, such as graphic aspects, particularly concerning colors, contrasts, shapes (e.g., Naz et al., 2017), and narrative (Bouchard et al., 2008; Gorini et al., 2011). This could also have a significant impact on the cost and development process of producing virtual content, reducing the need for “a race to the most technologically advanced virtual reality system,” to emphasize instead the fundamental role of an expert of emotions, such as a human–computer interaction psychologist.

Finally, the third objective of this study was to test whether and to what extent ease of interaction scores were associated with sense of presence, and to positive emotions. The result of the equation suggested a statistically significant association between ease of interaction and sense of presence, as well as between ease of interaction and positive emotions. Data derived from this study appear to support what has been observed in some previous research, which reported that the ease of interaction with the virtual system is related both to emotional responses and to the perceived sense of presence experienced by individuals (Marsh et al., 2001; Pallavicini et al., 2013). In line with these studies, results of this study showed that participants who reported a higher ease of interaction also reported higher levels of positive emotions as well as a higher sense of presence, meaning that the easier the interaction, the more the sense of presence and positive emotions were reported by users.

This result is interesting not only because it offers insights into a topic that is still little explored on a scientific level, but also because of the impact it could have on the design of virtual reality content. In particular, if ease of interaction is such an important factor for creating experiences with a strong emotional response and that are highly engaging for the user, designers should focus on this aspect from the initial phases of the development of the virtual content. As noted by Marsh and colleagues (2001), one of the main goals of virtual reality, like film, is to maintain users' attention in the content/illusion of a virtual reality system, keeping the spectators' illusion of non-mediations. As in Hollywood filmmaking, the development of an “invisible style” becomes central, in order to allow the viewers to focus on the content and immerse themselves in the variety of experiences intended by the developers. Otherwise, the user becomes conscious (momentarily or totally) of the external world, with a “break in illusion” (Marsh et al., 2001). Awareness of this phenomenon is useful; for example, it could be used in the development of virtual content in the Slater and Steed's break in presence (BIP) model (Slater & Steed, 2000; Steed, Vinayagamoorthy, & Brogni, 2005), which provides a framework and a method for usability analysis during the development process of virtual content.

Future studies should thoroughly investigate the effect of ease of interaction on emotions and sense of presence, for instance, experimentally manipulating the level of ease of interaction with the virtual content (e.g., fluid or with breaks of different intensity), or the type of interaction breakdowns (i.e., navigation and exploration, or object manipulation). Furthermore, in order to be able to clarify the exact link between ease of interaction, emotions and presence, future studies, instead of using regression—adopted in this study because of the limited sample size—could analyze the relationship among these variables through statistical methods with greater explanatory power, such as the partial least squares path modeling (PLS-PM). In such a way, a more complete overview would be obtained.

5.1 Limitations

Even though the results of this study contribute to the scientific context of virtual reality, this research has several important limitations that may restrict the generalizability of what has been observed. First of all, it is important to note that the obtained results refer to a specific virtual reality system (i.e., HTC Vive). It would be interesting to replicate this study using other commercial virtual reality systems with different characteristics in terms of immersion and possible interaction with the content, including for instance, mobile virtual reality systems such as Samsung Gear VR (Oculus) or more complex systems, such as CAVE. Moreover, results emerging from this study refer to specific commercial content, in this case created for the promotion of renovation projects of several historical buildings of the city of Milan, sponsored by the insurance company owning the buildings themselves. Such content presents mixed characteristics, involving pure entertainment (i.e., offers a positive experience to visitors thanks to the adoption of a not-well-known technology), the promotion of cultural heritage (i.e., provides the possibility to visit historically relevant buildings normally closed to the public), and marketing elements (i.e., the possibility for the insurance company to be positively associated with this experience). It would be interesting to untie such different factors and to study commercial content specifically created for each of these contexts, for instance, evaluating virtual reality advertising content developed to increase sales of a specific product (e.g., cars, clothes, etc.), or to advertise a specific museum or cultural event.

Secondly, due to the necessity to limit the length of the questionnaires in order not to tamper with the user experience during the on-site event, in this study the ease of interaction was evaluated using a single-item questionnaire. Even though previous research demonstrated a good test-retest reliability of single-item questionnaires (Bouchard et al., 2004; Dolbier et al., 2005; Littman et al., 2006), a better strategy in the future would be to adopt longer questionnaires or other instruments to assess ease of interaction such as the walkthrough method developed by Sutcliffe and Kaur (2000), or to measure the number and type of interaction issues shown by users in the virtual content. In addition, only some aspects of the virtual experience and their effects on the user have been evaluated in this study. For instance, it would be interesting for future studies to include other possible influencing factors, such as the perceived usefulness as suggested by technology acceptance models (Davis, 1989).

Moreover, it would be interesting to investigate the level of cybersickness experienced by the users during the experience, because of its possible negative impact on user experience (e.g., Lin et al., 2015; Nalivaiko et al., 2015), and to determine the impact of graphic aspects of the virtual reality content as perceived by the users, particularly concerning colors, contrasts, shapes and level of graphic realism, on self-reported emotions and sense of presence (e.g., Naz et al., 2017). Furthermore, it would be interesting to explore the behavioral effect of the virtual content on the users, for instance, the willingness to visit the building in real life after experiencing it in virtual reality. In our case, this aspect could not be evaluated, as the content was not developed with the intention to promote visits to the buildings because they still are under renovation and, therefore, are closed to the public. Such aspect may be worth exploring, in particular comparing the effectiveness of virtual reality with traditional methods (e.g., press or television campaigns, etc.) in encouraging visits to museums and/or cultural heritage sites.

Finally, it is important to emphasize the specificity of the included sample, that is, volunteers recruited during a design-related event in the city of Milan. Despite the event being located in a central area of the city, a way station for numerous people with a variety of demographic characteristics and different levels of previous experience with virtual reality systems, the sample was not recruited through a randomized method. Future studies should be conducted on a heterogeneous sample in terms of age, gender, and level of interest in technology and virtual reality, and a means of randomization should be introduced in the recruiting process in order to avoid bias.

6 Conclusion

To summarize, the present exploratory study gives evidence that: (a) the commercial virtual reality experience was effective in inducing positive emotions; (b) the positive emotions reported by users were associated with the sense of presence experienced in the virtual environment, with a direct directional effect from emotion to sense of presence; and (c) the easier the interaction, the more the sense of presence and positive emotions were reported by users.

Acknowledgments

The authors would like to thank the company Proxima Milano, and especially Andrea Masera, Claudio Falconi, Riccardo Gemelli, Aimone Bodini, Davide Mensi, Cristina Panizzuti, Nicola Danese, Valentina Arcelloni, Francesco Ferraresi, Federico Gnoni, Davide Mensi, Felice Amoruso, and Operà Music for the artistic and technical development of the Urban Up Milan Virtual Experience.

References

Alsina-Jurnet
,
I.
,
Gutiérrez-Maldonado
,
J.
, &
Rangel-Gómez
,
M. V.
(
2011
).
The role of presence in the level of anxiety experienced in clinical virtual environments
.
Computers in Human Behavior
,
27
(
1
),
504
512
.
Altarteer
,
S.
,
Charissis
,
V.
,
Harrison
,
D.
, &
Chan
,
W.
(
2013
). Interactive virtual reality shopping and the impact in luxury brands (pp.
221
230
).
Berlin, Heidelberg
:
Springer
.
Alvares
,
E.
(
2016
).
Inside Adidas' new tech-infused store in New York City
. Retrieved from https://www.engadget.com/2016/12/10/adidas-new-flagship-store-new-york-city/
American Psychological Association.
(
2010
).
2010 amendments to the 2002 “Ethical principles of psychologists and code of conduct.”
American Psychologist
,
65
(
5
),
493
.
Arbuckle
,
J. L.
(
2014
).
Amos 23.0 user's guide
.
Chicago
:
IBM SPSS
.
Baharom
,
S. N.
,
Tan
,
W. H.
, &
Idris
,
M. Z.
(
2014
).
Emotional design for games: A framweork for player-centric approach in the game design process
.
International Journal of Multimedia and Ubiquitous Engineering
,
9
(
10
),
387
398
.
Baños
,
R.M.
,
Botella
,
C.
,
Alcañiz
,
M.
,
Liaño
,
V.
,
Guerrero
,
B.
, &
Rey
,
B.
(
2004
).
Immersion and emotion: Their impact on the sense of presence
.
CyberPsychology & Behavior
,
7
(
6
),
734
741
.
Baños
,
R.M.
,
Botella
,
C.
,
Rubió
,
I.
,
Quero
,
S.
,
García-Palacios
,
A.
, &
Alcañiz
,
M.
(
2008
).
Presence and emotions in virtual environments: The influence of stereoscopy
.
CyberPsychology & Behavior
,
11
(
1
),
1
8
.
Baños
,
R.M.
,
Etchemendy
,
E.
,
Farfallini
,
L.
,
García-Palacios
,
A.
,
Quero
,
S.
, &
Botella
,
C.
(
2014
).
EARTH of well-being system: A pilot study of an information and communication technology-based positive psychology intervention
.
The Journal of Positive Psychology
,
9
(
6
),
482
488
.
Batson
,
C. D.
,
Shaw
,
L. L.
, &
Oleson
,
K. C.
(
1992
). Differentiating affect, mood, and emotion: Toward functionally based conceptual distinctions. In
M. S.
Clarck
(Ed.),
Review of personality and social psychology, No. 13. Emotion
(pp.
294
326
).
Thousand Oaks, CA
:
Sage Publications, Inc.
Beedie
,
C.
,
Terry
,
P.
, &
Lane
,
A.
(
2005
).
Distinctions between emotion and mood
.
Cognition & Emotion
,
19
(
6
),
847
878
.
Bodenhausen
,
G. V
,
Sheppard
,
L. A.
, &
Kramer
,
G. P.
(
1994
).
Negative affect and social judgment: The differential impact of anger and sadness.
European Journal of Social Psychology
,
24
(
1
),
45
62
.
Botella
,
C.
,
Garcia-Palacios
,
A.
,
Baños
,
R. M.
, &
Quero
,
S.
(
2009
).
Cybertherapy: Advantages, limitations, and ethical issues.
PsychNology Journal
,
7
(
1
),
77
100
.
Bouchard
,
S.
,
Robillard
,
G.
,
St-Jacques
,
J.
,
Dumoulin
,
S.
,
Patry
,
M. J.
, &
Renaud
,
P.
(
2004
).
Reliability and validity of a single-item measure of presence in VR
.
Proceedings of the Second International Conference on Creating, Connecting and Collaborating through Computing
,
59
61
.
Bouchard
,
S.
,
St-Jacques
,
J.
,
Robillard
,
G.
, &
Renaud
,
P.
(
2008
).
Anxiety increases the feeling of presence in virtual reality.
Presence: Teleoperators and Virtual Environments
,
17
(
4
),
376
391
. Retrieved from https://doi.org/10.1162/pres.17.4.376
Boucsein
W.
(
1992
)
Electrodermal activity
.
New York
:
Plenum University Press
.
Bowman
,
D. A.
,
Gabbard
,
J. L.
, &
Hix
,
D.
(
2002
).
A survey of usability evaluation in virtual environments: Classification and comparison of methods.
Presence: Teleoperators and Virtual Environments
,
11
(
4
),
404
424
.
Bozdogan
,
H.
(
1987
).
Model selection and Akaike's Information Criterion (AIC): The general theory and its analytical extensions.
Psychometrika
,
52
(
3
),
345
370
.
Bystrom
,
K.-E.
,
Barfield
,
W.
, &
Hendrix
,
C.
(
1999
).
A conceptual model of the sense of presence in virtual environments.
Presence: Teleoperators and Virtual Environments
,
8
(
2
),
241
244
.
Cabin
,
R. J.
, &
Mitchell
,
R. J.
(
2000
). To Bonferroni or not to Bonferroni: When and how are the questions.
Bulletin of the Ecological Society of America
.
Wiley Ecological Society of America
.
Carrozzino
,
M.
, &
Bergamasco
,
M.
(
2010
).
Beyond virtual museums: Experiencing immersive virtual reality in real museums.
Journal of Cultural Heritage
,
11
(
4
),
452
458
.
Chanel
,
G.
,
Kronegg
,
J.
,
Grandjean
,
D.
, &
Pun
,
T.
(
2006
). Emotion assessment: Arousal evaluation using EEG's and peripheral physiological signals (pp.
530
537
).
Berlin, Heidelberg
:
Springer
.
Chirico
,
A.
,
Cipresso
,
P.
,
Yaden
,
D. B.
,
Biassoni
,
F.
,
Riva
,
G.
, &
Gaggioli
,
A.
(
2017
).
Effectiveness of immersive videos in inducing awe: An experimental study.
Scientific Reports
,
7
(
1
),
1218
.
Chirico
,
A.
,
Ferrise
,
F.
,
Cordella
,
L.
, &
Gaggioli
,
A.
(
2018
).
Designing awe in virtual reality: An experimental study
.
Frontiers in Psychology
,
8
,
2351
.
Cronbach
,
L. J.
(
1951
).
Coefficient alpha and the internal structure of tests.
Psychometrika
,
16
(
3
),
297
334
.
Davis
,
F. D.
(
1989
).
Perceived usefulness, perceived ease of use, and user acceptance of information technology.
Management Information Systems Quarterly
,
13
(
3
),
319
340
.
Desmet
,
P. M.
,
Porcelijn
,
R.
, &
Dijk
,
M. B.
(
2007
).
Emotional design: Application of a research-based design approach.
Knowledge, Technology & Policy
,
20
(
3
),
141
155
.
Diemer
,
J.
,
Alpers
,
G. W.
,
Peperkorn
,
H. M.
,
Shiban
,
Y.
, &
Mühlberger
,
A.
(
2015
).
The impact of perception and presence on emotional reactions: A review of research in virtual reality.
Frontiers in Psychology
,
6
(
JAN
),
1
9
.
Diener
,
E.
(
2000
).
Subjective well-being: The science of happiness and a proposal for a national index.
American Psychologist
,
55
(
1
),
34
43
.
Dolbier
,
C. L.
,
Webster
,
J. A.
,
McCalister
,
K. T.
,
Mallon
,
M. W.
, &
Steinhardt
,
M. A.
(
2005
).
Reliability and validity of a single-item measure of job satisfaction.
American Journal of Health Promotion
,
19
(
3
),
194
198
.
Ekman
,
P.
, &
Davidson
,
R. J.
(Eds). (
1994
).
Series in affective science. The nature of emotion: Fundamental questions
.
New York
:
Oxford University Press
.
Ekman
,
P.
(
1992
).
An argument for basic emotions.
Cognition and Emotion
,
6
(
3
4
),
169
200
.
Estupiñán
,
S.
,
Rebelo
,
F.
,
Noriega
,
P.
,
Ferreira
,
C.
, &
Duarte
,
E.
(
2014
).
Can virtual reality increase emotional responses (arousal and valence)? A pilot study.
Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
,
Vol. 8518
,
541
549
.
Felnhofer
,
A.
,
Kothgassner
,
O. D.
,
Hauk
,
N.
,
Beutl
,
L.
,
Hlavacs
,
H.
, &
Kryspin-Exner
,
I.
(
2014
).
Physical and social presence in collaborative virtual environments: Exploring age and gender differences with respect to empathy
.
Computers in Human Behavior
,
31
,
272
279
.
Felnhofer
,
A.
,
Kothgassner
,
O. D.
,
Schmidt
,
M.
,
Heinzle
,
A.-K.
,
Beutl
,
L.
,
Hlavacs
,
H.
, &
Kryspin-Exner
,
I.
(
2015
).
Is virtual reality emotionally arousing? Investigating five emotion inducing virtual park scenarios
.
International Journal of Human–Computer Studies
,
82
,
48
56
.
Foa
,
E. B.
, &
Kozak
,
M. J.
(
1986
).
Emotional processing of fear: Exposure to corrective information.
Psychological Bulletin
,
99
(
1
),
20
35
.
Forgas
,
J. P.
(
1995
).
Mood and judgment: The affect infusion model (AIM).
Psychological Bulletin
,
117
(
1
),
39
66
.
Fredrickson
,
B L.
(
2001
).
The role of positive emotions in positive psychology. The broaden-and-build theory of positive emotions.
The American Psychologist
,
56
(
3
),
218
226
.
Fredrickson
,
B. L.
(
2000
).
Cultivating positive emotions to optimize health and well-being.
Prevention & Treatment
,
3
(
1
),
0001a
.
Freeman
,
D.
,
Reeve
,
S.
,
Robinson
,
A.
,
Ehlers
,
A.
,
Clark
,
D.
,
Spanlang
,
B.
, &
Slater
,
M.
(
2017
).
Virtual reality in the assessment, understanding, and treatment of mental health disorders.
Psychological Medicine
,
47
(
14
),
2393
2400
.
Freeman
,
D.
,
Garety
,
P. A.
,
Bebbington
,
P.
,
Slater
,
M.
,
Kuipers
,
E.
,
Fowler
,
D.
et al
, et al. (
2005
).
The psychology of persecutory ideation II: A virtual reality experimental study.
Journal of Nervous and Mental Disease
,
193
(
5
),
309
315
.
Frijda
,
N. H.
, &
Scherer
,
K. R.
(
2009
). Affect (psychological perspectives). In
D.
Sander
&
J. R.
Scherer
(Eds.),
The Oxford companion to emotion and the affective sciences
(p.
10
).
Oxford
:
Oxford University Press
.
Gaggioli
,
A.
,
Pallavicini
,
F.
,
Morganti
,
L.
,
Serino
,
S.
,
Scaratti
,
C.
,
Briguglio
,
M.
et al
, et al. (
2014
).
Experiential virtual scenarios with real-time monitoring (interreality) for the management of psychological stress: A block randomized controlled trial.
Journal of Medical Internet Research
,
16
(
7
),
e167
.
Garau
,
C.
, &
Ilardi
,
E.
(
2014
).
The “non-places” meet the “places”: Virtual tours on smartphones for the enhancement of cultural heritage.
Journal of Urban Technology
,
21
(
1
),
79
91
.
George
,
D.
, &
Mallery
,
M.
(
2010
).
SPSS for Windows step by step: A simple guide and reference, 17.0 update
(10th ed.).
Boston
:
Pearson
.
Gorini
,
A.
,
Griez
,
E.
,
Petrova
,
A.
, &
Riva
,
G.
(
2010
).
Assessment of the emotional responses produced by exposure to real food, virtual food and photographs of food in patients affected by eating disorders
.
Annals in General Psychiatry
,
9
,
30
.
Gorini
,
A.
,
Capideville
,
C. S.
,
De Leo
,
G.
,
Mantovani
,
F.
, &
Riva
,
G.
(
2011
).
The role of immersion and narrative in mediated presence: The virtual hospital experience.
Cyberpsychology, Behavior, and Social Networking
,
14
(
3
),
99
105
.
Gorini
,
A.
, &
Riva
,
G.
(
2008
).
The potential of virtual reality as anxiety management tool: A randomized controlled study in a sample of patients affected by Generalized Anxiety Disorder.
Trials
,
9
(
1
),
25
.
Greenwald
,
M.
(
2017
).
From storytelling to VR “storyliving”: Future marketing communications.
Retrieved from https://www.forbes.com/sites/michellegreenwald/2017/07/31/from-storytelling-to-vr-storyliving-future-marketing-communications/
Guttentag
,
D. A.
(
2010
).
Virtual reality: Applications and implications for tourism.
Tourism Management
,
31
(
5
),
637
651
.
Hair
,
J. F.
,
Black
,
W. C.
,
Babin
,
B. J.
, &
Anderson
,
R. E.
(
2010
).
Multivariate data analysis: A global perspective
(7th ed.).
Upper Saddle River, NJ
:
Pearson Education
.
Hancock
,
P. A.
,
Pepe
,
A. A.
, &
Murphy
,
L. L.
(
2005
).
Hedonomics: The power of positive and pleasurable ergonomics.
Ergonomics in Design: The Quarterly of Human Factors Applications
,
13
(
1
),
8
14
.
Hassenzahl
,
M.
, &
Tractinsky
,
N.
(
2006
).
User experience—A research agenda.
Behaviour & Information Technology
,
25
(
2
),
91
97
.
Heeter
,
C.
(
1992
).
Being there: The subjective experience of presence.
Presence: Teleoperators and Virtual Environments
,
1
(
2
),
262
271
.
Herrero
,
R.
,
García-Palacios
,
A.
,
Castilla
,
D.
,
Molinari
,
G.
, &
Botella
,
C.
(
2014
).
Virtual reality for the induction of positive emotions in the treatment of fibromyalgia: A pilot study over acceptability, satisfaction, and the effect of virtual reality on mood.
Cyberpsychology, Behavior, and Social Networking
,
17
(
6
),
379
384
.
Hoyle
,
R. H.
(
1995
).
Structural equation modeling: Concepts, issues, and applications
.
Newbury Park, CA
:
Sage
.
Hu
,
L.
, &
Bentler
,
P. M.
(
1999
).
Cutoff criteria for fit indices in covariance structure analysis: Conventional criteria versus new alternatives
.
Structural Equation Modeling
,
6
,
1
55
.
Huang
,
Y. C.
,
Backman
,
K. F.
,
Backman
,
S. J.
, &
Chang
,
L. L.
(
2016
).
Exploring the implications of virtual reality technology in tourism marketing: An integrated research framework.
International Journal of Tourism Research
,
18
(
2
),
116
128
.
Hudlicka
,
E.
(
2003
).
To feel or not to feel: The role of affect in human–computer interaction.
International Journal of Human–Computer Studies
,
59
(
1
2
),
1
32
.
IJsselsteijn
,
W. A.
,
de Ridder
,
H.
,
Freeman
,
J.
, &
Avons
,
S. E.
(
2000
).
Presence: Concept, determinants, and measurement
.
International Society for Optics and Photonics
,
520
529
.
Jeon
,
M.
(
2017
). Emotions and affect in human factors and human–computer interaction: Taxonomy, theories, approaches, and methods. In
Emotions and affect in human factors and human–computer interaction
(pp.
3
26
).
Amsterdam
:
Elsevier
.
Kalawsky
,
R. S.
(
2000
).
The validity of presence as a reliable human performance metric in immersive environments
.
3rd International Workshop on Presence
,
1
16
.
Kline
,
R. B.
(
2015
).
Principles and practice of structural equation modeling
.
New York
:
Guilford Press
.
Kolenikov
,
S.
, &
Bollen
,
K. A.
(
2012
).
Testing negative error variances: Is a Heywood case a symptom of misspecification?
Sociological Methods and Research
,
40
(
1
),
124
167
.
Lee
,
K. M.
(
2004
).
Presence, explicated.
Communication Theory
,
14
(
1
),
27
50
.
Lin
,
C. J.
,
Chen
,
H.-J.
,
Cheng
,
P.-Y.
, &
Sun
,
T.-L.
(
2015
).
Effects of displays on visually controlled task performance in three-dimensional virtual reality environment.
Human Factors and Ergonomics in Manufacturing & Service Industries
,
25
(
5
),
523
533
.
Littman
,
A. J.
,
White
,
E.
,
Satia
,
J. A.
,
Bowen
,
D. J.
, &
Kristal
,
A. R.
(
2006
).
Reliability and validity of 2 single-item measures of psychosocial stress.
Epidemiology
,
17
(
4
),
398
403
.
Lombard
,
M.
, &
Ditton
,
T.
(
1997
).
At the heart of it all: The concept of presence.
Journal of Computer-Mediated Communication
,
3
(
2
). doi.org/10.1111/j.1083=6101=1997.tb00072.x
Mandelbaum
,
A.
(
2015
).
How companies are marketing with virtual reality
. Retrieved from https://www.chiefmarketer.com/companies=marketing=virtualreality
Maples-Keller
,
J. L.
,
Bunnell
,
B. E.
,
Kim
,
S.-J.
, &
Rothbaum
,
B. O.
(
2017
).
The use of virtual reality technology in the treatment of anxiety and other psychiatric disorders.
Harvard Review of Psychiatry
,
25
(
3
),
103
113
.
Marsh
,
H. W.
, &
Hau
,
K.-T.
(
1996
).
Assessing goodness of fit.
The Journal of Experimental Education
,
64
(
4
),
364
390
.
Marsh
,
T.
,
Wright
,
P.
, &
Smith
,
S.
(
2001
).
Evaluation for the design of experience in virtual environments: Modeling breakdown of interaction and illusion.
CyberPsychology & Behavior
,
4
(
2
),
225
238
.
Muzellec
,
L.
,
Lynn
,
T.
, &
Lambkin
,
M.
(
2012
).
Branding in fictional and virtual environments.
European Journal of Marketing
,
46
(
6
),
811
826
.
Nalivaiko
,
E.
,
Davis
,
S. L.
,
Blackmore
,
K. L.
,
Vakulin
,
A.
, &
Nesbitt
,
K. V.
(
2015
).
Cybersickness provoked by head-mounted display affects cutaneous vascular tone, heart rate and reaction time
.
Physiology & Behavior
,
151
,
583
590
.
Naz
,
A.
,
Kopper
,
R.
,
McMahan
,
R. P.
, &
Nadin
,
M.
(
2017
).
Emotional qualities of VR space
.
IEEE Virtual Reality
,
3
11
.
Norman
,
D.
(
2004
).
Emotional design: Why we love (or hate) everyday things
.
New York
:
Basic Books
.
Norman
,
D. A.
(
1992
).
Design principles for cognitive artifacts.
Research in Engineering Design
,
4
(
1
),
43
50
.
Pallavicini
,
F.
,
Ferrari
,
A.
,
Garcea
,
G.
,
Zanacchi
,
A.
, &
Mantovani
,
F.
(
2018
).
Effectiveness of virtual reality survival horror games for the emotional elicitation: Preliminary insights using Resident Evil 7: Biohazard
.
Universal Access in Human–Computer Interaction. Virtual, Augmented, and Intelligent Environments: 12th International Conference
,
87
101
.
Pallavicini
,
F.
,
Algeri
,
D.
,
Repetto
,
C.
,
Gorini
,
A.
, &
Riva
,
G.
(
2009
).
Biofeedback, virtual reality and mobile phones in the treatment of generalized anxiety disorder (GAD): A phase-2 controlled clinical trial.
Journal of Cyber Therapy and Rehabilitation
,
2
(
4
),
315
327
.
Pallavicini
,
F.
,
Argenton
,
L.
,
Toniazzi
,
N.
,
Aceti
,
L.
, &
Mantovani
,
F.
(
2016
).
Virtual reality applications for stress management training in the military.
Aerospace Medicine and Human Performance
,
87
(
12
),
1021
1030
.
Pallavicini
,
F.
,
Cipresso
,
P.
,
Raspelli
,
S.
,
Grassi
,
A.
,
Serino
,
S.
,
Vigna
,
C.
et al
, et al. (
2013
).
Is virtual reality always an effective stressor for exposure treatments? Some insights from a controlled trial.
BMC Psychiatry
,
13
(
1
),
52
.
Pallavicini
,
F.
,
Pepe
,
A.
, &
Minissi
,
M. E.
(
2019
).
Gaming in virtual reality: What changes in terms of usability, emotional response and sense of presence compared to non-immersive video games?
Simulation & Gaming
,
50
(
2
),
136
159
.
Pantano
,
E.
, &
Corvello
,
V.
(
2014
).
Tourists' acceptance of advanced technology-based innovations for promoting arts and culture.
International Journal of Technology Management
,
64
(
1
),
3
16
.
Parsons
,
T. D.
, &
Rizzo
,
A. A.
(
2008
).
Affective outcomes of virtual reality exposure therapy for anxiety and specific phobias: A meta-analysis.
Journal of Behavior Therapy and Experimental Psychiatry
,
39
(
3
),
250
261
.
Pharr
,
J. M.
(
2011
).
A research agenda for investigating the effectiveness of branding in virtual reality
.
Journal of Management and Marketing Research
,
8
,
1
.
Price
,
M.
,
Mehta
,
N.
,
Tone
E. B.
, &
Anderson
,
P. L.
(
2007
).
Does engagement with exposure yield better outcomes? Components of presence as a predictor of treatment response for virtual reality exposure therapy for social phobia.
Journal of Anxiety Disorders
,
25
(
6
),
763
770
.
Quesnel
,
D.
,
DiPaola
,
S.
, &
Riecke
,
B. E.
(
2017
).
Deep learning for classification of peak emotions within virtual reality systems.
Workshop on Artificial Intelligence Meets Virtual and Augmented Worlds (AIVRAR) in Conjunction with SIGGRAPH
. doi: 10.13140/RG.2.2.15443.27686
Repetto
,
C.
,
Gaggioli
,
A.
,
Pallavicini
,
F.
,
Cipresso
,
P.
,
Raspelli
,
S.
, &
Riva
,
G.
(
2011
).
Virtual reality and mobile phones in the treatment of generalized anxiety disorders: A phase-2 clinical trial
.
Personal and Ubiquitous Computing
,
253
260
.
Riva
,
G.
(
2009
).
Is presence a technology issue? Some insights from cognitive sciences.
Virtual Reality
,
13
,
159
169
.
Riva
,
G.
,
Baños
,
R. M.
,
Botella
,
C.
,
Mantovani
,
F.
, &
Gaggioli
,
A.
(
2016
).
Transforming experience: The potential of augmented reality and virtual reality for enhancing personal and clinical change
.
Frontiers in Psychiatry
,
7
,
164
.
Riva
,
G.
,
Baños
,
R. M.
,
Botella
,
C.
,
Wiederhold
,
B. K.
, &
Gaggioli
,
A.
(
2012
).
Positive technology: Using interactive technologies to promote positive functioning.
Cyberpsychology, Behavior, and Social Networking
,
15
(
2
),
69
77
.
Riva
,
G.
,
Mantovani
,
F.
,
Capideville
,
C. S.
,
Preziosa
,
A.
,
Morganti
,
F.
,
Villani
,
D.
et al
, et al. (
2007
).
Affective interactions using virtual reality: the link between presence and emotions.
CyberPsychology & Behavior
,
10
(
1
),
45
56
.
Riva
,
G.
,
Waterworth
,
J. A.
, &
Waterworth
,
E. L.
(
2004
).
The layers of presence: A bio-cultural approach to understanding presence in natural and mediated environments.
Cyberpsychology and Behavior
,
7
(
4
),
402
416
.
Riva
,
G.
,
Waterworth
,
J. A.
,
Waterworth
,
E. L.
, &
Mantovani
,
F.
(
2011
).
From intention to action: The role of presence.
New Ideas in Psychology
,
29
(
1
),
24
37
.
Rizzo
,
A.
,
Parsons
,
T. D.
,
Lange
,
B.
,
Kenny
,
P.
,
Buckwalter
,
J. G.
,
Rothbaum
,
B.
et al
, et al. (
2011
).
Virtual reality goes to war: A brief review of the future of military behavioral healthcare.
Journal of Clinical Psychology in Medical Settings
,
18
(
2
),
176
187
.
Robillard
,
G.
,
Bouchard
,
S.
,
Fournier
,
T.
, &
Renaud
,
P.
(
2003
).
Anxiety and presence during VR immersion: A comparative study of the reactions of phobic and non-phobic participants in therapeutic virtual environments derived from computer games.
CyberPsychology & Behavior
,
6
(
5
),
467
476
.
Russell
,
J. A.
(
1980
).
A circumplex model of affect.
Journal of Personality and Social Psychology
,
39
(
6
),
1161
1178
.
Schermelleh-Engel
,
K.
,
Moosbrugger
,
H.
, &
Müller
,
H.
(
2003
).
Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures.
Methods of Psychological Research Online
,
8
(
2
),
23
74
.
Schubert
,
T.
,
Friedmann
,
F.
, &
Regenbrecht
,
H.
(
2001
).
The experience of presence: Factor analytic insights.
Presence: Teleoperators and Virtual Environments
,
10
(
3
),
266
281
.
Serino
,
S.
,
Triberti
,
S.
,
Villani
,
D.
,
Cipresso
,
P.
,
Gaggioli
,
A.
, &
Riva
,
G.
(
2014
).
Toward a validation of cyber-interventions for stress disorders based on stress inoculation training: A systematic review.
Virtual Reality
,
18
(
1
),
73
87
.
Shelstad
,
W. J.
,
Smith
,
D. C.
, &
Chaparro
,
B. S.
(
2017
).
Gaming on the rift: How virtual reality affects game user satisfaction.
Proceedings of the Human Factors and Ergonomics Society Annual Meeting
,
61
(
1
),
2072
2076
.
Skrondal
,
A.
, &
Rabe-Hesketh
,
S.
(
2004
).
Generalized latent variable modeling: Multilevel, longitudinal, and structural equation models
.
Boca Raton, FL
:
Chapman and Hall/CRC
.
Slater
,
M.
,
Khanna
,
P.
,
Mortensen
,
J.
, &
Yu
,
I.
(
2009
).
Visual realism enhances realistic response in an immersive virtual environment
.
IEEE Computer Graphic Application
,
76
84
.
Slater
,
M.
(
2003
).
A note on presence terminology.
Presence Connect
,
3
(
3
),
1
5
.
Slater
,
M.
(
2004
).
Presence and emotions.
CyberPsychology & Behavior
,
7
(
1
),
121
.
Slater
,
M.
,
Linakis
,
V.
,
Usoh
,
M.
, &
Kooper
,
R.
(
1996
).
Immersion, presence, and performance in virtual environments: An experiment with tri-dimensional chess
.
Proceedings of the 3rd ACM Symposium on Virtual Reality Software and Technology
,
163
172
.
Slater
,
M.
, &
Steed
,
A.
(
2000
).
A virtual presence counter.
Presence: Teleoperators and Virtual Environments
,
9
(
5
),
413
434
.
Slater
,
M.
,
Usoh
,
M.
, &
Steed
,
A.
(
1994
).
Depth of presence in virtual environments.
Presence: Teleoperators and Virtual Environments
,
3
(
2
),
130
144
.
Steed
,
A.
,
Vinayagamoorthy
,
V.
, &
Brogni
,
A.
(
2005
).
Using breaks in presence to identify usability issues.
HCI International 2005, the 11th International Conference on Human Computer Interaction
.
Stott
,
J.
(
2016
).
Virtual reality: Content marketing's next big trend.
Retrived from https://econsultancy.com/virtual-reality-content-marketing-s-next-big-trend/
Sutcliffe
,
A. G.
, &
Kaur
,
K. D.
(
2000
).
Evaluating the usability of virtual reality user interfaces.
Behaviour & Information Technology
,
19
(
6
),
415
426
.
Thüring
,
M.
, &
Mahlke
,
S.
(
2007
).
Usability, aesthetics and emotions in human–technology interaction.
International Journal of Psychology
,
42
(
4
),
253
264
.
Tofighi
,
D.
, &
MacKinnon
,
D. P.
(
2016
).
Monte Carlo confidence intervals for complex functions of indirect effects.
Structural Equation Modeling
,
23
(
2
),
194
205
.
Triberti
,
S.
,
Chirico
,
A.
,
La Rocca
,
G.
, &
Riva
,
G.
(
2017
).
Developing emotional design: Emotions as cognitive processes and their role in the design of interactive technologies
.
Frontiers in Psychology
,
8
,
1773
.
Villani
,
D.
,
Repetto
,
C.
,
Cipresso
,
P.
, &
Riva
,
G.
(
2012
).
May I experience more presence in doing the same thing in virtual reality than in reality? An answer from a simulated job interview.
Interacting with Computers
,
24
(
4
),
265
272
.
Villani
,
D.
, &
Riva
,
G.
(
2012
).
Does interactive media enhance the management of stress? Suggestions from a controlled study.
Cyberpsychology Behavior and Social Networking
,
15
(
1
),
24
30
.
Villani
,
D.
,
Lucchetta
,
M.
,
Preziosa
,
A.
, &
Riva
,
G. M.
(
2009
).
The role of interactive media features on the affective response: A virtual reality study
.
E-Minds. International Journal on Human Computer Interaction
,
1
,
1
35
.
Walker
,
M.
,
Burton
,
B.
, &
Michele Cantara.
M.
(
2016
).
Hype cycle for emerging technologies 2016
.
Stamford, CT
:
Gartner
.
Willaert
,
W. I. M.
,
Aggarwal
,
R.
,
Van Herzeele
,
I.
,
Cheshire
,
N. J.
, &
Vermassen
,
F. E.
(
2012
).
Recent advancements in medical simulation: Patient-specific virtual reality simulation.
World Journal of Surgery
,
36
(
7
),
1703
1712
.
Winograd
,
T.
, &
Flores
,
F.
(
1987
).
Understanding computers and cognition: A new foundation for design
.
Reading, MA
:
Addison-Wesley Longman
.
Witmer
,
B. G.
, &
Singer
,
M. J.
(
1998
).
Measuring presence in virtual environments: A presence questionnaire.
Presence: Teleoperators and Virtual Environments
,
7
(
3
),
225
240
.
Xu
,
Y.
,
Park
,
H.
, &
Baek
,
Y.
(
2011
).
A new approach toward digital storytelling: An activity focused on writing self-efficacy in a virtual learning environment.
Educational Technology & Society
,
14
,
181
191
.
Zendejas
,
B.
,
Brydges
,
R.
,
Wang
,
A. T.
, &
Cook
,
D. A.
(
2013
).
Patient outcomes in simulation-based medical education: A systematic review.
Journal of General Internal Medicine
,
28
(
8
),
1078
1089
.