We propose a new video conferencing system that uses an array of cameras to capture a remote user and then show the video of that person on a spherical display. This telepresence system has two key advantages: (i) it can capture a near-correct image for any potential observer viewing direction because the cameras surround the user horizontally; and (ii) with view-dependent graphical representation on the spherical display, it is possible to tell where the remote user is looking from any viewpoint, whereas flat displays are visible only from the front. As a result, the display can more faithfully represent the gaze of the remote user. We evaluate this system by measuring the ability of observers to accurately judge which targets the actor is gazing at in two experiments. Results from the first experiment demonstrate the effectiveness of the camera array and spherical display system, in that it allows observers at multiple observing positions to accurately tell at which targets the remote user is looking. The second experiment further compared a spherical display with a planar display and provided detailed reasons for the improvement of our system in conveying gaze. We found two linear models for predicting the distortion introduced by misalignment of capturing cameras and the observer's viewing angles in video conferencing systems. Those models might be able to enable a correction for this distortion in future display configurations.

Users of immersive virtual reality (VR) are often observed to act realistically on social, behavioral, physiological, and subjective levels. However, experimental studies in the field typically collect and analyze metrics independently, which fails to consider the synchronous and multimodal nature of the original human activity. This paper concerns multimodal data capture and analysis in immersive collaborative virtual environments (ICVEs) in order to enable a holistic and rich analysis based on techniques from interaction analysis. A reference architecture for collecting multimodal data specifically for immersive VR is presented. It collates multiple components of a user's nonverbal and verbal behavior in single log file, thereby preserving the temporal relationships between cues. Two case studies describing sequences of immersive avatar-mediated communication (AMC) demonstrate the ability of multimodal data to preserve a rich description of the original mediated social interaction. Analyses of the sequences using techniques from interaction analysis emphasize the causal interrelationships between the captured components of human behavior, leading to a deeper understanding of how and why the communication may have unfolded. In presenting our logging architecture, we hope that we will initiate a discussion of a logging standard that can be built by the community so that practitioners can share data and build better tools to analyze the utility of VR.

This paper presents the use of our multimodal mixed reality telecommunication system to support remote acting rehearsal. The rehearsals involved two actors, located in London and Barcelona, and a director in another location in London. This triadic audiovisual telecommunication was performed in a spatial and multimodal collaborative mixed reality environment based on the “destination-visitor” paradigm, which we define and put into use. We detail our heterogeneous system architecture, which spans the three distributed and technologically asymmetric sites, and features a range of capture, display, and transmission technologies. The actors' and director's experience of rehearsing a scene via the system are then discussed, exploring successes and failures of this heterogeneous form of telecollaboration. Overall, the common spatial frame of reference presented by the system to all parties was highly conducive to theatrical acting and directing, allowing blocking, gross gesture, and unambiguous instruction to be issued. The relative inexpressivity of the actors' embodiments was identified as the central limitation of the telecommunication, meaning that moments relying on performing and reacting to consequential facial expression and subtle gesture were less successful.

Brain–computer interfaces (BCIs) provide a novel form of human–computer interaction. The purpose of these systems is to aid disabled people by affording them the possibility of communication and environment control. In this study, we present experiments using a P300 based BCI in a fully immersive virtual environment (IVE). P300 BCIs depend on presenting several stimuli to the user. We propose two ways of embedding the stimuli in the virtual environment: one that uses 3D objects as targets, and a second that uses a virtual overlay. Both ways have been shown to work effectively with no significant difference in selection accuracy. The results suggest that P300 BCIs can be used successfully in a 3D environment, and this suggests some novel ways of using BCIs in real world environments.

We have set up a brain-computer interface (BCI) to be used as an input device to a highly immersive virtual reality CAVE-like system. We have carried out two navigation experiments: three subjects were required to rotate in a virtual bar room by imagining left or right hand movement, and to walk along a single axis in a virtual street by imagining foot or hand movement. In this paper we focus on the subjective experience of navigating virtual reality “by thought,” and on the interrelations between BCI and presence.

Healthy participants are able to move forward within a virtual environment (VE) by the imagination of foot movement. This is achieved by using a brain-computer interface (BCI) that transforms thought-modulated electroencephalogram (EEG) recordings into a control signal. A BCI establishes a communication channel between the human brain and the computer. The basic principle of the Graz-BCI is the detection and classification of motor-imagery-related EEG patterns, whereby the dynamics of sensorimotor rhythms are analyzed. A BCI is a closed-loop system and information is visually fed back to the user about the success or failure of an intended movement imagination. Feedback can be realized in different ways, from a simple moving bar graph to navigation in VEs. The goals of this work are twofold: first, to show the influence of different feedback types on the same task, and second, to demonstrate that it is possible to move through a VE (e.g., a virtual street) without any muscular activity, using only the imagination of foot movement. In the presented work, data from BCI feedback displayed on a conventional monitor are compared with data from BCI feedback in VE experiments with a head-mounted display (HMD) and in a high immersive projection environment (Cave). Results of three participants are reported to demonstrate the proof-of-concept. The data indicate that the type of feedback has an influence on the task performance, but not on the BCI classification accuracy. The participants achieved their best performances viewing feedback in the Cave. Furthermore the VE feedback provided motivation for the subjects.

Presence research relies heavily on empirical experiments involving subjects in mediated environments. Since presence is a complex, multidimensional concept, experiments on presence can be extremely resource intensive and produce large amounts of data of different types. As the presence community matures, we would like to suggest that data collected in experiments be made publicly available to the community. This will allow the verification of experimental results, comparison of results of experiments carried out in different laboratories, and evaluation of new data-analysis methods. This will, eventually, lead to consistency in approaches and increased confidence in results. In this paper we present the complete dataset from a large-scale experiment that we have carried out in highly immersive virtual reality. We describe the data we have gathered and give examples of the types of analysis that can be made based on that data.

We present a class of partitioning scheme that we have called frontier sets . Frontier sets build on the notion of a potentially visible set (PVS). In a PVS, a world is subdivided into cells and for each cell all the other cells that can be seen are computed. In contrast, a frontier set considers pairs of cells, A and B. For each pair, it lists two sets of cells (two frontiers), F AB and F BA . By definition, from no cell in F AB is any cell in F BA visible and vice versa. Our initial use of frontier sets has been to enable scalability in distributed networking. This is possible because, for example, if at time t 0 Player1 is in cell A and Player2 is in cell B, as long as they stay in their respective frontiers, they do not need to send update information to each other. In this paper we describe two strategies for building frontier sets. Both strategies are dynamic and compute frontiers only as necessary at runtime. The first is distance-based frontiers . This strategy requires precomputation of an enhanced potentially visible set . The second is greedy frontiers . This strategy is more expensive to compute at runtime, however it leads to larger and thus more efficient frontiers. Network simulations using code based on the Quake II engine show that frontiers have significant promise and may allow a new class of scalable peer-to-peer game infrastructures to emerge.

Virtual environments systems based on immersive projection technologies (IPTs) offer users the possibility of collaborating intuitively in a 3D environment. While considerable work has been done to examine interaction in desktop-based collaborative virtual environments (CVEs), there are currently no studies for collaborative interaction using IPTs. The aim of this paper is to examine how immersive technologies support interaction and to compare this to the experience with desktop systems. A study of collaboration is presented where two partners worked together using networked IPT environments. The data collected included observations, analysis of video and audio recordings, questionnaires and debriefing interviews from both IPT sites. This paper focuses on the successes and failures in collaboration through detailed examination of particular incidents during the interaction. We compare these successes and failures with the findings of a study by Hindmarsh, Fraser, Heath, & Benford (Computer Supported Collaborative Work, CSCW'98, 1998, pp. 217–226) that examined object-focused interaction on a desktop-based CVE system. Our findings identify situations where interaction is better supported with the IPT system than the desktop system, and situations where interaction is not as well supported. We also present examples of how social interaction is critical to seamless collaboration.

This paper presents the results of a formal experiment to compare different interaction techniques across two types of immersive display: an immersive projection technology (IPT) and a head-mounted display (HMD). Our aim is to investigate the effectiveness of two widely used interaction metaphors, virtual hand and ray casting, on these two display technologies. Our motivation is that design and evaluation of interaction techniques for immersive egocentric display systems has been undertaken almost exclusively on HMDs. We argue that basing interaction for IPTs on techniques developed for other types of immersive systems is a flawed approach, as there are some categorical differences between the experience given by an IPT and an HMD. For example, an IPT user has a much wider field of view than an HMD user. We have chosen two types of interaction tasks to study: simple selection of objects both near to and at some distance from the user, and manipulation of objects involving a change of both position and orientation. As previous studies have found, we find that ray casting is preferable for selection and virtual hand is preferable for manipulation for a HMD. We show that this is also the case for the IPT. More interestingly, while we find performance on selection tasks is much better on the IPT, for manipulation tasks there is little difference between the two display technologies.

Many tasks require teamwork. Team members may work concurrently, but there must be some occasions of coming together. Collaborative virtual environments (CVEs) allow distributed teams to come together across distance to share a task. Studies of CVE systems have tended to focus on the sense of presence or copresence with other people. They have avoided studying close interaction between us-ers, such as the shared manipulation of objects, because CVEs suffer from inherent network delays and often have cumbersome user interfaces. Little is known about the ef-fectiveness of collaboration in tasks requiring various forms of object sharing and, in particular, the concurrent manipu-lation of objects. This paper investigates the effectiveness of supporting teamwork among a geographically distributed group in a task that requires the shared manipulation of objects. To complete the task, users must share objects through con-current manipulation of both the same and distinct at-tributes. The effectiveness of teamwork is measured in terms of time taken to achieve each step, as well as the impression of users. The effect of interface is examined by comparing various combinations of walk-in cubic immersive projection technology (IPT) displays and desktop devices.

This paper presents results of the longitudinal usability and network trials that took place throughout the COVEN (COllaborative Virtual ENvironments) Project. To address the lack of understanding about usability design and evaluation for collaborative virtual environments (CVEs), a deductive analysis was used to systematically identify areas of inquiry. We present a summary of the analysis and the resulting framework through which various complementary methods were utilized during our studies. The objective of these studies was to gain a better understanding about design, usability, and utility for CVEs in a multidisciplinary setting. During the studies, which span four years, we undertook longitudinal studies of user behavior and computational demands during network trials, usability inspections of each iteration of the project demonstrators, consumer evaluations to assess social acceptability and utility of our demonstrators, and continuous preparations of design guidelines for future developers of CVEs. In this paper, we discuss the need for such activities, give an overview of our development of methods and adaptation of existing methods, give a number of explanatory examples, and review the future requirements in this area.

Collaborative virtual environments (CVEs) are a promising technology enabling remote participants to share a common place through three-dimensional graphical scenes. Within the COVEN project (Normand, 1999), we have run prolonged series of Internet trials that have allowed us to gather valuable data to formulate usability guidelines and networking requirements. However, running such trials in a real setting and making sure that the application and networking infrastructures will be stable enough is still a challenge. In this paper, we describe some of our experiences, together with the technical choices that have permitted many hours of successful Internet trials. We also make a thorough analysis of different correlated logging data. This analysis allows us to propose and confirm a model of a CVE application's network behavior, together with a number of interesting results that disprove some common assumptions. Furthermore, we use the model and the logging data to highlight the benefits of IP multicasting and for predicting traffic behaviors and bandwidth use on top of different logical network topologies.

A central aim of the COVEN project was to prototype large-scale applications of collaborative virtual environments (CVEs) that went beyond the existing state of the art. These applications were used in a series of real-scale networked trials that allowed us to gather many interesting human and technological results. To fulfill the technological and experimental goals of the project, we have modified an existing CVE platform: the DIVE (distributed interactive virtual environment) toolkit. In this paper, we present the different services and extensions that have been implemented within the platform during the four years of the project. Such a presentation will exemplify the different features that will have to be offered by nextgeneration CVE platforms. Implementation of the COVEN services has had implications at all levels of the platform: from a new networking layer through to mechanisms for high-level semantic modeling of applications.

This paper describes a new measure for presence in immersive virtual environments (VEs) that is based on data that can be unobtrusively obtained during the course of a VE experience. At different times during an experience, a participant will occasionally switch between interpreting the totality of sensory inputs as forming the VE or the real world. The number of transitions from virtual to real is counted, and, using some simplifying assumptions, a probabilistic Markov chain model can be constructed to model these transitions. This model can be used to estimate the equilibrium probability of being “present” in the VE. This technique was applied in the context of an experiment to assess the relationship between presence and body movement in an immersive VE. The movement was that required by subjects to reach out and touch successive pieces on a three-dimensional chess board. The experiment included twenty subjects, ten of whom had to reach out to touch the chess pieces (the active group) and ten of whom only had to click a handheld mouse button (the control group). The results revealed a significant positive association in the active group between body movement and presence. The results lend support to interaction paradigms that are based on maximizing the match between sensory data and proprioception.