Driven by concerns of supporting new forms of awareness and communication between the inhabitants of distributed environments, we develop the approach of creating boundaries between different kinds of space. In particular we focus on the relationship between physical and virtual space and the concept of mixed reality boundaries as a way of joining the two to produce new types of collaborative environment. The approach is based on the combined use of video projection in a physical space and video texturing in a synthetic one.
Our current work is concerned with the different properties of boundaries and their effects. The issue is of interest as the characteristics of the deployed mixed reality boundary determine the possibilities for awareness and communication between the participants in the joined spaces.
We explore an approach to creating shared mixed realities based on the construction of boundaries between real and virtual spaces. This work has been motivated by the following factors:
1. Own experience of developing and using collaborative virtual environments
The MASSIVE VR system [Greenhalgh95] has been used to support a series of meetings over wide area network. Several papers have discussed the results of these trials including [Bowers96a] and [Bowers96b]. Both of these papers emphasise the significant effect that events and interactions occurring in the real world have on those in the virtual world. The need for awareness of the physical becomes even more pertinent when the participants within the virtual world actually wish to manipulate and refer to objects within their local physical environment. The message seems to be that further consideration needs to be given to how to relate events in the physical world to those in the virtual.
2. Recent developments in the field of mixed reality
Recent research in the field of augmented reality suggests new ways of mixing the real and virtual. Milgram and Kishino introduce the idea of a virtuality continuum whose extremes are characterised by real and virtual environments and at whose center the technology of augmented reality is found [Milgram94]. These ideas led to the consideration of how the concept of mixing reality can be applied to shared spaces and offer a potential solution to the problem raised in 1.
Mixed Reality Boundaries
Mixed reality boundaries are a way of joining a physical and a synthetic space [Benford96, Benford99]. As such they can be used to construct a new kind of collaborative environment, supporting new forms of awareness and communication between the inhabitants of distributed spaces. Figure 1 shows how a physical space can be linked to a synthetic one through the creation of a simple boundary. The method is based on a combination of video projection into the physical space and video texturing into the virtual one. That is, the changing geometry of the synthetic space and the avatars in it are transmitted across the network, rendered and then projected into the physical space. At the same time a live video image of the physical space is transmitted across the network and then displayed in the synthetic space through a process of dynamically texture mapping the incoming frames so that it appears as an integrated part of the virtual environment. Consequently, the inhabitants of the physical space would see the synthetic space as an extension of their physical environment an vise versa. Provided with an additional audio link between the two spaces, the inhabitants of each are able to communicate with one another across the boundary.
Figure 1: a simple boundary between a physical and a synthetic space
The resulting mixed reality and its application is quite different from that proposed by Milgram and Kishino . They refer to mixed reality as "the merging of real and virtual worlds" such that "real world and virtual world objects are present together within a single display" [Milgram94]. This approach is suited to a range of applications which involve fine grained interaction with physical objects and their virtual analogues, such as medical training, tele-surgery, machine maintenance and many others.
Our approach, on the other hand, does not result in the single syntesised space, rather the mixed reality boundary transmits information between the physical and synthetic environments to give the sense that they are adjacent but distinct parts of a combined space. This technique creates the possibility for new types of applications that facilitate cooperative activity between participants in shared virtual and physical spaces. It is particularly useful where knowledge of events in the real world needs to be combined with distributed access to electronic data. Another powerful and distinguishing feature of our boundary-based approach to mixed reality is the ability to join together many physical and synthetic spaces into an integrated whole, a so called "tessellated mixed reality. This idea represents a key direction in future research.
In the long term the two approaches of augmented reality and mixed reality boundaries may prove to be complimentary, one focusing on the "micro-level" issues of merging two specific spaces and the other addressing the "macro-level" issues of building large mixed reality structures.
The Properties of Mixed Reality Boundaries
We introduces a set of properties that allow mixed reality boundaries to be configured for different cooperative activity. Our choice of properties has been influenced by analogies with the real-world boundaries that partition physical space and also by our previous work on developing boundaries within virtual space [Benford99]. Our proposed properties are grouped into the three general categories of permeability, situation, and dynamics [Koleva98]. We also include the two meta-properties of symmetry and representation that apply to the others.
Permeability describes how the boundary affects sensory information passing between the linked spaces. It can be broken down into the following components:
The potential for combining different visibility and audibility effects with varying degrees of solidity allows the definition of a wide range of boundary types. These include analogies of familiar everyday physical boundaries such as windows, walls, curtains, fences, one-way mirrors and even lines on the ground, as well as new kinds of boundary that have no common physical counterpart. By exploring all the possible combinations of permeability properties, we might eventually identify the fundamental building materials that can be used to join together physical and virtual spaces.
Situation concerns the spatial relationships between the mixed reality boundary, the physical and virtual spaces that it connects and the participants and objects that these contain. This category includes:
Between them, these properties determine the spatial understanding that the participants in one space have of the connected space.
Dynamics concern the temporal properties of the boundary, including its lifetime and its degree of configurability:
Symmetry refers to the extent that the properties of a mixed reality boundary are configured to be the same on both of its sides (i.e., from the physical to the virtual and vice versa).
Representation describes the how visible are the properties of the boundary to participants and what means of representation are used (e.g. explicitly through controls and labels, or implicitly through metaphor, interior design or architecture).
[Benford96] Benford, S., Brown, C., Reynard, G. and Greenhalgh, C., Shared Spaces: Transportation, Artificiality and Spatiality, Proc. CSCW'96 Boston, Massachusetts, ACM Press, 16-20 November, 1996, pp 77-85
[Benford99] Benford, S., Greenhalgh, C., Reynard, G., Brown, C. and Koleva, B., Understanding and Constructing Shared Spaces with Mixed Reality Boundaries, Transactions on Computer Human Interactions (TOCHI), ACM Press, issue?
[Bowers96a] Bowers, J., Pycock, J. and O'Brian, J., Talk and Embodiment in Collaborative Virtual Environments, Proc. ACM CHI'96, ACM Press, 1996
[Bowers96b] Bowers, J., O'Brian, J. and Pycock, J., Practically Accomplishing Immersion, Cooperation in and for Virtual Environments, Proc. ACM Conference on Computer Supported Cooperative Work (CSCW'96), Boston, ACM Press, Vol 16-20, 1996, pp. 380-389
[Greenhalgh95] Greenhalgh, C. and Benford, S., MASSIVE: A Virtual Reality System for Tele-conferencing, ACM Transactions on Computer Human Interaction (TOCHI), ACM Press, 1995
[Koleva98] Koleva, B. and Benford, S., The Properties of Mixed Reality Boundaries, 1998, Report No. NOTTCS-R-98-10
[Milgram94] Milgram, P. and Kishino, F., A Taxonomy of Mixed Reality Visual Displays, IEICE Transactions on Information Systems, Vol E77-D, No 12, December 1994
The following are links to related pages:
The Internet Foyer is an example of a mixed reality - an electronic meeting space which spans the physical and virtual worlds. It links a visualisation of an organisation's World Wide Web home pages in a CVE to its physical reception area so as to create a combined entry point into its physical and electronic manifestations.
In the Mixed Reality performance a poet appeared on a virtual sage and performed to an audience that were located in a remote physical theatre. The awareness and communication between the performer and the audience were entirely facilitated though a mixed reality boundary.