Mixed Realities

Our current work is concerned with the different properties of boundaries and their effects. The issue is of interest as the characteristics of the deployed mixed reality boundary determine the possibilities for awareness and communication between the participants in the joined spaces.

We explore an approach to creating shared mixed realities based on the construction of boundaries between real and virtual spaces. This work has been motivated by the following factors:

1. Own experience of developing and using collaborative virtual environments

The MASSIVE VR system [Greenhalgh95] has been used to support a series of meetings over wide area network. Several papers have discussed the results of these trials including [Bowers96a] and [Bowers96b]. Both of these papers emphasise the significant effect that events and interactions occurring in the real world have on those in the virtual world. The need for awareness of the physical becomes even more pertinent when the participants within the virtual world actually wish to manipulate and refer to objects within their local physical environment. The message seems to be that further consideration needs to be given to how to relate events in the physical world to those in the virtual.

2. Recent developments in the field of mixed reality

Recent research in the field of augmented reality suggests new ways of mixing the real and virtual. Milgram and Kishino introduce the idea of a virtuality continuum whose extremes are characterised by real and virtual environments and at whose center the technology of augmented reality is found [Milgram94]. These ideas led to the consideration of how the concept of mixing reality can be applied to shared spaces and offer a potential solution to the problem raised in 1.

Mixed Reality Boundaries

Mixed reality boundaries are a way of joining a physical and a synthetic space [Benford96, Benford99]. As such they can be used to construct a new kind of collaborative environment, supporting new forms of awareness and communication between the inhabitants of distributed spaces. Figure 1 shows how a physical space can be linked to a synthetic one through the creation of a simple boundary. The method is based on a combination of video projection into the physical space and video texturing into the virtual one. That is, the changing geometry of the synthetic space and the avatars in it are transmitted across the network, rendered and then projected into the physical space. At the same time a live video image of the physical space is transmitted across the network and then displayed in the synthetic space through a process of dynamically texture mapping the incoming frames so that it appears as an integrated part of the virtual environment. Consequently, the inhabitants of the physical space would see the synthetic space as an extension of their physical environment an vise versa. Provided with an additional audio link between the two spaces, the inhabitants of each are able to communicate with one another across the boundary.

The resulting mixed reality and its application is quite different from that proposed by Milgram and Kishino . They refer to mixed reality as "the merging of real and virtual worlds" such that "real world and virtual world objects are present together within a single display" [Milgram94]. This approach is suited to a range of applications which involve fine grained interaction with physical objects and their virtual analogues, such as medical training, tele-surgery, machine maintenance and many others.

Our approach, on the other hand, does not result in the single syntesised space, rather the mixed reality boundary transmits information between the physical and synthetic environments to give the sense that they are adjacent but distinct parts of a combined space. This technique creates the possibility for new types of applications that facilitate cooperative activity between participants in shared virtual and physical spaces. It is particularly useful where knowledge of events in the real world needs to be combined with distributed access to electronic data. Another powerful and distinguishing feature of our boundary-based approach to mixed reality is the ability to join together many physical and synthetic spaces into an integrated whole, a so called "tessellated mixed reality. This idea represents a key direction in future research.

In the long term the two approaches of augmented reality and mixed reality boundaries may prove to be complimentary, one focusing on the "micro-level" issues of merging two specific spaces and the other addressing the "macro-level" issues of building large mixed reality structures.

The Properties of Mixed Reality Boundaries

We introduces a set of properties that allow mixed reality boundaries to be configured for different cooperative activity. Our choice of properties has been influenced by analogies with the real-world boundaries that partition physical space and also by our previous work on developing boundaries within virtual space [Benford99]. Our proposed properties are grouped into the three general categories of permeability, situation, and dynamics [Koleva98]. We also include the two meta-properties of symmetry and representation that apply to the others.

Permeability describes how the boundary affects sensory information passing between the linked spaces. It can be broken down into the following components:

• Visibility - considers what visual information is permitted through the boundary and consists of two components: visual resolution and field of view. Visual resolution concerns the amount of visual information obtained through the boundary. Field of view describes the volume of the connected space that is made visible through the boundary.
• Audibility - considers what audio information is permitted through the boundary.
• Solidity - refers to the ability to traverse the boundary. This includes metaphorically extending a limb through the boundary and manipulating a remote object; pushing an object through the boundary so that it becomes available on the other side; and stepping through the boundary and assuming control of an avatar on the other side.

The potential for combining different visibility and audibility effects with varying degrees of solidity allows the definition of a wide range of boundary types. These include analogies of familiar everyday physical boundaries such as windows, walls, curtains, fences, one-way mirrors and even lines on the ground, as well as new kinds of boundary that have no common physical counterpart. By exploring all the possible combinations of permeability properties, we might eventually identify the fundamental building materials that can be used to join together physical and virtual spaces.

Situation concerns the spatial relationships between the mixed reality boundary, the physical and virtual spaces that it connects and the participants and objects that these contain. This category includes:

• Location - describes the placement of the boundary within the connected spaces (e.g. vertical vs horizontal).
• Alignment - concerns the orientation of the boundary with respect to the different participants and objects and may establish different possibilities for turn taking and access to other participants.
• Mobility - describes whether the boundary assumes a static situation, thus offering connection between two fixed sections of the linked spaces, or whether the boundary can join different parts of the spaces over time.
• Segmentation - describes whether the boundary is made up of on or more segments (fragmentation can be in terms of boundary properties and/or spatial location).
• Spatial consistency - describes how the spatial co-ordinate system of one space is related to that of the other.

Between them, these properties determine the spatial understanding that the participants in one space have of the connected space.

Dynamics concern the temporal properties of the boundary, including its lifetime and its degree of configurability:

• Lifetime - refers to when and for how long the boundary is in existence.
• Configurability - describes how dynamically the various boundary properties can be changed.

Symmetry refers to the extent that the properties of a mixed reality boundary are configured to be the same on both of its sides (i.e., from the physical to the virtual and vice versa).

Representation describes the how visible are the properties of the boundary to participants and what means of representation are used (e.g. explicitly through controls and labels, or implicitly through metaphor, interior design or architecture).

References

[Benford96] Benford, S., Brown, C., Reynard, G. and Greenhalgh, C., Shared Spaces: Transportation, Artificiality and Spatiality, Proc. CSCW'96 Boston, Massachusetts, ACM Press, 16-20 November, 1996, pp 77-85

[Benford99] Benford, S., Greenhalgh, C., Reynard, G., Brown, C. and Koleva, B., Understanding and Constructing Shared Spaces with Mixed Reality Boundaries, Transactions on Computer Human Interactions (TOCHI), ACM Press, issue?

[Bowers96a] Bowers, J., Pycock, J. and O'Brian, J., Talk and Embodiment in Collaborative Virtual Environments, Proc. ACM CHI'96, ACM Press, 1996

[Bowers96b] Bowers, J., O'Brian, J. and Pycock, J., Practically Accomplishing Immersion, Cooperation in and for Virtual Environments, Proc. ACM Conference on Computer Supported Cooperative Work (CSCW'96), Boston, ACM Press, Vol 16-20, 1996, pp. 380-389

[Greenhalgh95] Greenhalgh, C. and Benford, S., MASSIVE: A Virtual Reality System for Tele-conferencing, ACM Transactions on Computer Human Interaction (TOCHI), ACM Press, 1995

[Koleva98] Koleva, B. and Benford, S., The Properties of Mixed Reality Boundaries, 1998, Report No. NOTTCS-R-98-10

[Milgram94] Milgram, P. and Kishino, F., A Taxonomy of Mixed Reality Visual Displays, IEICE Transactions on Information Systems, Vol E77-D, No 12, December 1994

Links

The following are links to related pages:

People

Boriana Koleva

Rob Clarke

Gail Reynard

Professor Steve Benford

Example Projects

The Internet Foyer

The Internet Foyer is an example of a mixed reality - an electronic meeting space which spans the physical and virtual worlds. It links a visualisation of an organisation's World Wide Web home pages in a CVE to its physical reception area so as to create a combined entry point into its physical and electronic manifestations.

The Mixed Reality Performance

In the Mixed Reality performance a poet appeared on a virtual sage and performed to an audience that were located in a remote physical theatre. The awareness and communication between the performer and the audience were entirely facilitated though a mixed reality boundary.

Publications

• Benford, S., Brown, C., Reynard, G. and Greenhalgh, C., Shared Spaces: Transportation, Artificiality and Spatiality, Proc. CSCW'96, Boston, Massachusetts, ACM Press, 16-20 November, 1996, pp 77-86, ftp://ftp.crg.cs.nott.ac.uk/pub/papers/CSCW96.ps.gz

• Koleva, B. and Benford, S., Theory and Application of Mixed Reality Boundaries, Proc. UK-VRSIG 98, http://www.dcs.ex.ac.uk/ukvrsig98/pap2_01.htm

• Benford, S., Greenhalgh, C., Reynard, G., Brown, C. and Koleva, B., Understanding and Constructing Shared Spaces with Mixed Reality Boundaries, ACM Transactions on Computer Human Interfaces (TOCHI), ACM Press, (in press, to appear in 1999)

---

CRG Technologies
CRG Research
Communications Research Group
Department of Computer Science
University of Nottingham

---

URL:

http://www.crg.cs.nott.ac.uk/research/technologies/mixed

Author:

Boriana Koleva (bnk@cs.nott.ac.uk - Department of Computer Science)

Created:

19 November 1997

Last-modified:

4 January 1999