Methodology of Usability Evaluation for Collaborative Virtual Environments

In a changing world

The computer side of HCI has not remained stable yet for any long period of time. We have moved from textual interfaces to graphical interfaces, to MultiMedia, CSCW, and immersive systems, from single-user systems to multiple-user systems. Any one technique of designing or evaluating interfaces is thus bound to come short of the full range of aspects which are involved with a particular design, system or technology.

Evaluators need to question their approach

Thus evaluators continually need to change their approach, to adapt to the new kinds of interfaces, and to question the effectiveness of the possibilities which these new technologies make available. Although the need for the development of a special tool-kit for VR has been expressed by many researchers, there has been a tendency to ignore or minimise the evaluation of VR applications.

Evaluation is the key to quality

Evaluation studies are needed to ensure that the social, perceptual and cognitive needs and capabilities of human beings, as well as the needs of the specific tasks under consideration, are reflected in system design; that the hardware and software deliver the Collaborative Virtual Environment application in a cost effective manner and that CVE technology represents a significantly better way of doing old things or of doing new things that have not been possible before.

The specific type of evaluation depends on the characteristics of the system to be evaluated and on the purpose of the evaluation. Evaluation tasks can be roughly divided in two groups; evaluation of system characteristics and performance, and observation and measurement of human behaviour with and within the CVE applications. These two groups of data sources influence each other in many ways. It is the interaction we are interested in usually when doing usability studies. These two concerns are mutually compatible: a standard HCI focus, and a scientific enquiry focus.

When designing a usability study it is useful to think of the CVE as a set of layers in which different human activities, and different representations of information take place. In the beginning there is just the raw data space, which is basically the size of the files of which the CVE is made up, stored on the Hard Disk(s). This data is compiled and loaded into the memory of the computer so that it forms spaces in which people can navigate and interact; the architectural space. Submerged in that space of rooms, buildings, and worlds, is the semantic space. The semantic space is effectively all objects which represent information and their meaningful relationships. Relationships are expressed by color, shape, distance, boundaries, paths, etc.

Finally, superimposed on these spaces, is the social space. The CVE, with its rooms, and tools to visualise information, is made for human users. These users need to interact with each other. They need to establish contact, recognise each other, and work together. When addressing usability issues it seems like a good idea to establish which of these spaces are represented in the CVE you are planning to evaluate, and how these spaces are presented to the users.

Thus, the general evaluation goal is the degree and ways in which the CVE allows users to feel present and aware of each other within the three superimposed spaces described above. The specific evaluation goal however, will always be application dependent. For instance, when evaluating a 3D data-base visualisation tool which people can access through a CVE, you would firstly like to focus on the semantic space, and how effectively this space is implemented. The effectiveness of the implementation depends on the tasks which have to be performed to do queries of the data-base. Secondly, you would like to look at the social space to see whether the data-base allows people to collaborate effectively, and thirdly, you would like to see how the changes in either of the two spaces affect the other spaces and the users. For instance, when one user makes a query, how is this represented to the other users?