The session was well attended by some 30 delegates and, from the discussion during the brief session (45 minutes), there was a clear indication of the widespread interest in the associated subjects of usability and evaluation of interfaces to virtual environments.

Attendees introduced themselves and their interest in the subject, which served to reflect the wide range of circumstances within which usability is a key issue. These ranged from the use of VEs to address specific domain problems (surgeon training) to rese arch into acquistion of spatial knowledge to the use of specific interaction techniques (using natural language).

We tried to centre the discussion around the following cycle of empirical activities:

1. Explore + observe user behaviour in real-world and virtual environments;
2. Measure task performance;
3. Find behavioural ? and design change;
4. Formulate hypotheses and experimental tasks;
5. Propose design to support user needs;
6. Test hypotheses;
7. Back to 1) if unsure, on to 2) if appropriate task measures have been identified.

A number of themes emerged during the discussion:

• the extent to which the tasks that users of VEs will carry out can be specified;
• the need to include the impact of VEs on work processes and consider the cost-benefit within key domain areas;
• the issue of ecological validity, namely the extent to which small scale evaluation studies will scale up to real world use contexts;
• the need for guidelines to determine the appropriateness of immersive and desk-top solutions respectively;
• the problems caused by the necessity of small sample sizes for studies involving long trials per subject.

The issue which generated most debate was whether or not usability of VEs was significantly different from usability of any other type of application, and whether or not existing methods for studying usability were directly applicable for use with VEs.

A 1-day workshop to focus on evaluation and usability issues in VEs was announced and this will be held at De Montfort University on 17th December 1998. We promised that the call for papers would be circulated shortly.

Discussion Organizers: Chris Hand, Howell Istance, De Montfort University Leicester; Kulwinder Kaur, City University London; Anthony Steed, University college London (COVEN); Jolanda Tromp, University of Nottingham (COVEN).

CVE'98 Conference Organizers: Dave Snowdon, Elizabeth Churchill.

One up in the hierarchy:

Page with general links about VE Evaluation discussion.

CRG Technologies
CRG Research
Communications Research Group
Department of Computer Science
University of Nottingham