HCI VISTAS, VOLUME-II, 2006-2007

 

Usability Testing: A Reality Show?

Dr. Dinesh S. Katre

Article INS-16./Jan. 2007
 
BIG BOSS, the Indian adaptation of the internationally hit TV program called Big Brother. 13 celebrities holed up in a house for 100 days with no contact with the outside world (no newspapers, no magazines, no phones, no television, no radio, no watches or clocks) and under 24-hour surveillance by 30 cameras.

Does Big Boss setup appear somewhat similar to the usability test lab with video cameras? Well, let me make it very clear. I neither wish to propose Big Boss as great entertainment nor as the model for on-camera usability testing. Lets keep aside the glamour and show-biz of this program.
After watching a few episodes, I felt that the program offers free lessons to usability practitioners. Especially if one is dealing with 'on-camera usability testing'. Why not take some advantage of it?
 
 
Figure 1. Big Boss Participants and Usability Lab
 
Past several weeks, I have been observing the celebrity participants of Big Boss along with other television viewers across the nation, but rather differently, and more as a usability practitioner. In both cases, (participants of Big Boss or the Usability Test) you are observing either through camera or in person.

This discussion is more about the human behavior under the gaze of somebody.

In a typical on-camera usability test, in the lab environment, we ask the subjects to perform certain tasks using the product. Such tests are time-bound and follow strict procedures in terms of user profiling, briefing, task descriptions, actual performance and think aloud exercise, screen capturing (in case of software), etc. You do the video recording of usability test to observe the user performance, their response and behavior. All this is perfectly fine.

Coming back to certain observations from the Big Boss reality show. I found that the participants of Big Boss have evolved through distinct psychological stages, which are sequentially progressive.

Self-consciousness
It means acute self-awareness, in which one freezes the movements or expressions due to unpleasant feeling of being observed or watched by someone. The participants feel shy, restricted, frozen; they become introverted and behave unnaturally. The big boss participants did not go through this stage, as most of them are actors. But the subjects of usability tests often become self-conscious.

Camera-consciousness
This is the state of mind where one succumbs to self-objectification. One is aware of the video camera. As a result of self-objectification the participants try to look nice, photogenic; become more extroverted and dramatic, and behave unnaturally.

The celebrity participants of Big Boss displayed distinct camera-consciousness in the beginning. Even the users go through these stages during on-camera usability testing.

Collective-consciousness
The celebrity participants in Big Boss slowly developed collective-consciousness. In this process, the group began to develop formal or informal values and rules to be followed by everyone to achieve certain objectives collectively. A group resorts to collective-consciousness with the purpose of co-existence. It gets into the social mode. This is not particularly applicable in the context of usability testing (one user at a time). But it may be applicable if the users are tested in the group activity.

Natural-self
Off and on the participants also become their natural selves. After spending several weeks in the house of Big Boss, the participants have forgotten about cameras and have turned into their natural selves. During lab usability tests, very rarely the user gets time to become natural self. It asks for more time.

Situational-consciousness
The Big Boss participants have arrived at the stage, wherein they are scheming to meet personal objectives. Their desires and traits are noticeable. Those who are tired of isolation, desperately wish to get evicted from the house of Big Boss. They are deliberately working towards getting voted out by viewers. There are some who want to continue for longer duration so as to make more money. Also their inter-personal equations in terms of friendship, hate, rivalry, disgust and other complex human emotions are evidently visible. It is an interesting case for observing group-dynamics.

Coming back to on-camera usability testing. I am not too sure, but based on what I have seen and read, I feel that we push the users into on-camera usability testing too soon. Without giving them sufficient time to become their natural selves. Otherwise, one is likely to record the test while users are struggling with self-consciousness or camera-consciousness or both simultaneously. To aggravate their problems, they are susceptible to task-consciousness or test-consciousness as well. As a result, they fumble and make trivial mistakes, which can be misleading input for design! Such tests may not serve the purpose very effectively.

Of course, it also depends on the objective of usability testing. Above approach might be working in some type of usability feedback sessions. Sometimes you need immediate response from users, in well-defined conditions. So that is okay.

To capture more authentic data, particularly about the natural behavior of user and the usage of product in real life conditions, it is necessary to observe the field performance. That captures the reality of product's usability for usability practitioners.

So the ritual of usability testing makes sense only if
it captures the natural response and behaviour of user!
And that is the true challenge.
 
 
Courtsey for images:
Sony Entertainment Television India
Sun Usability Lab
Feedback at
Contribute Articles To
© Copyright 2006-2007
Perspectives