We conduct all of our usability test sessions remotely, around the world, using Gotomeeting online meeting software, and Camtasia to record the sessions.
Advantages of remote usability testing
Although we encountered a few teething pains early on in our testing, we’ve found remote usability testing provides us with a number of distinct advantages, including:
- Savings in cost and time
- No travel required for facilitator or participant
- No test facility or usability lab required
- Usability tests can be run in parallel with additional facilitators
- Less incentive required
- Fewer “No shows”
- Easier to recruit
- Larger and more diverse pool available than in a particular location
- Easier to access your own site visitors if you also recruit online
- Specialists don’t waste time travelling
- More consistent and flexible testing
- More realistic environment – in context of other software and apps participant uses
- Participants feel more at ease in their own home or office
- Potential to have participant control “proprietary computer and software”
- Multiple locations can be tested around the world the same day
- More uniform method and hints from same facilitators
- Remote observation more effective
- Does not interfere with test session
- Allows people anywhere in the world to observe – great for widely-dispersed stakeholders (e.g. global corporations)
- Observation space is not limited – could have hundreds observing if in groups
- Observers can see what is happening on screen better
- Observers can easily pass notes or comments to facilitator
Disadvantages of remote usability testing
Although there are a number of disadvantages, which may affect the choice for some studies, we have found most of them to be easy to deal with, or to have a minimal impact, when testing web-based applications or Internet/intranet websites.
- Not face to face
- Confirming identification of participants is more difficult – no image or photo ID
- Non-disclosure agreements have to be done electronically
- Distributing incentives can be more difficult – use electronic gift cards or cheques versus cash
- Test environment not as controlled
- Open office configurations make it difficult for participant to follow “think aloud” protocol
- Firewalls prevent joining gotomeeting
- Perceived risk that participant’s computer may not function properly or could be harmed
- Logistics can be slightly more difficult
- Different time zones may make scheduling more difficult for facilitators
- Potential for network delays to cause synchronization problems
- Limited to testing on high-speed connections
Remote test environment
The typical characteristics of the remote test environment are:
- User is in their natural home/office environment
- Facilitator and observers view screen and mouse movements of participant
- Facilitator and participant interact via a telephone connection and chat
- Observers hear audio and view screen interactions via computer and internet connection
- Participant screen, audio, and keyboard/mouse activity are recorded for subsequent analysis
Our experience
To us, the main advantage is being able to test in a more realistic, contextually relevant environment — people are in their homes or offices, using familiar equipment and interacting with the software applications they use every day. Although there is some loss of control, we feel it is more than made up for by being able to observe people using the web as they would normally, subject to phone calls, pop-ups, email arrivals, and so on.
Seeing people’s personal computer environments can sometimes be enlightening. Compare the amount of screen real estate for these two participants (see image below) and notice how differences in browser configuration and resolution can drastically affect the amount of content visible to the participant.
Lack of video, which typically shows the participant’s face and body language, was a concern at first but we have found most non-verbal cues are readily discernable in the participant’s voice and in their cursor movements. In conjunction with our use of a “think aloud” protocol, which asks the user to verbalize what they are thinking or experiencing, we can generally tell when someone is getting tired, frustrated or confused.
Recruiting has been easier because people do not have to travel and try to find the test location. Because the time commitment is often one half or less of that required for conventional testing, they are much more willing and able to take an hour out of their workday. Because we are not constrained by geographical location, one facilitator has been able to test people in London, San Francisco, Toronto, and New York during the same day. We’ve also been able to test with blind participants using screen readers. Normally, this would be difficult to test in a lab environment as blind individuals often customize their computing environment extensively.
Our invitation email provides a simple link which allows them to test their ability to connect ahead of time and then connects them in to a meeting at the assigned time. Most people had no difficulty connecting the first time. Only a handful of people required any instruction and this was typically done in the first few minutes of the test session.
Being able to invite observers at any time is a big plus for stimulating interest in the usability results and for letting stakeholders share in the experiences of their customers. Once they’ve been exposed to usability testing, they have a much better appreciation of how it can contribute to more successful websites or web applications.
Technical glitches are no more prevalent than we typically experienced in face-to-face testing in the past. We’ve encountered some firewall issues and occasionally, a participant’s computer running slowly or not being able to get a very good connection speed. In general, participant’s were very tolerant of any network issues.
We have been very pleased with the quality of results we obtain from our remote usability testing. From our experience, they are on par or superior to the results we’ve obtained previously using face-to-face techniques. The superior elements are due to us gaining a better understanding of contextual issues – for example, type of browser, screen resolution, window sizing, multi-tasking, interactions with other software, etc.
A comparison study by Bolt | Peters in 2005 found “no significant differences in the quality and quantity of usability findings between remote and in-lab approaches.” However, consistent with our experience, the study showed key advantages of remote testing in the areas of time, recruiting, and the ability to test geographically diverse audiences.
Quotes of the month
“If your goal is to sell usability in your organization, then I believe 3-4 users will be sufficient. Much more important than the number of users is the sensible involvement of your project team in the test process and proper consensus-building after the test”
Rolf Molich, 2002
“Interior pages accounted for 60 percent of the initial page views. Recognize this and support it. Don’t try to force users to enter on the homepage.”
Jakob Nielson and Hoa Loranger, Prioritizing Web Usability, 2006
