Are we overlooking some usability testing methods? A comparison of lab, beta, and forum tests

Abstract

We compared the effectiveness of Lab testing, Beta testing, and Forum testing at identifying software usability problems. Thirty participants were involved in the experiment, with ten participants in each of the three test conditions. The Lab test involved participants performing prescribed scenarios with the software in a controlled Lab environment, while human factors engineers recorded participant's problems. The Beta test method had participants use the software in their own environment to perform their real world work, and record their own problems. The Forum test was similar to the Beta test, except that the software was made available on a company-wide computer bulletin board and the participants selected themselves. Findings show that the Beta test method was as effective as the Lab test method in the number of problem types identified. The Lab test uncovered a larger proportion of serious usability problems than did the Beta test. The Beta test method was the most cost-effective method. The Forum test method found the fewest number of problem types, and was the least cost-effective. Thus, the results of this study broaden the current literature by showing that the Beta test method may be a cost-effective alternative to the traditional lab test.