Welcome to iorg.com - business aligned web sites
HomePapersCase StudiesAbout iorg.com

The iorg.com Newsletter - February 2004

Strengths and Weaknesses of Heuristic Reviews

Heuristic reviews can be a valuable design and implementation tool. But don’t mistake them for a full practices checklist audit, or a usability test.

What Is a Heuristic Review

Jakob Nielsen and Rolph Molich introduced the heuristic review in the 1980s as a faster and less expensive way to approximate the results of a usability test. Unlike usability testing, heuristic reviews contain pre-specified parameters that are given numeric scores by the experts while role-playing a specific visitor scenario. It generally is recommended that at least three experts conduct the review independently and average their scores.

It is important to remember that heuristic reviews were developed in pre-web days for use with computer applications. They provided a framework to help usability experts identify and report common usability barriers to developers and designers. When web sites came into existence in 1993, heuristic reviews were applied to them almost immediately.

By the late 1990s, the heuristic review became widely used and modified as a method for comparing web sites to each other. Unlike usability testing, the pre-specified parameters and numeric scores allowed a single numeric value to be calculated for a web site. This meant sites could be compared using the single score based on common parameters. A number of consulting and advisory firms offered modified heuristic reviews to companies primarily as a form of marketing bragging rights for their web sites, even though there was never evidence that the scores had any significant effect on visitors choosing a site.

However, even when the objective is to get the highest score, the heuristic approach has serious drawbacks. First, assigning the numeric values to the parameters is highly subjective.   Second, heuristic sets have been adapted for standardized comparison not actionable response. Recommendations for how to improve a web site for a higher score require expert advice, rather than being presented as precisely defined and cleanly organized practices in the heuristic review framework itself.

Rolf Molich, said in a recent interview: “Heuristic inspections are cheap, simple to explain, and deceptively simple to execute. However, I don’t use this method very often and I don’t recommend it to my clients… heuristic inspection is based solely on opinions.”(1)

How to Use a Heuristic Review

This does not mean that heuristic reviews have no value in the web site design process. The heuristic review concept and framework do provide value, in the context of their initial purpose, as a tool to help designers and developers stay on track when making decisions on design and implementation options.

Key visitor scenarios used with a heuristic review at regular intervals in the design and development cycle act much like a carpenter’s plumb line. They help keep specific design and implementation decisions square with the objectives of the web site. However, heuristic reviews should not be confused with a full practices checklist audit(2 & 3) that specifies all practice attributes known to be important or for a usability test for verifying specific hypotheses.

There are a number of heuristic review sets available. The choice of one set over another is less important than the development of the scenarios used in the review. The scenario provides the description of the visitor that helps the designers and implementers stay on track. If the scenario is not well constructed and aligned with the business and visitor objectives for the site, the result will be out of alignment regardless of the review set chosen.  On the other hand, most heuristic review sets will produce similar results when used by the same experts. Just having a structured framework when applying the scenario is more important than which structured framework is chosen.

  • Heuristic reviews have limited value in web site design and implementation
  • Designers and implementers should use heuristic reviews
    • When they are trying to decide between design or implementation options
    • Late in development as a quick check to see if anything pops out as missing
  • Reviews should be used as a utility plumb line, not in place of a practices audit or a usability test
  • Developing good scenarios is more important than the specific heuristic review set chosen.

(1) Rolf Molich, Interview with Christine Perfetti for User Interface Engineering
(2) The Rise of Web Site Principles and Practices
(3) Defining a Basic Unit of Web Site Behavior

Please feel free to forward this newsletter to a friend or colleague who might be interested.

Home - Papers - Case Studies - About iorg.com - Privacy - Legal

For more information contact: info@iorg.com
© Copyright 1997-2004 iorg.com