How to run a heuristic evaluation
The process we use for heuristic evaluations today can be traced directly back to the early 1990’s. Rolf Molich and Jakob Nielsen published their foundational ‘Improving a human-computer dialogue’ in 1990, building on work by people like David Cheriton (1976), Neilsen’s future consulting partner Donald Norman (1983), and Ben Schneiderman (1987), amongst others.
The good and the bad
There are some good reasons why you might use this technique:
It can be a quick and inexpensive way to generate feedback for designers
It can be used pretty early in the design process
It can give a more comprehensive assessment of the system than usability testing
Assigning the correct heuristic can suggest a good place to start for corrective measures
You can use it together with other usability methods
But there are also some fundamental problems with the technique, so it is critical that you understand its limitations and dangers too:
For it to be done well, the evaluators should ideally be double-experts; usability experts as well as experts in the subject domain of the project (i.e. finance, education, insurance, etc.)
You need to use more than one evaluator—this is often forgotten. A single expert working in isolation may only pick up 20% of the usability issues. Even ten experts may only surface 85%. A good compromise between effectiveness and practicality is to use between 3-5 evaluators, which gives you about a 60% hit rate.
Finding people suitable as evaluators can be difficult, and nowadays it may turn out to be more expensive than running a proper usability test with 5 participants.
Using heuristics to identify usability issues is a relatively black-and-white approach. It will identify more minor issues than usability testing, but it will also have plenty of ‘false positive’ issues that aren’t really problems at all.
Placing all of your trust in the heuristics may not be well founded. I’m not sure that Nielsen’s heuristics have been formally validated. Does anyone know?
The argument goes: “If you’re such an expert and have the experience you claim, how come you can’t just give us the answers for the right design?” Well, if you have sufficient experience and expertise in usability, you’ll also be well aware that users are notoriously unpredictable. It’s common to experience ‘aha!’ moments in usability tests that show something we never expected, even when the design seems to conform to key heuristics perfectly.
Heuristic evaluations are no substitute for usability testing, but they can help improve the potential of usability tests if they’re used in conjunction; running a heuristic evaluation before beginning a round of usability tests will reduce the number and severity of design errors discovered by users, helping minimise problems and distractions during the testing.
Running your own heuristic evaluation
What you’ll need:
- 3-5 experts
- a set of heuristics
- a list of user tasks
- a system to test (or screen shots/prototypes at a stretch)
- a standard form for recording your notes
Before you start:
- You need to know who your users are, and what their goals are. Your existing research may set the scene with scenarios, personas or story mapping, so use those to get up to speed. Do more research if you don’t have a good idea of whom you’re designing for.
- You also need to define the tasks that your users need to accomplish to achieve their goals, and how these works with the design vision you want to evaluate.
- There will be lots of different tasks, so I find it useful to rank them and focus on the most important. You can achieve this by listing them all (there may be hundreds of tasks) and getting lots of users to choose just their top 5. This is a statistical, quantitative method, so the more the merrier—two hundred user respondents is a ballpark. It’s also critical that you use the right language and terminology for your tasks, as users are only going to be skim reading to try and pick out their favourite tasks. Get your handful of stakeholders to do the same activity too, and then compare the results. Hopefully, you have a handful of tasks that users suggest are clearly ahead of the pack. If these match with what your stakeholders say, add these tasks directly to the priority list. Some other tasks considered important by users and stakeholders may need more discussion!
- Work out what the best evaluation method is, given the project and the tasks you’re investigating. Heuristic evaluation may not be the right answer.
- Decide on a set of heuristics to use. If you don’t have a good reason not to, I would recommend Neilsen’s or Gerhardt-Powals.
- Select your team of 3-5 evaluators. Ideally they both usability experience as well as domain knowledge related to your project, and might be found in your design team or list of professional contacts. You should give them all the same training on the principles and process, and ensure they’re interpreting the heuristics properly.