ISO 9241-11 Revised: What Have We Learnt

About Usability Since 1998?

Nigel Bevan1(&), James Carter2, and Susan Harker3

1
Professional UX Services, 12 King Edwards Gardens, London W3 9RG, UK
[email protected]
2
Computer Science Department, University of Saskatchewan, Saskatoon,
Canada
[email protected]
3
Loughborough Design School, Loughborough University, Loughborough
LE11 3TU, UK
[email protected]

Abstract. A revision is currently being undertaken of ISO 9241-11, published

in 1998 to provide guidance on usability. ISO-9241-11 defines usability in terms
of effectiveness, efficiency and satisfaction in a particular context of use. The
intention was to emphasise that usability is an outcome of interaction rather than
a property of a product. This is now widely accepted. However, the standard
also places emphasis on usability measurement and it is now appreciated that
there is more to usability evaluation than measurement. Other developments
include an increasing awareness of the importance of the individual user’s
emotional experience as discretionary usage of complex consumer products and
use of the World Wide Web have became more widespread. From an organi-
sational perspective, it is now appreciated that usability plays an important role
in managing the potentials risks that can arise from inappropriate outcomes of
interaction. The revision of ISO 9241-11 takes account of these issues and other
feedback.

Keywords: Standards
Usability
User experience

1 Origins of ISO 9241-11

What is usability?1 In the English language, usability is typically defined as the

“capability of being used”, implicitly the capability of an entity to be used. In the 1980s
and 1990s a considerable amount of material was published describing the attributes
that would make a product usable (e.g. Smith and Mosier [20], and the early parts of
the ISO 9241 series [9]). From this perspective, usability could be designed into the
product, and evaluated by assessing consistency with these design guidelines, or by
heuristic evaluation. This was the perspective taken in ISO 9126:1992: “Software
engineering—Product quality”, which defined usability as “a set of attributes of

1
This is a problem that has been troubling the first author since work started on developing the
original ISO 9241-11 in 1988 [1–3].

© Springer International Publishing Switzerland 2015

M. Kurosu (Ed.): Human-Computer Interaction, Part I, HCII 2015, LNCS 9169, pp. 143–151, 2015.
DOI: 10.1007/978-3-319-20901-2_13
144 N. Bevan et al.

software which bear on the effort needed for use and on the individual assessment of
such use by a stated or implied set of users”.
However an alternative approach recognised that the same product could have
significantly different levels of usability depending on who was using it with what
goals. In order to achieve usability for the intended users it would be necessary to
address the actual outcomes of their use of the product. This was the approach
advocated by Whiteside, Bennett and Holzblatt in 1988, based on an operational view
of usability in terms of user performance and satisfaction [22].
The approach taken to usability in ISO 9241-11 (1998) was similar, defining
usability as: “The extent to which a product can be used by specified users to achieve
specified goals with effectiveness, efficiency and satisfaction in a specified context of
use.”
This approach has the benefit that it directly relates to user and business require-
ments: effectiveness means success in achieving goals, efficiency means not wasting
time and satisfaction means willingness to use the system.
ISO 9241-11:1998 explains that in order to specify or measure usability it is
necessary to identify the goals and to decompose effectiveness, efficiency and satis-
faction and the components of the context of use into sub-components with measurable
and verifiable attributes. The standard identifies the benefits of this approach:
• The framework can be used to identify the usability measures and the components
of the context of use to be taken into account when specifying, designing or
evaluating the usability of a product.
• The performance (effectiveness and efficiency) and satisfaction of the users can be
used to measure the extent to which a product is usable in a particular context.
• Measures of the performance and satisfaction of the users can provide a basis for the
comparison of the relative usability of products with different technical character-
istics which are used in the same context
• The level of usability needed for a product can be defined, documented and verified
(e.g. as part of a quality plan).

2 What Have We Learnt About Usability Since 1998?

ISO 9241-11:1998 has been highly successful in providing an internationally accepted

basis for understanding and applying usability. Its definition of usability is widely
referenced in research, industry, and other international standards. This widespread use
has been accompanied by a greater level of understanding of usability than was
available when this first version of ISO 9241-11 was developed. As a result, work was
started in 2011 to revise this standard to provide enhanced guidance based on over a
decade of experience with the concepts that it introduces. This revision is intended to
retain the basic concept of usability and to provide users of the concept with further
levels of understanding about usability taking account of what we have learnt about
usability since 1998, including the issues identified below.
 ISO 9241-11 Revised: What Have We Learnt About Usability Since 1998? 145

2.1 It Is Important to Understand the User’s Experience

In much early work in industry, usability was operationalised primarily in terms of the
user’s performance (effectiveness and efficiency), which was regarded as the prime
issue given the many problems that were experienced by users of commercial systems.
But as use of complex consumer products and of the World Wide Web became
widespread, there was an increasing awareness of the importance of the user’s sub-
jective reactions and emotional experience. This has led some authors to regard
usability as being restricted to “ease-of-use”, and relegate it to the role of a “hygiene
factor”, e.g. [5] usability “…is a thermometer that sets the ‘hygiene’ level of a product.
Users today take the ‘ease of use’ part of product concepts for granted and will not
praise the fact that a product or service has good usability.” Similarly, Hassenzahl et al.
[4] differentiate between ease of use as a “pragmatic quality being a ‘hygiene factor’,
enabling the fulfilment of needs through removing barriers but not being a source of
positive experience in itself”, and “hedonic quality being a ‘motivator’, capturing the
product’s perceived ability to create positive experiences through need fulfilment”. ISO
9241-210:2010 “Human-centred design for interactive systems” defines user experi-
ence as a “person’s perceptions and responses resulting from the use and/or anticipated
use of a product, system or service”. User experience focuses on the experience of an
individual in contrast with the view of effectiveness, efficiency and satisfaction as
representing the collective responses of a group of users.
One objective of the revision of ISO 9241-11 is to clarify that the satisfaction
component of usability includes aspects of user experience.

2.2 There Is More to Usability Evaluation Than Usability Measurement

ISO 9241-11:1998 focussed on the evaluation of usability by user based measurement
of effectiveness, efficiency and satisfaction, as this was a convincing way of demon-
strating the existence of usability problems to system developers. ISO/IEC DIS
25066:2015 “Common Industry Format (CIF) for usability: Evaluation report” pro-
vides a broader view explaining that usability evaluation can be based on inspection to
identify potential usability problems, in addition to observation of user behaviour and
the collection of user-reported data. The revision of ISO 9241-11 needs to make it clear
that effectiveness, efficiency and satisfaction represent the intended outcomes of
interaction, but that their measurement does not represent the only way of evaluating
usability.
Another issue is that many authors have emphasised the importance of more spe-
cific aspects of usability. For example Nielsen’s definition of usability [19] includes, in
addition to efficiency in normal use and satisfaction with use:
• Learnability in early use.
• Memorability after a period of non-use.
• That errors during use can be corrected, and do not lead to undesirable
consequences.
146 N. Bevan et al.

2.3 Avoidance of Negative Outcomes

In the existing ISO 9241-11 effectiveness focuses on the accuracy and completeness
with which goals are achieved. But an unintended outcome can have significant
undesirable negative consequences for the individual or the organisation (such as
inconvenience, wasted time, or financial loss). The ISO/IEC 25010:2010 “System and
software quality models” standard takes account of both positive and negative out-
comes by defining “quality in use” as a combination of the positive outcomes of
usability in the existing ISO 9241-11 combined with freedom from risk of negative
outcomes. The revised ISO 9241-11 also needs to explain how to take account of the
risk of any negative outcomes that could arise from inadequate usability.

3 The New Version of ISO 9241-11

3.1 Requests for Changes and Feedback

The revision of ISO 9241-11 was preceded by consultation with the countries that
participate in the ISO TC159/SC4 Ergonomics standards committee and the related
ISO working groups. The feedback (which identified many of the issues above) pro-
vided a starting point for the changes listed below that have been made in the first draft
that has been circulated for national voting and comment. The text will be revised
based on these comments, and will be circulated for further voting and comment later
in 2015.

3.2 Changes Made in the New Draft

Systems, Products, Services and Environments. ISO 9241-11 currently applies to the
usability of “products”. In line with changes in standards such as ISO 9241-210 this has
been extended in the new draft to “products, systems and services”, as the concept of
usability applies equally well to all these categories. The new draft also explains that
although environments are considered as part of the context of use, user interactions
with a specific environment or component of the environment can be considered in
terms of the usability of an environment (e.g. the smoothness of a path used by a
wheelchair).
Goals. The current standard only mentions goals that achieve well-defined outputs.
In reality, people may have other reasons for interacting with a product, system or
service, so the new draft takes account of a much wider range of goals that include
aspects of user experience:
(a) output related outcome(s) that could either be assigned (in an organisational
context) and/or personally chosen;
(b) personal outcomes such as entertainment or personal development;
(c) usability outcomes in terms of levels of specific (sub)dimensions of usability, such
as the desired level of accuracy;
(d) other outcomes (e.g. related to safety, security or privacy) to be satisfied in the
course of achieving output-related or personal outcomes.
 ISO 9241-11 Revised: What Have We Learnt About Usability Since 1998? 147

Effectiveness. The effectiveness with which a goal is achieved was previously defined in
terms of accuracy and completeness. Appropriateness has been added as an additional
consideration that can include: a) the form and needed degree of precision of the output
(e.g. is information displayed appropriately on a web page?) and b) avoidance of errors
and minimization of the risk of any unacceptable consequences that could arise from
lack of accuracy and completeness. Adding appropriateness to the other components of
effectiveness goes some way towards taking account of both the potential positive
outcomes (accuracy and completeness) and the risk of negative outcomes.

Efficiency. Efficiency was previously defined as the ratio of effectiveness divided by

the resources consumed. While this is scientifically correct as a productivity measure, it
is only meaningful for continuous output, which is not a common situation. So, effi-
ciency has been redefined in the revised standard as the resources (time, human effort,
costs and material resources) that are expended when achieving a specific goal (e.g. the
time to complete a specific task).

Satisfaction. In the current ISO 9241-11, satisfaction is defined as “freedom from

discomfort, and positive attitudes towards the use of the product”. The new draft
identifies a much wider range of personal responses, including those that have been
highlighted in research on user experience: “the extent to which attitudes related to the
use of a system, product or service and the emotional and physiological effects arising
from use are positive or negative”.

Context of use. The current ISO 9241-11 does not say anything about how wide a
range of users, tasks and environments should be included in the context of use. The
revised standard explains that usability can be related to: (a) All potentially relevant
contexts of use (when considering overall usability), (b) Specified contexts of use (the
users, goals and environments of particular interest), (c) A single instance of the
context of use, or (d) The context of use for a single individual (for individual usability
when considering a user’s experience).
The current ISO 9241-11 provides detailed information on how to specify the
context of use. This has been removed from the new standard as it is now included in
ISO/IEC 25063:2014 “Common Industry Format (CIF) for usability: Context of use
description”.

Scope of “Usability”. The revision explains the relationships between the approach to
usability in ISO 9241-11, and other interpretations. By defining usability in terms of the
measurable outcomes of effectiveness, efficiency and satisfaction, ISO 9241-11 takes
an approach to usability that:
(a) Focuses on the outcomes of interaction rather than the user interface design
activities and resulting product attributes that make a product usable.
The new draft explains that the term “usability” can also be used as a qualifier to
refer to design related activities and product attributes. Thus it may be used refer to the
knowledge, competencies, activities and design attributes that contribute to usability
(such as usability expertise, usability professional, usability issue, usability method,
usability evaluation, usability problem, and usability guidance).
148 N. Bevan et al.

(b) Defines usability a high-level concept rather than referring to normal use of a
product in contrast to learning to use or reuse a product.
The new draft makes it clear that usability applies to all aspects of use, including:
• Learnability, to enable new users to be effective, efficient and satisfied when
learning to use a new system.
• Regular use, to enable users to achieve their goals effectively, efficiently and with
satisfaction.
• Error protection, to minimise the possibility that users can make errors that could
lead to undesirable consequences.
• Accessibility, so that the system if effective, efficient and satisfying for users with
the widest range of capabilities.
• Maintainability, to enable maintenance tasks to be completed effectively, efficiently
and with satisfaction.
Usability Measures. The current ISO 9241-11 focuses on specification and mea-
surement of usability, and includes detailed information on usability measures and
specification of usability. This information has not been included in the new draft. It is
possible that it might be included in a new related standard on usability measurement.

3.3 Additional Proposed Changes

Some of the other aspects of usability that have been proposed as relevant, but that
have not yet been included are explained below.

System and User Outcomes. One of the issues raised in the feedback was that it is
sometimes necessary to take account of both the objective system outcome (e.g. whether
an ecommerce transaction has been completed) and the subjective user outcome (e.g.
whether the user believes that the transaction is complete). Both types of goals may need
to be achieved for a successful outcome that can be regarded as effective.
An additional distinction could be made in the standard between goals for system
outcomes resulting from interaction (such as information provided to the user, com-
pleting a purchase or casting a vote), and the outcomes for the user such as acquiring
knowledge or making a decision based on the output of a system, product or service.

Evolving Goals. The user’s goals can evolve during interaction, particularly when the
user is exploring use of a product. So a product may have low usability for the initially
intended outcomes, but be quite usable for the final outcomes achieved. Conversely,
with new goals the user may find that the level of usability is lower for these goals.
Usability can be considered in relation to the user’s goals at any stage during the time
that the user interacts with the product, system or service.

Social Responsibility. One critique of ISO 9241-11 is that it ignores social respon-
sibility (for which there is now a standard: ISO 26000). A clear distinction needs to be
made between considering usability for the user’s intended outcomes and for another
 ISO 9241-11 Revised: What Have We Learnt About Usability Since 1998? 149

stakeholder’s intended outcomes. Taking account of the user’s goals satisfies funda-
mental human needs and produces designs that respects human dignity.
The new draft defines accessibility as usability for people with the widest range of
capabilities. To respect social responsibility, systems, products and services should be
designed to be usable by people with the widest range of capabilities who could
potentially use the system, product or service.

3.4 Related Concepts

In addition to elaborating on the understanding of usability, the revised version of ISO
9241-11 provides an explanation of the relationship between usability and some
associated concepts including:

Human-Centred Quality. The objectives of human-centred design are identified in

ISO CD 9241-220:2015 “Processes for enabling, executing and assessing
human-centred design within organizations” as achievement of human-centred quality,
which is composed of usability, accessibility, user experience and risk reduction. While
usability as described in the revised ISO 9241-11 takes account of aspects of accessi-
bility, user experience and the risks that could arise from poor usability, use of the
concept human-centred quality makes each of these explicit and independent objectives.

Accessibility. ISO 26800 emphasises the importance of widening the target population
to take account of the range and diversity of human characteristics, thus making
products, systems, services, environments and facilities more accessible to more peo-
ple. ISO 9241-11 interprets accessibility as usability for people with the widest range of
capabilities, which is applied in the same way as usability. This provides a basis for
specifying and evaluating accessibility in terms of effectiveness, efficiency and satis-
faction for a wider range of user capabilities.

User Experience. User experience focuses on the user’s preferences, perceptions,

emotions and physical and psychological responses that occur before, during and after
use, rather than the observed effectiveness and efficiency. While usability typically
deals with goals shared by a user group, user experience is concerned with individual
goals, which can include personal motivations including needs to acquire new
knowledge and skills, to communicate personal identity and to provoke pleasant
memories. User experience also puts emphasis on how the experience changes with
repeated use.
One source of potential confusion is the increasingly widespread use of the term
user experience to refer to an overall view of all aspects of the user’s interaction with a
system, product or service, rather than the original meaning that emphasized the
importance of emotional experience. This use of the term user experience is closer to
the concept of usability in the revised version of 9241-11, which explicitly includes the
personal factors for individuals.
150 N. Bevan et al.

4 Contributing to the Development of ISO 9241-11

If you would like to contribute to the development of ISO 9241-11, or to comment on

drafts, you can either do this via your national standards body [6], or if you are a
member of one of the ISO TC159/SC4 liaison organisations [7] such as UXPA [21]
you can participate through the liaison organisation.

Acknowledgements. ISO 9241-11 is being developed by the ISO working group

TC159/SC4/WG6 and the new draft was produced by the authors of this paper. Particular thanks
are due to Karsten Nebe for his feedback on the paper, and to the other members of the working
group that include: S. Fukuzumi, C. Hoback, J. Earthy, K. Enflo, T. Geis, T. Jokela, C. Lutsch, S.
Turner, A. Walldius, and J. Williams.

References
1. Bevan, N.: Usability is quality of use. In: Anzai, Y., Ogawa, K., Hirohiko, M. (eds.)
Symbiosis of Human and Artifact: Proceedings 6th International Conference on Human
Computer Interaction, vol. 2. Elsevier, Amsterdam, July 1995
2. Bevan, N.: Extending the concept of satisfaction in ISO standards. In: Proceedings of
KEER2010, Paris, 2–4 Mar 2010
3. Bevan, N., Kirakowski, J., Maissel, J.: What is usability? In: Bullinger, H.J. (eds.)
Proceedings of the 4th International Conference on Human Computer Interaction, Stuttgart.
Elsevier, Sept 1991
4. Hassenzahl, M., Diefenbach, S., Göritz, A.: Needs, affect, interactive products - facets of
user experience. Interact. Comput. 22, 353–362 (2010)
5. Hellman, M., Rönkkö, K.: Controlling user experience through policing in the software
development process. In: Proceedings of I-USED 2008, Pisa, Sept 2008
6. ISO: Members. www.iso.org/iso/home/about/iso_members.htm
7. ISO: TC159/SC4 Ergonomics of human-system interaction. www.iso.org/iso/home/
standards_development/list_of_iso_technical_committees/iso_technical_committee.htm?
commid=53372
8. ISO/IEC 9126: Software engineering—product quality (1991)
9. ISO 9241: Ergonomic requirements for office work with visual display terminals (VDTs)
10. ISO 9241-11: Ergonomic requirements for office work with visual display terminals (VDTs)
- Part 11 Guidance on usability (1998)
11. ISO CD 9241-11: Ergonomics of human-system interaction – Part 11: Usability: definitions
and concepts (2015)
12. ISO 9241-210: Ergonomics of human-system interaction – Part 210: Human-centred design
for interactive systems (2010)
13. ISO CD 9241-220: Ergonomics of human-system interaction - Part 220: Processes for
enabling, executing and assessing human-centred design within organizations (2015)
14. ISO/IEC 25010: Systems and software engineering – systems and software product quality
requirements and evaluation (SQuaRE) – system and software quality models (2011)
15. ISO/IEC 25063: Systems and software engineering – systems and software product quality
requirements and evaluation (SQuaRE) – common industry format (CIF) for usability:
Context of use description (2014)
 ISO 9241-11 Revised: What Have We Learnt About Usability Since 1998? 151

16. ISO/IEC DIS 25066: Systems and software engineering – software product quality
requirements and evaluation (SQuaRE) – common industry format (CIF) for usability:
Evaluation report (2015)
17. ISO DIS 26800: Ergonomics – general approach, principles and concepts (2011)
18. ISO 26000: Social responsibility (2010)
19. Nielsen, J.: Usability Engineering. Academic Press, Waltham (1993)
20. Smith, S.L., Mosier, J.N.: Guidelines for Designing User Interface Software. MITRE
Corporation, Bedford, Mass. ESD-TR-86-278 (1986)
21. UXPA: User experience professionals association liaison with ISO. www.uxpa.org/
standards
22. Whiteside, J., Bennett, J., Holzblatt, K.: Usability engineering: our experience and evolution.
In: Helander, M. (ed.) Handbook of Human-Computer Interaction. Elsevier, New York
(1988)