2019 International Conference on Computing, Mathematics and Engineering Technologies – iCoMET 2019

User Experience and Recommender Systems

Zohreh Dehghani Champiri Ghulam Mujtaba

Department of Software Engineering, Faculty of Computer Department of Computer Science,
Science and Information Technology, University of Malaya, Sukkur IBA University,
50603 Kuala Lumpur, Malaysia Sukkur, Pakistan
[email protected] [email protected]

Siti Salwah Salim Chun Yong Chong

Department of Software Engineering, Faculty of Computer School of Information Technology, Monash University
Science and Information Technology, University of Malaya, Malaysia, Kuala Lumpur, Malaysia
50603 Kuala Lumpur, Malaysia [email protected]
[email protected]

Abstract—Since the advent of recommender systems (RS), the field of RSs but research on UX of RS is quite new [20].
majority of the published work aim to propose new algorithms Therefore, in this study, we discuss definition of UX especially
that are more accurate in predicting the users’ need. Recently, in the field of RS, as well as methods in UX evaluation and
researchers have acknowledged that embedding User eXperience review relevant UX models proposed in the existing studies.
(UX) into recommender systems increase the effectiveness of
recommendations dramatically for the end users. However, UX is
essentially a subjective matter and it is not easy to be measured II. DEFINITION OF UX
without direct user’s feedback on the recommendation. There In the early 1990’s, cognitive scientist, Don Norman coined
are few studies investigating the role of UX in the field of RSs but the term UX. He indicated that UX is about the user’s feeling
research on UX of RS is relatively less explored. Therefore, in (positive and negative) about a product over a period of time
this study, we review the existing studies and discuss definition of [21]. He elucidated that the term of UX was invented because
UX especially in the field of RS. Our review showed that despite there was a need for an umbrella term beyond the human
the recent attempts on UX evaluation of RS, this area is still new interface and usability to include all aspects of the user’s
and needs further investigations. experience with a system. The examples of these aspects are
interface, graphics, and interaction.
Keywords—Recommender system, User Experience, User
eXperience, User Experience Models, User-centric evaluation, User Therefore, so far the term of UX has been considered as an
Experience evaluation, Human-computer interaction. umbrella term to motivate researchers to focus on the above
mentioned aspects which are beyond usability [22] and mostly
I. INTRODUCTION in the field of Human Computer Interaction (HCI). There is no
a clear definition of UX [23][24]. However, the ISO (ISO
In 1997, the term Recommender Systems (RSs) was coined
9241-110:2010 (clause 2.15)) defines the UX as an area which
by Paul Vesnick and Hal R. Varian [1]. They described the
“focuses on a person’s perception and the responses resulting
RSs as a tool for applying decision making in addition to
from the use or anticipated use of a product, system, or
information retrieval [2]. Apart from information retrieval [3],
service”.
the birth of RSs was derived from various domains such as of
cognitive science [4], marketing [5], management [6], as well Kraft (2012) emphasized that UX is not about creating the
as forecasting [7] and approximation theories [8][9]. RSs, are newest and cutting-edge technologies but a great UX around
considered an appropriate tool to facilitate and accelerate the the product. For example, Nokia, as a leader of UX, failed to
process of information seeking by offering appropriate search create an acceptable UX for touch screens in the mobile phone
result suggestions to users [10] [11] when they are reviewing industry but Apple Inc. managed to reinvent UX for mobile
huge amount of relevant and irrelevant sources in scholarly phones. Therefore, it has been said that today, UX is the key
repositories [12, 13]. battleground for all kind of products in the consumer business
market [25].
Since the advent of RSs, many papers have been
published[14][11] which the majority aim to create more In software engineering, as a matter of fact, if a product
accurate algorithms. Recently, RSs researchers pointed out that fails to meet rising end users’ needs, it makes both the product
considering the UX in the process of recommendation and the company (creator of product) obsoleted. UX is
generation tremendously impact on the users of the RSs in term becoming the key competitive factors in more and more
of values [15-17]. However, as mentioned before, UX is industries. Users are demanding products that are both easy to
subjective and it is extremely difficult to measure UX of a RS use as well as joyful and fun to use. Users will choose the
[18] [19]. There are few studies investigating the role of UX in

978-1-5386-9509-8/19/$31.00 ©2019 IEEE

products that put a smile on their face when using the product. between usability tests and UX. Usability tests tend to focus on
In other words, users will choose products with great UX [19]. task performance whereas UX focuses on lived experiences
[28]. In the following subsections, two different perspectives in
III. UX EVALUATION UX evaluation posted in the RSs literature are expressed
briefly.
UX is dynamic and change in the different circumstances
by users’ contexts and emotional states, before and after an A. Objective UX (Cognitive Load)
interaction with a product [26]. In light of dynamic changes of
users’ goal and need, it is crucial to evaluate the RSs beyond Objective UX measures the motivations behind the users’
the static aspects and investigate the temporal aspects of UX to ratings [17]. In the other words, it is about the amount of the
know how and why experiences evolve over time [20]. required memory being used by the working memory of the
Moreover, users’ expectations and evaluation influence the UX user to achieve his/her goal [29]. The lesser the amount of
element. Therefore, it has to be considered from the beginning cognitive load, the more positive feeling is discerned by the
of the design process [17]. users while interacting with the system. Harper et al. (2005)
investigated motivations behind user’s rating behaviour in an
A comprehensive understanding of users’ experience, online movie RS. They constructed an empirical model to
regardless whether it is positive or negative feelings, is the formalize their initial understanding of the system and
essence of UX evaluation [25]. However, the main challenge in conducted survey to collect behavioral data. They found out
UX is that different people react differently to different that users perceive rating-time costs increase when they
situations. Apart from that, the same person may get different provided ratings. In another study, Sparling et al. (2011) study
feelings in the same situation depending on the context. The the mental cost and benefits on different rating scales upon
goal is, of course, to maximize the positive moments for users 12,847 movie and product review ratings collected from 348
when they’re using the system, and ideally to make the users users through an online survey. The authors proposed a rating
love the product, at least some or most of the time [22]. time approach to estimate cognitive load because user mental
A collection of UX definitions and evaluation methods costs is difficult to be measured accurately [17].
along with the relevant references are accessible at
http://www.allaboutux.org/ux-definitions. The review of all B. Subjective UX (Self-report)
methods is beyond of the scope of this paper and some of the Knijnenburg et al.’s proposed a framework for the
definitions are not applicable for RSs. In the following, only evaluation of UX of RSs. The framework aims to investigate
the methods that extent the understanding of UX evaluation the impact of objective system aspects like the user interface
and have been applied in RSs are discussed briefly. on the users' subjective perceptions and experience [18]. They
provided a questionnaire survey of 7-likert-scale statements to
The UX Curve [27] is one of the UX evaluation methods
measure the subjective system aspect.
This method reports the changes of users’ experiences with a
product over time and in long term. Figure 1 shows an example
of UX Curve. The Pn represents the positive feelings while the IV. UX MODELS
UX curve goes up. The Nn shows the negative feelings when Researchers indicated that UX is a subjective phenomenon.
the curve goes down. However, its impacts might be reflected by the users’
If the Curve’s ups and downs are repeated too much during observable behaviours [18]. Since the 1980s, several models
the process, it shows that the user’s feelings are not stable and and theories have been developed to illuminate the interactive
the user probably end up being pretty unhappy most of the time experience of using digital technologies [30]. Once researchers
when using the system. Another very important point is that discovered that usability does not account for subjective
one negative user experience may need dozen of good emotions, UX emerged to explain the personal experience
experiences to make the user satisfied again and sometimes the when a user is interacting with a product or a system and led to
user does not come back and even is not willing to use other a shift from designing for users to designing with users [31].
services or products from the same company [25]. This shift mostly brought concepts such as fun [32], pleasure
[33], aesthetics [34] and hedonic qualities [22] up to
understanding of UX. Among the existing models, several
models have taken into consideration UX as a cognitive
process that applied to evaluate changes in users’ perception
and judgement over time. “Sander’s Experience Model” [31,
35] as shown in Figure 2, postulates that experience is an
intersection of memories of the past, current experience, and
future dreams which is felt individually. Sander believes that
UX can be involved in the process of design once we have
access to people’s experiences (past, present and potential)
[35].
Figure 1: UX Curve
The UX curve helps to look beyond static aspects and to
evaluate UX changes over time. This is also the difference
 V. UX MODELS FOR RSS
As indicated earlier, recently researchers have
acknowledged that embedding the RSs methods into UX
impacts dramatically on the effectiveness of recommendations
for the users [16, 18]. In the field of RSs, UX is the delivery of
the recommendations to the user and the interaction of the user
with those recommendations which undoubtedly includes
algorithms embedded in the context of a certain application
[18].
Figure 2: Sander's Experience Model
There is a user interface quality assessment model
According to Sander’s model, by accessing what people
developed by Hartmann et al.’s [38]. This model presents
say, think, do, use, know, feel, and dream, four sources of
users’ background, goals, and task that have an impact on the
knowledge including explicit knowledge, observed experience,
tacit knowledge, and latent needs are achievable. system assessment as well as decision-making criteria
Consequently, the resonance with people is established by (usability, aesthetic, etc.). The overall system evaluation is also
having that knowledge [35]. influenced by decision-UX models for RSs. Zins and
Bauernfield [39] conducted a survey among users of two travel
Over time, the three levels of design theory are also RSs and an application for finding digital cameras. Then, based
considered as part of cognitive process of UX. This theory was on the survey results, they constructed a model for the UX of
proposed by Don Norman [36], who is most well-known for RS. The model displays how users’ satisfaction is influenced
his advocacy of user-centric design. He discussed that different by trust, flow, browsing behaviour, as well as how these in turn
design levels including visceral, behavioural, and reflective are affected by the personal characteristics. However, this
triggers different experience. The visceral reaction, immediate model does not explain how objective system aspects such as
and often beyond our control, is the one precipitated by the quality of recommendation affect UX which is a prominent
initial sensory scan of the experience. The behavioural limitation of their model. McNee et al. [40, 41] have proposed
experience is when the user is using the product and eventually an analytic model called Human-Recommender Interaction
there’s an experience beyond the initial one of using a product. (HRI). He accentuated the significant role of the context such
It’s the experience of association and familiarity when the user as users’ tasks on the UX of RSs. They also put emphasis on
is not holding the product but has feelings for it, and is able to the appropriate RS dialogue through the analysis of users’
put values on the product in retrospect. needs. McNee believes that if the recommender is going to
help users, it has to be designed based on the real-world
Mahlke & Thüring [37] proposed the CUE (Components of information-seeking tasks. Not considering users’ information
User Experience) model which consists of three UX needs and their background knowledge puts the
components; instrumental, non-instrumental, and emotional recommender’s designers into pitfalls [42].
reactions of the user. In CUE model, system properties, user
characteristics, and task/context influence interaction Xiao and Benbasat [43] conducted a vast and considerable
characteristics. In turn, interaction characteristics influence literature review of the business and marketing-oriented
perceptions of instrumental qualities and perceptions of non- research on RSs and then proposed a framework showing how
instrumental qualities, both of which lead to emotional certain characteristics of RSs such as type and process, as well
reactions. All three are antecedents of appraisal of the system as output design influence users’ trust and satisfaction. They
(Figure 3). considered personal and situational characteristics as the
moderator variables which influence users’ evaluation. Ozok et
al.’s [44] studies are mainly on RS usability to yield design
guidelines based on the users’ survey. The proposed guideline
mostly describe the impacts of specific system aspects on the
usability of RSs. However, their approach is mostly descriptive
which depends on the users’ opinions on RSs rather than
experimental results of a specific system. Pu and Chen's
framework [45] consists of four main dimensions as shown in
Figure 4. This framework relies on a user-centric approach to
RS evaluation and links user’s perception of quality to user’s
Figure 3: The integrated CUE model proposed by beliefs while user’s beliefs are antecedents of the users’
Mahlke & Thüring[37] attitudes that are antecedents of the behavioural intentions
(inspired by TRA). This framework brought a new approach to
Hassenzahl foreground the impact of hedonic attributes like user’s perception of recommendation quality that becomes the
pleasure in Hassenzahl et al.’s UX model [22], while all TRA- hallmark of this approach compared to the existing
related theories, only discuss pragmatic attributes. He indicates frameworks. However, the framework does not explain which
that the product evaluations in terms of appeal, pleasure, and factors influence user’s perceptions.
satisfaction is affected by perceptions of product characteristics
including pragmatic quality and hedonic quality [28].
 information plays an important role in the recommendation
because it can represent the status of people, places, objects
and devices in the environment. The results of the survey done
by [12] on existing studies have drawn the conclusion that UX
rarely receives attention in the field of SRSs and the concept is
new in this field of research. Besides, among the current UX
models for RSs, there is a lack for a framework, which
investigates empirically the impact of contextual information
on the UX of RSs in a real world by using real users. This
framework not only enriches our conceptual understanding of
how contextual information influences UX of RSs but also
serves as a foundation for further theoretical and empirical
investigations. Moreover, a better understanding of relevant
contextual information can also help researchers to design
effective context-aware RSs and reduce the complexity of data.
Figure 4: Pu and Chen's Framework of perceived
qualities of recommenders REFERENCES
Knijnenburg et al. disputed that to analyse the UX of RSs,
[1] P. Resnick and H. R. Varian, "Recommender systems," Communications
the accuracy of recommendations and consideration of other of the ACM, vol. 40, pp. 56-58, 1997.
aspects is also essential since measuring accuracy is [2] R. V. Ali, "Content-based recommender system for an academic social
insufficient on its own [18]. They advocated an evaluation network/Vala Ali Rohani, " University of Malaya, 2014.
framework that examines the influence of subjective system [3] G. Salton, "Automatic text processing: The transformation, analysis, and
aspects, such as recommendation and interaction towards retrieval of," Reading: Addison-Wesley, 1989.
objective user behaviours. Example of objective user [4] E. Rich, "User modeling via stereotypes," Cognitive science, vol. 3, pp.
behaviours include purchase and use, as something personal. 329-354, 1979.
They found that the subjective aspects such as perception of [5] G. L. Lilien, P. Kotler, and K. S. Moorthy, Marketing models: Prentice
quality, interaction usability, and appeal, have strong Hall, 1992.
correlation with users’ behaviours. Also, their experiments [6] B. Murthi and S. Sarkar, "The role of the management sciences in
showed why and how subjective system aspects bring about the research on personalization," Management Science, vol. 49, pp. 1344-
1362, 2003.
user experience of RSs. However, they have not elaborated the
[7] J. S. Armstrong, Principles of forecasting: a handbook for researchers and
subjective system aspects and the relationships with situational practitioners vol. 30: Springer Science & Business Media, 2001.
and personal characteristics. [8] M. J. D. Powell, Approximation theory and methods: Cambridge
university press, 1981.
[9] G. Adomavicius and A. Tuzhilin, "Toward the next generation of
recommender systems: A survey of the state-of-the-art and possible
extensions," IEEE transactions on knowledge and data engineering, vol.
17, pp. 734-749, 2005.
[10] Z. Dehghani Champiri, H. R. Jamali, E. Afshar, H. R. Jamali, and M. A.
Nematbakhsh, "A multi-layer contextual model for recommender systems
in digital libraries," in Aslib Proceedings, 2011, pp. 555-569.
[11] J. Beel, B. Gipp, S. Langer, and C. Breitinger, "paper recommender
systems: a literature survey," International Journal on Digital Libraries,
vol. 17, pp. 305-338, 2016.
[12] Z. D. Champiri, S. R. Shahamiri, and S. S. B. Salim, "A systematic
review of scholar context-aware recommender systems," Expert Systems
with Applications, vol. 42, pp. 1743-1758, 2015.
Figure 5: UX evaluation framework for the RSs [18] [13] Z. D. C. S. S. B. S. S. R. Shahamiri, "The Role of Context for
Recommendations in Digital Libraries," International Journal of Social
VI. CONCLUSION AND DISCUSSIONS Science and Humanity, vol. 5, 2015.
[14] J. Beel, S. Langer, M. Genzmehr, B. Gipp, C. Breitinger, and A.
Recent studies in the field of RSs emphasize on user-centric Nürnberger, "Research paper recommender system evaluation: a
evaluation of RSs which is beyond the accurate prediction [46]. quantitative literature survey," in Proceedings of the International
This is actually a paradigm shift in RSs research since prior to Workshop on Reproducibility and Replication in Recommender Systems
Evaluation, 2013, pp. 15-22.
that, all the researchers have been trying to develop more
accurate algorithms [15-17]. UX is affected by the limits of [15] S. M. McNee, J. Riedl, and J. A. Konstan, "Being accurate is not enough:
how accuracy metrics have hurt recommender systems," in CHI'06
human perception and the preconceptions of the individual. extended abstracts on Human factors in computing systems, 2006, pp.
Researchers of RSs state that factors such as personal and 1097-1101.
situational characteristics, which are mostly considered as [16] J. A. Konstan and J. Riedl, "Recommender systems: from algorithms to
contextual information, affect the UX with RSs [18]. They user experience," User Modeling and User-Adapted Interaction, vol. 22,
started to improve classical recommender methods by pp. 101-123, 2012.
modelling contextual information. In this regard, contextual
[17] T. Nguyen, "Enhancing User Experience With Recommender Systems extended abstracts on Human factors in computing systems, 2006, pp.
Beyond Prediction Accuracies," PhD Dissertation,The University of 1103-1108.
Minnesota 2016. [42] S. M. McNee, N. Kapoor, and J. A. Konstan, "Don't look stupid: avoiding
[18] B. P. Knijnenburg, M. C. Willemsen, Z. Gantner, H. Soncu, and C. pitfalls when recommending research papers," in Proceedings of the 2006
Newell, "Explaining the user experience of recommender systems," User 20th anniversary conference on Computer supported cooperative work,
Modeling and User-Adapted Interaction, vol. 22, pp. 441-504, 2012. 2006, pp. 171-180.
[19] R. Bernhaupt, Evaluating User Experience in Games: Concepts and [43] B. Xiao and I. Benbasat, "E-commerce product recommendation agents:
Methods: Springer Publishing Company, Incorporated, 2010. use, characteristics, and impact," MIS quarterly, vol. 31, pp. 137-209,
[20] B. P. Knijnenburg and M. C. Willemsen, "The effect of preference 2007.
elicitation methods on the user experience of a recommender system," in [44] A. A. Ozok, Q. Fan, and A. F. Norcio, "Design guidelines for effective
CHI'10 Extended Abstracts on Human Factors in Computing Systems, recommender system interfaces based on a usability criteria conceptual
2010, pp. 3457-3462. model: results from a college student population," Behaviour &
[21] D. Norman, The design of everyday things: Revised and expanded Information Technology, vol. 29, pp. 57-83, 2010.
edition: Basic Books (AZ), 2013. [45] P. Pu, L. Chen, and R. Hu, "A user-centric evaluation framework for
[22] M. Hassenzahl and N. Tractinsky, "User experience-a research agenda," recommender systems," in Proceedings of the fifth ACM conference on
Behaviour & information technology, vol. 25, pp. 91-97, 2006. Recommender systems, 2011, pp. 157-164.
[23] J. McCarthy and P. Wright, "Technology as experience," interactions,
vol. 11, pp. 42-43, 2004. VII. J. BEEL, C. BREITINGER, S. LANGER, A. LOMMATZSCH, AND
[24] E. L.-C. Law, V. Roto, M. Hassenzahl, A. P. Vermeeren, and J. Kort, B. GIPP, "TOWARDS REPRODUCIBILITY IN RECOMMENDER-
"Understanding, scoping and defining user experience: a survey SYSTEMS RESEARCH," USER MODELING AND USER-ADAPTED
approach," in Proceedings of the SIGCHI conference on human factors in INTERACTION, VOL. 26, PP. 69-101, 2016.
computing systems, 2009, pp. 719-728.
[25] C. Kraft, User experience innovation: User centered design that works:
Apress, 2012.
[26] A. P. Vermeeren, E. L.-C. Law, V. Roto, M. Obrist, J. Hoonhout, and K.
Väänänen-Vainio-Mattila, "User experience evaluation methods: current
state and development needs," in Proceedings of the 6th Nordic
Conference on Human-Computer Interaction: Extending Boundaries,
2010, pp. 521-530.
[27] S. Kujala, V. Roto, K. Väänänen-Vainio-Mattila, E. Karapanos, and A.
Sinnelä, "UX Curve: A method for evaluating long-term user
experience," Interacting with Computers, vol. 23, pp. 473-483, 2011.
[28] M. Hassenzahl, K. Eckoldt, S. Diefenbach, M. Laschke, E. Len, and J.
Kim, "Designing moments of meaning and pleasure. Experience design
and happiness," International Journal of Design, vol. 7, 2013.
[29] J. Sweller, "Cognitive load theory, learning difficulty, and instructional
design," Learning and instruction, vol. 4, pp. 295-312, 1994.
[30] J. Hart, "Investigating User Experience and User Engagement for
Design," 2015.
[31] F. S. Visser, P. J. Stappers, R. Van der Lugt, and E. B. Sanders,
"Contextmapping: experiences from practice," CoDesign, vol. 1, pp. 119-
149, 2005.
[32] A. Monk, M. Hassenzahl, M. Blythe, and D. Reed, "Funology: designing
enjoyment," in CHI'02 Extended Abstracts on Human Factors in
Computing Systems, 2002, pp. 924-925.
[33] W. S. Green and P. W. Jordan, Pleasure with products: Beyond usability:
CRC Press, 2003.
[34] N. Tractinsky, A. S. Katz, and D. Ikar, "What is beautiful is usable,"
Interacting with computers, vol. 13, pp. 127-145, 2000.
[35] E. B.-N. Sanders, "From user-centered to participatory design
approaches," Design and the social sciences: Making connections, vol. 1,
2002.
[36] D. Norman, "The design of everyday things: Revised and expanded
edition," Basic Books, 2013.
[37] M. Thüring and S. Mahlke, "Usability, aesthetics and emotions in
human–technology interaction," International Journal of Psychology, vol.
42, pp. 253-264, 2007.
[38] J. Hartmann, A. Sutcliffe, and A. D. Angeli, "Towards a theory of user
judgment of aesthetics and user interface quality," ACM Transactions on
Computer-Human Interaction (TOCHI), vol. 15, p. 15, 2008.
[39] U. Bauernfeind and A. H. Zins, "The perception of exploratory browsing
and trust with recommender websites," Information Technology &
Tourism, vol. 8, pp. 121-136, 2005.
[40] S. M. Mcnee, Meeting user information needs in recommender systems:
Proquest, 2006.
[41] S. M. McNee, J. Riedl, and J. A. Konstan, "Making recommendations
better: an analytic model for human-recommender interaction," in CHI'06