2008:43

DOCTOR A L T H E S I S

Johnny Wingstedt
Making Music Mean
On Functions of, and Knowledge about,

Making Music Mean On Functions of, and Knowledge about, Narrative Music in Multimedia
Narrative Music in Multimedia

Johnny Wingstedt

Luleå University of Technology

Department of Music and Media
2008:43

2008:43|:-1544|: - -- 08 ⁄43 -- 

Universitetstryckeriet, Luleå
 MAKING MUSIC MEAN

On Functions of, and Knowledge about,

Narrative Music in Multimedia

Johnny Wingstedt

Doctoral thesis

Luleå University of Technology

Department of Music and Media
Cover art: Jacob Svensson

ii
 To Fanny and Liang

iii
iv
 ABSTRACT

Wingstedt, J. (2008). Making Music Mean: On Functions of, and Knowledge

about, Narrative Music in Multimedia (Doctoral dissertation). Luleå University of
Technology, Department of music and media.

Narrative media music – music used for narrative purposes in multimedia such as
film and computer games – is often, especially for young people, the largest
source of daily musical experience. This PhD thesis is based on three articles, in
different ways exploring functions of, and knowledge about, narrative media
music. The overarching research question of the thesis is: ‘How can meaning-
making functions of narrative media music be described – and how are attitudes,
awareness and knowledge about such functions expressed through the different
modes of musical sound and speech?’ The first article discusses how the musical
underscore in narrative media achieves meaning in multimodal interplay with the
visuals and other available modes of representation. Three short film scenes are
examined from the perspectives of ideational, interpersonal and textual meanings.
Even if music in such situations usually does not attain a high degree of conscious
salience, it is clear that it often contributes meaning that is essential for the
understanding of the overall narrative. It is concluded that what we (think we) see
is often to a large degree determined by what we hear. The second article presents
the first part of a study, where 23 young participants (12-13 years old), using a
software tool, were given the task of adapting musical expression to make it ‘fit as
well as possible’ different visual scenes shown on a computer screen. They also
answered a questionnaire, asking about their musical training and media habits.
Numerical data from the trial sessions, representing differences in musical
expression, were analyzed statistically. The results indicated a strong degree of in-
group conformity and consensus, indicating knowledge of culturally available
functions and conventions of narrative media music. The third article describes
the second part of the study. Each participant was interviewed in a stimulated
recall situation where they commented and reflected on their own musical
expressions of their completed trials. From the analysis of the interviews,
examining the verbal expression of ideational, interpersonal and textual meanings,
five different types of statements could be discerned: the Unclear, Intuitive,
Associative, Analytical and Transformative types. These statements were also seen
as reflecting various aspects of Swanwick’s (1994) concepts of intuitive and
analytical knowledge. Combining the verbal statements with how each participant
musically had demonstrated conformity or non-conformity to narrative
conventions, contributed to a fuller and more nuanced account of their expressed
musical knowledge. The thesis concludes with a discussion of the learning
opportunities offered by narrative multimedia in the escalating media society and
of its implications for formal music education.

Keywords
Film music, Game music, Media music, Multimodality, Music education, Musical
knowledge, Narrative music, Social semiotics.

v
vi
 PREFACE

Having worked, since the mid-70’s, as a composer of music for various kinds
of multimedia (film, television, theatre, advertising etc), I have often had
opportunity to reflect on the reasons given by directors and producers for
wanting music as part of their productions. Quite frequently I’ve been told that
the sole reason is to add mood or more ‘feeling’ – or sometimes just that
‘music makes it much nicer’. These reasons are of course good and valid.
Music is exceptionally good at communicating on an emotional level, in
narrative multimedia providing everything from subtle and discrete moods to
the strongest passion and overwhelming feeling. Also, music, when used
appropriately, provides aesthetic dimensions that greatly enhance the viewing
and listening experience. As one producer once put it; ‘music simply
contributes to making films and computer games so much more enjoyable’.
Well, I think these points are quite obvious and true to most of us. But maybe
because they are so apparent, they can sometimes make it harder to look (and
listen) past the obvious. What is readily understood might be taken for
constituting the whole truth.

As will be apparent in this book, I view narrative media music as a powerful

means of communication on many simultaneous levels. Besides the above
mentioned features, it may also provide factual information, describe aspects
of the environment, guide and urge the listener on, make rhetorical comments,
provide continuity, define dramaturgical form and much, much more. When
watching films and playing computer games it is apparent that we have got
knowledge about such musical narrative functions, because we respond to and
react on them. It may however not always be clear to us that we have this
knowledge – we don’t know that we know.

During all these years of working as a media composer – and also, since more
than a decade, as a college lecturer of film scoring, jinglewriting and game
music – many questions have had time and opportunity to surface and
accumulate. These questions are typically concerned with the ‘what’ and
‘how’ aspects of musical narrative functions, for example; What is, and can be,
communicated through narrative media music – and how does such
communication take place? How do we, as listeners and users, make meaning
from music in a multimodal and narrative context? How is the musical
knowledge constituted, that we as audience-members or users need to
continually develop in order to make sense of a story multimodally told?
Questions such as these have over the years continued to inspire and fuel my
fascination of the subject of narrative media music. When the chance appeared
to turn this fascination into a PhD project, I naturally jumped at the
opportunity. This book is to be seen as an attempt to begin, at least in part, to
explore some of the questions.

The importance of context – the multimodal, narrative and situational, as well

as the social and cultural situatedness of communication, meaning-making and

vii
learning – is emphasized in this thesis. The very existence of this book is in
itself certainly a manifestation of contextual importance. It would not have
been possible without the help and support of the many individuals involved
in my different contexts of work, study and private life.

First of all I want to thank Professor Sture Brändström, my supervisor at the

Department of Music and Media in Piteå, Luleå University of Technology.
With all his impressing experience and knowledge, Sture has throughout this
project, steadily but gently and patiently, made sure than I’m on the track –
always providing constructive and encouraging feedback. Having spent most
of my doctoral studies as a long distance student, living in Stockholm, I am
very grateful for the confidence and trust that Sture has shown, which has
allowed for a very creative and inspiring working situation. It has been a really
good example of the kind of learning environment which in this thesis is
described as ‘a meeting-place for intuitive and analytical knowledge’. Warm
thanks also to Dr Jan Berg, my assistant supervisor. I’ve truly enjoyed our
interesting and stimulating conversations and discussions. Your keen eye and
sharp analyses have been of invaluable help in my process.

I’m also very grateful to the entire research group in Piteå, which has been a
melting pot – the environment where a large amount of the ‘intuitive-
analytical’ process has been taking place. Our seminars, papers, written
responses, discussions, chats, friendships, lunches, dinners, trips and
everything else over the years, have meant so much. Even if I haven’t been
able to participate in the Piteå research environment on a daily basis, I’ve
certainly felt part of it. Big thanks also to Luleå University of Technology and
the Department of Music and Media in Piteå for contributing to the financing
of my project, for enabling a formal collaboration with the Royal College of
Music in Stockholm, and for support and help in all kinds of matters over the
years!

A special thank you to Dr Tore West, Department of Didactic Science and

Early Childhood Education at Stockholm University, for the sharp, thorough
and penetrating reading of my manuscript at the so called ‘ninety-percent
seminar’. Your comments, feedback and discussion at that seminar were
invaluable, and greatly contributed to the improvement of this thesis.

I am very grateful to the Committee of Research and Development (FoU-

nämnden) at the Royal College of Music (KMH) in Stockholm, for awarding
me a grant, partly financing the last semesters of my doctoral studies. Thank
you also to KMH for help and support, and for the smooth collaboration with
the Department of Music and Media in Piteå. Especially big thanks to Bo
Westman, Head of the Department of Music and Media Production at KMH,
where I teach, for all your help and support. This has involved generously
allowing time for my ‘competence development’, as part of my teaching
position, to be allotted to my doctoral studies - thereby contributing to its
financing. It has also involved great considerateness, going out of your way to
adapt my teaching schedule to accommodate my studies and travels – and not
even once complaining about all the extra work this has given you. I really

viii
appreciate all your support, encouragement, friendship and personal interest in
my project over the years!

Thanks also to colleagues and friends at KMH, for contributing to a uniquely

creative and inspiring working environment. Big thanks to all my students
over the years, the value of your input, talent and creativity cannot be
overestimated! It is you guys that make it possible for me to continually ‘make
strange again’ what is already learned and mastered.

I also want to thank Interactive Institute, studio Sonic in Piteå, which was my
main working environment during the first part of my doctoral studies. It was
here the first versions of the project took form. Many thanks to studio director
Katarina Delsing, who suggested turning my ideas into a PhD project. My
work at Sonic allowed me to make frequent trips to Piteå, provided a
stimulating space for work, made it possible to attend various seminars and
conferences – and also enabled the development of the REMUPP software.
Big thanks to Mats Liljedahl for programming the REMUPP interface and to
Stefan Lindberg for composing its basic music examples. Thanks also to Jacob
Svensson, then a student of Computer Graphics at LTU in Skellefteå, for the
great 3D animations. And warm thanks to Dr Nyssim Lefford, for much
appreciated support and great discussions! Many thanks also to the 23
participants of the study, and to their teachers for allowing me into their
classrooms.

My thanks are also extended to the Swedish Research Council

(Vetenskapsrådet), for a grant that made possible the first part of this project.

Finally, and most important, I wish to thank my closest family. Fanny, my

wife, where would I be without your support, help, love and patience. Thanks
for keeping things in perspective, for being you. And Liang (Lulu), my
stepson, you’re simply the best – as you have always been. Thanks for being
such a continuous source of inspiration and joy, for reminding me to keep
looking to the future.

Stockholm, August 2008

Johnny Wingstedt

ix
x
 TABLE OF CONTENTS

Abstract................................................................................................... v
Preface.................................................................................................... vii
List of articles.......................................................................................... xiii

1. Introduction........................................................................................ 1
Organization of the book..................................................................... 3

2. Background........................................................................................ 5

3. Thesis aims and questions................................................................. 9

4. Related previous research................................................................. 11

Aspects of media music....................................................................... 11
Musical knowledge and learning......................................................... 12
Multimodal perspectives...................................................................... 18
Contribution of this thesis.................................................................... 19

5. Summary of the licentiate thesis and the articles........................... 21

The licentiate thesis.............................................................................. 21
Article 1............................................................................................... 22
Article 2............................................................................................... 23
Article 3............................................................................................... 24

6. Theoretical perspectives.................................................................... 27
Social semiotics and multimodality..................................................... 27
Some key concepts.......................................................................... 28
Metafunctions of communication................................................... 31
Multimodality and learning............................................................ 34
Learning as motivated design......................................................... 36

7. Discussion........................................................................................... 41
Methodological considerations............................................................ 41
Evidence of learning....................................................................... 42
On causality and sources of knowledge.......................................... 42
Aspects of transformation and transduction................................... 45

xi
 Musical narrative functions................................................................. 46
Narrative functions and communicational metafunctions............. 47
Interactivity and musical functions................................................ 48
Conditions and opportunities for learning........................................... 48
Metafunctional configurations versus intuitive
and analytical knowledge............................................................... 49
Aspects of ‘interest’....................................................................... 51
Learning and design....................................................................... 51
Learning principles of narrative multimedia.................................. 53
Educational implications..................................................................... 57
Pedagogy as design for social futures............................................ 59
Conclusion........................................................................................... 62

References.............................................................................................. 65

Article 1

Article 2

Article 3

xii
 LIST OF ARTICLES

A1 Wingstedt, J., Brändström, S. & Berg, J. (in press). ‘Narrative Music,

Visuals and Meaning in Film’. Visual Communication.

A2 Wingstedt, J., Brändström, S. & Berg, J. (2008). ‘Young Adolescents’

Usage of Narrative Functions of Media Music by Manipulation of
Musical Expression’. Psychology of Music, Vol. 36 (No. 2), pp. 193-214.

A3 Wingstedt, J., Brändström, S. & Berg, J. (2008). Making Meaning of

Media Music: Expressions of Knowledge about Musical Narrative
Functions. Manuscript submitted for publication.

xiii
xiv
1. INTRODUCTION

In the title of this thesis, Making Music Mean, ironically the word ‘mean’ has several
potential – meanings. Incidentally, several of these meanings are of significance for the
main themes to be explored and discussed in this book. First and foremost, this thesis is
about meaning – to mean, as in signify, imply, connote, convey, express, represent or
denote. When looking for synonyms to this verb it becomes apparent how elusive the
meaning of ‘meaning’ is. Examining the synonyms given above, suggested by New Oxford
Thesaurus of English (2000), makes it apparent how each of them carry with them
somewhat different connotations. From experience we know that this happens with most
words as they are set up for closer examination. And, indeed, any mode of representation,
not just language, will offer a spectrum of meaning potentials, possible meanings, rather
than just serving to maintain fixed and static denotations.

The ability of music to function as a representational mode – to ‘bear meaning’ – has been
much discussed over the years. As will be argued, especially in article 1 of this thesis,
narrative media music – music used for narrative purposes in for example film and
computer games – certainly has the ability to mean. It is not only able to mean, but also to
signify, imply, connote, convey, express, represent – and even denote. The meanings will
usually, just as with language, become clearer if the context is relatively clear. Articles 2
and 3 will discuss how such meanings are handled and expressed by the young participants
of a study. How they ‘make music mean’.

In this thesis, the concepts of meaning-making, communication, learning and knowledge

are seen as closely linked. To make music mean, to make sense of music, is central to the
process of forming musical knowledge. Aspects of musical meaning-making related to
knowledge and learning are discussed throughout this thesis.

Another connotation of the word ‘mean’ has to do with ‘importance’. To make music be of
significance, have an ‘input on’, to matter. Music is important in our lives and in society.
Much can be said about music’s aesthetical importance, its input on identity, its
significance for belonging and much more. Narrative music used as an underscore in film
or computer games is typically experienced on a subconscious and unreflected level. Still, it

1
obviously ‘works’. As described in article 1, narrative media music contributes many kinds
of meaning on several simultaneous levels. The visual representations in film or computer
games may be consciously salient, but much of what we (think we) see is determined by
what we hear. By making meaning out of the musical sound we make it important, even if
on a subconscious level. We make music mean, we make it significant. In other places of
this thesis, for example in article 3, it is also argued for the importance of formal education
to acknowledge music’s expanding narrative uses in the new media society. Due to the
increasing consumption of television, video and computer games in our daily lives
(Kulturrådet, 2008), narrative media music is often responsible for the largest part of our
musical experiences. Knowledge is needed, both for researchers, educators and students,
about the significance of narrative music. We need to make the narrative aspects of music
mean – also in school.

Music can also express meanness, sound mean, as in unkind, nasty, spiteful, cruel or
vicious. Actually, often when we in film or games find a character, relationship or situation
‘mean’ in this respect, it is not seldom the music that gives us this impression. In article 2,
it is described how the participants of the study created music to, sometimes, express
meanness. Their task was then to make the music (sound) mean.

Similarly, the objective could also be to make the music excellent, magnificent, superb or
awesome – which are other ways to more informally connote the adjective ‘mean’. When
listening to their own finished results, as described in article 3, some of the participants
found that they had made the music sound ‘really cool and mean’. Many more meanings of
‘mean’ can certainly be found, but the ones mentioned so far are maybe especially pertinent
here.

Central to the theoretical perspectives of this thesis, most importantly multimodality and
social semiotics (see chapter 6), is the notion of social agency in relation to sign-making.
Meaning-making is here seen as a process where signs are constantly newly made (Jewitt &
Kress, 2003). This process can be seen as being internal or external, i.e. the task is to make
signs for the ‘reader’ as well as for the ‘producer’. In music, the composer, performer and
the listener are all active in the constant making and re-making of meaning. This is akin to
what Small (1998) calls musicking. In the title of the thesis, this process is emphasized by

2
the use of the present particle (the ‘-ing’ form) for the word ‘making’. Making music mean
is an ongoing and agentive process that is constantly made and re-made.

Organization of the book

The entire PhD thesis consists of a previously published licentiate thesis (Wingstedt, 2005)
and of this book. This book is based on three articles, each discussing different aspects of
functions and knowledge of narrative media music. They are included in full length at the
end of the book. With an aim to provide an overarching and uniting framework for these
articles, the chapters following this introduction are organized as follows:

The next chapter provides a general background to the thesis, discussing aspects of the
escalating media society. In chapter 3, aims and research questions of the thesis and of the
individual articles are presented. Chapter 4 gives an account of related previous research,
including writings from a range of different disciplines focusing on aspects of narrative
media music, musical knowledge and learning, and multimodality. Possible contributions
of this thesis to the field of music education research are also discussed. Chapter 5 provides
a summary of the three articles and also of the previously published licentiate thesis. In
chapter 6, the theoretical perspectives of this thesis are introduced and discussed. Key
concepts of social semiotics and multimodality are presented, and aspects of multimodality
and motivated design in relation to learning and knowledge are considered. Finally, in
chapter 7, methods, results and conceptions of concern to the thesis are discussed. These
include methodological considerations, aspects of musical narrative functions, conditions
and opportunities for learning, and educational implications.

3
4
2. BACKGROUND

Looking at the different opening passages of the three articles making up the basis of this
thesis, their common theme is ‘the emergence of new media in contemporary society’.
Much has been written on this topic in recent years. Plenty more is to come. Writing about
it is one way of trying to make sense of its implications. New communication media are
revolutionizing modern society. Still, the very essence of the new technologies is
fundamentally timeless and human. It is about communication.

When it comes to music in contemporary society, new technology has become an

inseparable part of most aspects of it. New musical instruments appear, sometimes digitally
emulating traditional instruments such as pianos, drums or church organs – sometimes
introducing new sound worlds or performance techniques. New tools for notation,
composition, recording and music production are changing the ways music is created.
Technology is transforming and redefining how music is produced, performed and
distributed. But then of course, music and technology have always belonged together. Any
musical instrument, from the simplest drum to the complicated workings of a piano or the
theatre organ, is to be seen as a piece of highly specialized technology.

The biggest musical changes brought on by the emergence of the new technologies,
however, is probably how we are confronted with and employ music in our daily lives.
How we engage and interact with music as listeners, audience, users or consumers. Again,
the new media are maybe not providing new basic functions per se. We use music pretty
much for similar reasons as we always have – for play, work, recreation, activation,
relaxing, dance, ceremonies, drama and much more. Through music we can define,
maintain and express aspects of identity, relations, community and society (e.g. DeNora,
2000; Martin, 1995). Music is used to express mood and emotions, to signal, to symbolize,
to provide continuity, to emphasize, to soften. For as long as we know, music has been
combined with other modes of expression, such as movement, words, image or drama. The
list of musical uses and functions goes on and on.

Also with the use of new media the basic musical functions remain pretty much the same.
So far nothing is really new. What is new, however, is the availability, the plenitude,

5
variety, portability, adaptability and control offered today. In new media, this is true not
only for music but for all the kinds of expressions, primarily visual and aural, made
available. With music this means that we have opportunities to hear music for hours every
day – on radio, television and video; in supermarkets, train stations, gyms and shopping
malls; on computers, game consoles, portable media players and mobile phones etc. Some
of this musical supply we can control – chose to turn it on or off, select songs, adjust the
volume – some we cannot. At the same time our possibilities to hear different kinds of
music – a variety of styles and genres, music from different cultures or eras – is greater than
anytime before in history. By the use of the Internet, a multitude of music clips are just a
few mouse clicks away. An enormous range of channels on television and radio (often also
available on the Internet) offer all kinds of music to choose from. Being part of this massive
musical supply, narrative media music, as heard in films, computer games etc, often
provide a variety of musical genres.

New media also offer increasing possibilities for anyone, not only ‘music specialists’, to
adapt, manipulate, edit, control – and even compose – music. The fact that music is now
something that we can own and personalize encourages further control over its execution.
Opportunities and forms for what Small (1998) refers to as musicking are expanding. New
ways to musick includes making playlists on computers and media players. Making
playlists involves arranging music according to certain themes, genres or uses. It implies
full control over the order of the chosen recordings – the kind of control that was earlier the
responsibility of the composer or publisher, and later of the record company. Another
musicking device is the choosing, editing – and not seldom creating from ‘scratch’ – ring
tones for mobile phones. Still other, quite sophisticated but common uses, are choosing and
editing music as part of the soundtracks for home videos or slide shows of photos. New
software, often included when purchasing computers or other devices, enables and
facilitates various ways to handle sound and music. Music software designed for easy
music making by combining and arranging ready-made musical phrases and loops are
steadily becoming more frequent, an example being Garageband® which currently comes
with all Macintosh® computers. Some popular computer games are based on the idea of
playing an instrument or singing, such as with GuitarHero® or Singstar®.

In a sense, this tendency towards a more agentive relationship with music can be seen as
endeavouring to ‘reclaim’ the musical interactivity and variability that was lessened, or at

6
least changed, when sound recording was invented in the late 19th century. Before the
modern technologization of media, listening to music can be said to always have involved a
certain degree of interactivity and variability. A live music situation will to some degree
always respond to the ‘unique necessities of the individual time, place and people involved’
(Buttram, 2004, p. 504) and is never repeated twice exactly the same way. The advent of
sound recording technology revolutionized music. For the first time in history it was
possible to exactly duplicate a musical performance and to make it available for listening in
a context where it was not originally performed. Music became an object as much as a
process, and was accessible for uses beyond control of the creator or performer. The
repeatability of a musical performance became the norm, and made possible new listening
habits such as memorizing every nuance and inflection of a specific performance – or the
opposite, the use of casual background music with no active or engaged listening involved.
The increasing possibilities for musical interactivity, brought about by recent years’
evolvement of new media, are in this perspective not really anything basically new. It is
rather answering to our appeal for musical participation. However, the contexts and means
for interaction are somewhat changing compared to how it used to be.

The portability of recorded music has made apparent its ability for assimilating and
contributing to new contexts. Cook (1998) argues that music and musical meaning cannot
be separated from its context, it can only be re-contextualized. He doubts the notion of
absolute, or ‘pure’, music: ‘Pure music, it seems, is an aesthetician’s (and music theorist’s)
fiction; the real thing unites itself promiscuously with any other media that are available’ (p.
92). Hodge and Kress (1988), as quoted in article 1 of this thesis, state that ‘the context,
both the physical referents and the social conditions of semiosis, is decisive for
communication to occur’ (p. 39). Also, Cope and Kalantzis (2000a) argue that ‘all
meaning-making is in its nature multimodal’ (p. 211). Meaning emerges from the
interweaving of representational modes in dynamic interplay. New media provide extensive
resources for re-contextualization and pronounced multimodal communication. Various
resources for communication, such as written and spoken language, image, video, sound
and music are easily combined and realized. In such contexts the meaning making
processes and functions of music are strongly emphasized. There is really nothing new to
these kinds of musical functions either. As mentioned earlier, music has always been prone
to integration with other communicational modes. But with new communication media
such integrations are made readily available and become important aspects of our daily

7
musical experiences in ways that we have not seen before. Making meaning from music, in
combination with other modes of representation, becomes an important part of how we deal
with the massive flow of information that surrounds us every day.

These implications for music – the availability, the plenitude, the variety, the offered
control, the potentiality for multimodal combinations and the affordances for meaning-
making – bring with them drastically changing opportunities for musical learning. The
meaning-making affordances of new media contribute to shaping the world around us, as
Folkestad (2006) puts it, ‘as an alternative arena for knowledge formation and learning,
with its own well-developed and established forms’ (p. 144). This entails profound
consequences for music educators and researchers as well as for musicians and users of
music. Folkestad expresses a view that music education researchers, besides doing research
in the school classrooms, also need to look at issues related to the various forms of musical
learning that takes place outside schools. Sharing this view and taking as a starting point the
changing opportunities for learning and knowledge formation offered by new
communication media is what have inspired the questions driving this thesis.

8
3. THESIS AIMS AND QUESTIONS

The thesis is based on three articles, summarized in chapter 5 and found in full length at the
end of this book. It continues, develops and expands the themes commenced in my
licentiate thesis (Wingstedt, 2005). Together, these two books comprise the complete PhD
thesis.

The general aim of the licentiate thesis was to explore knowledge and use of musical
narrative functions in multimedia such as film and computer games. The overall aim with
the material introduced in the present book is to further explore use and functions of, and
knowledge about, narrative media music – and to put these issues in a wider
communicational, theoretical and learning-related perspective.

This aim is relatively complex and multifaceted. Its different facets are addressed more
narrowly in each of the three articles forming the basis of this thesis. These articles are each
based on their own specific purposes and questions, related to the overall aim.

Article 1, analyzing the music and its interaction with other modes of representation in
three different film scenes, is based on the following question:

How can media music, seen as a representational mode, be described to achieve

narrative meaning in multimodal interplay with other modes of representation?

Article 2 presents the first part of a study where a group of young adolescents adapted
musical expression to make it fit different visual scenes. Its main research question is:

How is knowledge of musical narrative functions and conventions expressed through

musical sound?

Article 3, presenting the second part of the study, where the participants are interviewed
about their musical efforts, is based on three specific questions:

How do the participants of the study verbally describe the musical expression and
meaning-making functions of their versions of narrative media music?

9

What awareness and knowledge about musical narrative functions and conventions can
be discerned from their verbal statements?

How do the verbal and the musical statements of the participants relate to each other?

One more question relates to the entire study presented in articles 2 and 3:

How are differences in experience and learning backgrounds – including informal as

well as formal situations and practices – related to the knowledge expressed?

10
4. RELATED PREVIOUS RESEARCH

The interdisciplinary approach of this thesis has resulted in its embracing, and adjoining
with, several different academically and practice oriented fields. These include primarily
music education, social semiotics and multimodality, but also touch on areas such as film
studies, music sociology, music psychology, technology, musicology and more general
media and communication studies. To attempt an exhaustive overview of all these fields
would be neither feasible nor very meaningful in this context. The purpose of this chapter is
rather to include a selection of works and studies that in various ways have turned out to be
of importance and relevance and to different degrees have informed the process of shaping
this project.

Aspects of Media Music

In recent years there has been an increasing interest in film music which is reflected in the
publication of many new books and articles on the subject. A ‘contemporary classic’ in the
field is Gorbman (1987), providing analytical and historical perspectives on music’s
narrative mechanisms and meaning-making functions in film. Another influential work is
Prendergast’s (1992) comprehensive introduction to the subject from an ‘insider’s’ point of
view. Mentioned should also be Karlin and Wright (2004) with their extensive and more
practically oriented work. Analyses of scores, and discussions of musical narrative
functions and aesthetics of film music are offered by for example Brown (1994), Donnelly
(2001, 2005), Kalinak (1992) and Lipscomb and Tolchinsky (2005). Mera and Burnand
(2006) present a selection of European approaches to film scoring, while Davison (2004)
explores the influence of Hollywood practice in relatively recent non-Hollywood scores.
Extensive historical overviews of the development and trends of (mostly American) film
music are provided by for example Darby and Du Bois (1999) and Hickman (2006).

French composer-critic-filmmaker Michel Chion (1994) is an influential contemporary

work, focusing not specifically on music but exploring functions and aesthetics of film
sound and its interactions with the visuals. In the field of musicology, Tagg (1979/2000)
with his original approach in analysing the main title theme of the television series Kojak,
and also Tagg and Clarida (2003), have contributed influential theories and methods on the
analysis of narrative media music. Cook (1998) provides further analytical models,

11
exploring the role of music in a wide range of multimedia including television advertising
and music videos as well as film and opera. In music psychology research, there has in
recent years been an increased interest in the area of ‘music and emotion’ (e.g. Juslin &
Sloboda, 2001). Examples of studies of psychological effects (related to emotion as well as
perception) of music in film include Cohen (2001), Lipscomb and Kendall (1994) and
Marshall and Cohen (1988). The field of social psychology has contributed studies on the
relationships between music, advertising and consumer behaviour (e.g. North & Hargreaves,
1997, 2005).

Musical Knowledge and Learning

From the perspective of music education, topics of special interest for this thesis include
‘expressions of musical knowledge’ and ‘learning opportunities in informal situations’.
Works relating to these topics from various perspectives will be presented in the following.

The analysis of the interviews made in article 3 of this thesis makes use of Swanwick’s
(1994) concepts of intuitive and analytical knowledge, in turn based on theories of Italian
philosopher Benedettto Croce (1902/1992). Swanwick does not discuss aspects particular to
media music, but more generally investigates the notion of musical knowledge – of music
as a way of knowing. Intuitive knowledge is described as being central to all knowledge
‘[making] possible all other ways of knowing’ (Swanwick, 1994, p. 28). It includes the
creative forming of images, the exercise of imagination, aspects of appearance and
impressions, the intuitive forming of aesthetic values. It is emphasized that intuitive
knowledge is ‘not a form of day-dreaming but an active way of construing the world’ (p.
28). Analytical knowledge concerns intellectual and reflective aspects of knowledge, the
forming of conceptions, the conscious understanding of underlying form, relationships and
tradition. The development of knowledge is described as an ascending spiral oscillating
between the two poles of intuitive and analytical knowledge. Thus, these two forms of
knowledge are not to be seen as separate, or ‘lesser’ or ‘better’, forms of knowledge but as
mutually interdependent and continuously interacting. Swanwick describes the intuitive and
analytical forms of knowledge as ‘mutually reinforcing energies that together generate
musical knowledge’ (p. 119).

In recent years there has been an increased interest in exploring how musical learning takes
place in informal situations. Focus is thereby, as Folkestad (2006) points out, shifted from

12
the process of teaching to the process of learning. Research in this area typically studies
situational contexts outside of school, where the primary intention is not necessarily to
learn about music but to ‘play music, listen to music, dance to music or be together with
music’ (p. 136). An important and influential example of this kind of research is Green
(2002), who has examined musical enculturation and learning processes of popular
musicians in informal settings, based on interviews with fourteen musicians living in the
London area. Amongst the rich results, her study suggests that young popular musicians
largely teach themselves to play music. This process involves, besides peer learning and
several other practices, solitary listening. She distinguishes between purposive, attentive
listening and what she terms distracted listening – ‘when the music is being attended to on
and off, without any aim other than enjoyment or entertainment’ (p. 24). As a part of this
type of listening she also includes hearing – when we are aware that there is music playing
but are barely paying attention to it. She suggests that all these different kinds of listening
are central to the learning practices involved in informal situations. In a later book she
extends this work and proposes methods for bringing informal learning practices of popular
musicians into the classroom (Green, 2008).

In Sweden, several studies have been presented in recent years exploring various aspects of
informal learning, often relating to popular music genres. This is maybe reflecting, as
Green (2008) puts it, that ‘Scandinavian countries are probably among the first and most
far-reaching in bringing popular music into the school curriculum’ (p. 3). A few examples
will be mentioned here. Lilliestam (1995) studied playing by ear and oral transmission of
musical knowledge among blues and rock musicians, where he described three major steps
in their process – listening, practicing and performing. Gullberg (2002) presents three
studies of socialization in music. The first study explored relationships between formal and
informal learning situations and strategies, having two different music groups – one with a
formal training background and the other in a non-formal tradition – arrange and record a
rock song. This was followed up with listeners’ comments and interviews of music students.
The results showed, among other things, how the participants’ respective backgrounds were
decisive for their way of working as well as for their musical preferences. Johansson (2002)
examined learning and playing strategies of ear players, giving them the task of playing
along with recordings of varying complexity. The informants afterwards also commented
on their own playing. Among the findings, two major types of musical strategies were
discerned: listening strategies and playing strategies. Söderman (2007) studied artistic and

13
educational strategies of hip-hop musicians, from different perspectives through several
different studies. Results indicate, among other things, collective creative processes where
lyrics are of central importance.

Making music by the means of new technology has been explored in several studies. A
relatively early one is Folkestad (1996), who investigated processes and strategies of
adolescents’ computer-based creative music making. Two main procedural categories were
identified, horizontal and vertical, which were related to the character of the music being
created. Nilsson (2002) studied children’s creative music making with digital tools. He
found that the children were able to produce music with form and structure, and he
describes different strategies used to achieve this. The results suggest that the computer
simultaneously opened up for the different kinds of mediation represented by orality and
literacy.

One feature that the so far mentioned studies have in common is that they all look at
situations and processes where the learners are actively involved in performing or
producing music. But being ‘together with music’ of course also involves situations where
music has other functions. Ericsson (2002) discusses how musical learning today often
seem to take place in informal situations, through what he calls preoccupied assimilation (p.
230), where the act of learning, so to speak, becomes unintentional. This is akin to Green’s
concepts of distracted listening and hearing, discussed earlier – the main difference being
that Green specifically discusses how popular musicians learn while Ericsson brings up
how musical learning and enculturation takes place for all of us in everyday situations.

Studies of learning in such casual situations, not involving the learners actually performing
music, are however still relatively sparse. Small (1998) introduced the term musicking,
emphasizing the active role not only of the musical composer or performer, but also of the
listener or other participants of a musical event. Batt-Rawden and DeNora (2005)
approached this concept in a study where focus was on musical learning in everyday life
situations. They point out how learning takes place through musicking, by various ways of
using music in daily life, but that ‘the skills and knowledge of that use may never surface as
consciousness’ (p. 292). Part of the task for the participants in their study was to learn how
to ‘musick’, to make the musical knowledge become conscious by the exchange of musical
selections on CDs – intended to promote well-being and health for the participants who all

14
suffered from some form of chronic illness. This approach was combined with interviews.
The participants experienced that learning to musick was a way to gain autonomy and
empowerment. The authors however point out that music is no ‘magic pill’ – its
‘affordances are constituted through the ways music is framed and prepared for use’ (p.
296).

Karlsen (2007) gives examples of the listener’s active participation and informal learning
processes in her in-depth exploration of the music festival as an arena for learning. The
festival’s influence on audiences’ construction of musical self-narratives, development of
parallel musical identities and contribution to development of local identity of the festival’s
host-municipalities is explored. Based on theories of situated learning, the author discusses
how learning is brought about through audiences’ peripheral participation in the festival
community of practice. Their festival-related learning can be expressed in terms of learning
music, learning about music and learning via music. Karlsen concludes that ‘it became
evident that the total learning outcome was similar to expected outcome from other
informal as well as formal music educational settings’ (p. ix).

In examining various research studies focusing on issues of formal and informal musical
learning, Folkestad (2006) identifies four different ways of using or defining these concepts,
either explicitly or implicitly. To differentiate the formal and informal aspects, each type of
definition focuses on different aspects of learning: (1) The situation – where the learning
takes place, inside or outside of institutional settings; (2) Learning style – the nature and
quality of the learning process, such as learning to play written music or by ear; (3)
Ownership – who ‘owns’ the decisions of the activity, with a focus on didactic teaching
versus self-regulated learning; (4) Intentionality – towards what is the mind directed,
towards learning (how to play) or towards playing. Folkestad points out how it becomes
important to clearly define how these concepts are being used in specific writings and
contexts, as the distinction between different ways of using ‘formal’ or ‘informal’ will
otherwise easily become blurred. However, these definitions are not seen as contradictory,
it is possible to use combinations of them. Folkestad concludes that the view of formal
learning as taking place only in schools and of informal learning as occurring only outside
school is too simplified: ‘this static view has to be replaced with a dynamic view in which
what are described as formal and informal learning styles are aspects of the phenomenon of
learning, regardless of where it takes place’ (p. 142).

15
Several studies in the fields of music sociology have explored and discussed the function of
music in our daily lives. These studies do not primarily focus on the learning aspects of our
interaction with music, but provide important perspectives on the situated nature of musical
experiences and meaning-making in informal settings. In doing this, they also contribute to
a wider understanding of the opportunities for learning that emerge in such settings. A few
examples will be given here. Martin (1995), in his pioneering book addresses issues
emerging from the links between musical cultures and the social contexts in which they
develop. He refers to a number of studies concerning a wide variety of musical genres,
arguing that musical meaning must be understood as socially constructed rather than
inherent. DeNora’s (2000) influential work on music in everyday life, discusses how music
is used in our daily lives as mood regulator, as a technology for identity, emotion and
memory, as a tool for activation and relaxation etc. She presents material building on in-
depth interviews and a series of ethnographic studies, concerning for example the use of
music in aerobics classes, in karaoke evenings or as background music in the retail sector.

In the last few years there has also been an increasing interest in how learning is constituted
by the casual use of computers and by playing computer games in informal situations. The
results and arguments presented in the available studies have so far not specifically
addressed the role of music or musical learning, but rather discussed more general aspects
of learning. These discussions however form an interesting and useful basis for further
explorations of more specific learning areas, such as music. In this respect they have
inspired and informed certain themes developed in this thesis. One of the most influential
writers in this field is Gee (2007). In exploring learning opportunities afforded by computer
games, he identifies and discusses 36 different learning principles that he suggests are
being built into good computer games. These learning principles and opportunities are
related to the dynamic multimodal configurations and interactive possibilities available in
games, to the playing situation, narrative and semiotic concepts, aspects of identity and the
functions of affinity groups. Other examples of related works are Buckingham and Willett
(2006), a compilation exploring issues of gaming, the Internet, online communities, and
learning and education from various perspectives; Prensky (2006), arguing for the
importance of video and computer games in preparing young people for life in the 21st
century; and Shaffer (2006), discussing how computer games can help students think like
innovative professionals, such as engineers, urban planners, journalists and lawyers.

16
Jorgensen (1997) discusses various aspects of music education and proposes a global and
holistic view, making it ‘important to broaden our view to include a plethora of instances
and approaches besides school music that may also count as music education’ (p. xi). She
explores several conceptions of music education that have been evident historically, and
defines five concepts: schooling, training, eduction, socialization and enculturation. The
schooling concept focuses on institutional aspects of formal education. It can also be
understood figuratively to the undergoing of some sort of discipline whereby one is
‘formed’ or ‘patterned’. Training refers to the ways whereby a person is taught or learns
skills in a profession or practice, focusing on method rather than content – ‘know-how’
rather than ‘know-that’. The word eduction means to draw out, elicit or develop. It is used
to illustrate the idea that every person is seen as having the desire and potential to learn if
the conditions are right. This potential can be drawn or brought out in an environment that
stimulates and inspires learning. Socialization focuses on how a person, in becoming a
member of a group, is socialized into sharing the beliefs and values of the group. Musical
socialization takes place by participating within the domains or practices that constitutes
being for example a violinist or a rock musician. Enculturation can be seen from an
anthropological or an idealistic perspective. As the life-long and dynamic process through
which people acquire a personal and collective identity – or as that towards which humanity
strives. Enculturation can be understood as a wider concept than socialization, and
comprises the concepts of transmission and acculturation, two processes that can
sometimes be seen as being conflicting. Transmission is described as ‘the acquiring of
culture through the passage of wisdom from one generation to another’ (ibid., p. 24).
Acculturation can be understood as a process where a certain culture is influenced and to
some extent changed by another culture.

Jorgensen (ibid.) concludes that each of these five ways of conceptualizing music education
‘contributes to our understanding but is lacking in one respect or another’ (p. 29). When
taken together however, she proposes that the different concepts will provide a broader
view of music education that goes beyond the traditional concept of school music to include
the activities of other societal institutions and practices. She emphasizes that music should
be studied as a world phenomenon, holistically and contextually. Folkestad (2006),
commenting on Jorgensen’s five concepts, suggests that the first two (schooling and
training) might be seen as descriptions of formal learning situations. The last two

17
(socialization and enculturation) might similarly be seen as descriptions of informal
learning. The middle concept, eduction, is proposed by Folkestad as ‘the meeting place for
formal and informal learning’ (p. 139).

Multimodal Perspectives
The theoretical perspective of this thesis draws from theories of social semiotics and
multimodality. Communication and learning are here seen as socially situated and taking
place through a wide range of culturally available representational modes (e.g. Hodge &
Kress, 1988; Kress & van Leeuwen, 2001; van Leeuwen, 2005). In this framework, music
has so far not been much discussed, with the exception of van Leeuwen’s (1999)
exploration of the communicative aspects of speech, music and sound. Using a wide range
of examples he describes the elements of sound as semiotic resources rather than as fixed
codes. The theoretical aspects of social semiotics and multimodality will be discussed more
in depth in chapter 6.

Researchers from various educational fields have lately taken up these perspectives and
used them as platforms for descriptions and analyses of educational issues, such as
classroom interaction and design, textbooks and teaching materials or student work as
expressions of knowledge or evidence of learning. Some examples of educational subjects
that have been explored are: science education (Kress, Jewitt, Ogborn & Tsatsarelis, 2001),
mathematics (Jewitt, 2003), bilingual language education (Kenner, 2003), English (Kress et
al., 2005), and video and media education (Burn & Parker, 2003; Lindstrand, 2006). A first
Swedish compilation involving multimodality, and also the concept of ‘design for learning’
in a number of educational subjects, has also recently been released (Rostvall & Selander,
2008).

In the field of music education, however, the use of multimodal and social semiotic
perspectives have not yet quite been established as means for exploring aspects of musical
learning, at least not in Scandinavian research. An exception is the work of Rostvall and
West who have developed tools and methods for multimodal analysis used in their studies
of classroom interactions of instrumental music teaching (Rostvall, 2008; Rostvall & West,
2001, 2005, 2006; West, 2007). By examining and analysing music and gesture as well as
language they have studied how different interactional patterns during instrumental music
lessons affect students’ as well as teachers’ opportunities to learn. By the use of a

18
multimodal approach, findings from several studies show how contradicting messages from
teachers contribute to confusion. Also, during the lessons it became apparent how music
was often broken down into separate notes, rather than as coherent musical phrases,
rhythms, or melodies, and expressive qualities of music performance were not addressed.
The interaction was typically found to be asymmetric with the teachers in control, leaving
little agency to the students.

Contribution of this Thesis

The different themes presented in this chapter indicate the conceptual framework and
research context of the present PhD thesis. The articles of the thesis do however also touch
on some issues and topics where an existing strong research context has been difficult to
establish. Hopefully, such issues and topics will turn out to be areas where this thesis can
make a contribution. One such area would be that of the role of narrative musical functions,
how they contribute to our meaning-making in multimedia such as film and computer
games. As is argued elsewhere in this thesis, knowledge about these functions is becoming
more important as narrative media music increasingly takes more space in our daily lives.

Another area is that of the informal musical learning that takes place when conscious
attention and focus is directed towards other things than the music. This could be seen as an
extension of previous and ongoing research on learning in informal settings.

A third area concerns the study of music from multimodal and social semiotic perspectives.
As mentioned, music as a representational mode has so far not been much explored within
this framework. Conversely, the theoretical perspectives of multimodality and social
semiotics have not been much explored in the field of music education. Hopefully, this
thesis will be followed by more studies where aspects of musical expression,
communication and learning, in relation to this theoretical framework, can be further
developed and deepened.

19
20
5. SUMMARY OF THE LICENTIATE THESIS AND THE ARTICLES

The three articles that make up the basis of this thesis are, as mentioned earlier, included in
full length at the back of this book. As they will be frequently referred to in the following
chapters, a summary of each of them will be made in this chapter. Also, since my licentiate
thesis (Wingstedt, 2005) is also part of the complete PhD thesis, a summary will first be
made of its contents.

The Licentiate Thesis

The licentiate thesis, Narrative Music: Towards an Understanding of Musical Narrative
Functions in Multimedia (Wingstedt, 2005), was presented and defended in November
2005. The overall aim of the licentiate thesis was to explore use and knowledge of musical
narrative functions as they appear in multimedia such as film and computer games. The
thesis is based on three publications:

Publication 1, Narrative Functions of Film Music in a Relational Perspective (Wingstedt,

2004), proposes a classification of musical narrative functions, with six narrative classes
(the Emotive, Informative, Descriptive, Guiding, Temporal and Rhetorical classes) and
eleven sub-categories. The relational interplay of music with contextual factors is
emphasized.

Publication 2, REMUPP – An Interactive Tool for Investigating Musical Properties and

Relations (Wingstedt, Liljedahl, Lindberg & Berg, 2005), describes the design, functions
and possible uses of a software tool, REMUPP (Relations between Musical Parameters and
Perceived Properties), to be used for studies of musical expression. REMUPP allows for
real-time alteration of musical expression by the manipulation of musical parameters such
as tempo, harmony, rhythm, articulation, etc. Information acquired with this software can
be output as numerical data for statistical analysis, and the musical results and processes
can also be played back for analysis by more qualitatively oriented methods.

Publication 3, Young Adolescents’ Usage of Narrative Functions of Media Music by

Manipulation of Musical Expression, is an early version of the publication that is included
as article 2 of the current book. Its content will be described later in this chapter.

The licentiate thesis also contributes brief discussions of various topics related to the
concept of narrative media music, including discussing the music concept from a relational

21
and contextual perspective, aspects of multimodality and narrativity, the question of
convention versus creativity and accounts of musical interactivity in new media.

Article 1
The first article of this book, Narrative Music, Visuals and Meaning in Film (Wingstedt,
Brändström & Berg, in press), explores how the musical underscore in narrative media
achieves meaning in multimodal interplay with the visuals and other available modes of
representation. Using Halliday’s (1978) metafunctions of communication as a starting point,
three short film scenes (from Jaws, The Secret of My Success and The Birds) are examined
from the perspectives of ideational, interpersonal and textual meanings. These
metafunctional meanings are also put in relation to Wingstedt’s (2004, 2005) musical
narrative functions: the emotive, informative, descriptive, guiding, temporal and rhetorical
functions.

The Jaws scene is given an in-depth analysis, examining the use and functions of the
leitmotif. On the ideational level especially its informative and descriptive functions are
discussed. Interpersonally, the emotive and indicative (guiding) musical functions are
looked at, as well as aspects of ‘truth’ (modality) and the relationship of ‘producer’ and
‘audience’ in film. On the textual level, intramodal as well as intermodal structures and
relationships of the musical sound are examined. Aspects of musical placement, timing,
synchronicity, continuity and form are discussed. In the scene from The Secret of my
Success, intertextual issues and the use of music’s rhetorically commenting functions are
illustrated. In connection to the scene from the Birds, various implications of using diegetic
music (music that is part of the spatio-temporal world of the story told, ‘heard’ by the
characters of the story) are discussed.

The examples illustrate how musical meaning emerges from the interweaving of
representational modes in dynamic interplay, and how such meanings can be more or less
specific depending on the ambiguity of the context. Wider and more general meaning
potentials of musical expression become specific in situational and narrative contexts. Even
if music in such situations does not attain a high degree of conscious salience, it is clear that
it often contributes meaning that is essential for the understanding of the overall narrative.
It is concluded that what we (think we) see is often to a large degree determined by what
we hear.

22
Article 2
The second article, Young Adolescents’ Usage of Narrative Functions of Media Music by
Manipulation of Musical Expression (Wingstedt, Brändström & Berg, 2008b), presents the
first part of a study. An early version of this article was included in the licentiate thesis. It
was however decided to include it also in this book. There are two main reasons for this: (1)
In the process of publication in the journal Psychology of Music, the article was modified
and improved. It was determined to be of value for the dissertation to include this revised
and final version. (2) The study described is of central importance to this thesis. Moreover,
the second part of the study is described in article 3. To make a complete description of the
entire study available in the book, the inclusion here of this article was regarded as
necessary.

Using a software tool (REMUPP), 23 young participants (12-13 years old) were given the
task of adapting musical expression – by manipulating seven musical parameters – to make
it ‘fit as well as possible’ different 3D-animated visual scenes (movies) shown on a
computer screen. The interface was designed to allow the users to directly control musical
expression, not requiring any previous musical training. They also filled out a questionnaire,
answering questions about their musical training and media habits (amount of casual music
listening, watching video and playing computer games). In analysing the results of the trial
sessions using the software, the settings of the musical parameters were available as
numerical values. Thus, differences in musical expression could be described and analyzed
statistically. This data was combined with the results from the questionnaires.

The results indicated that the participants displayed a strong degree of in-group conformity
and consensus regarding the musical expressions created for each of the movies. This can
in turn be seen as to a certain degree mirroring knowledge of existing culturally available
narrative functions and conventions that we encounter in film, computer games and other
multimedia. The results also showed a correlation with the participants’ gender and
different backgrounds of musical training and media habits. Participants learning to play an
instrument or doing much recreational music listening generally preferred more complex
and expressive musical structures compared to the ones with less musical experience. Also,
participants playing comparatively more computer games and watching more film generally
expressed more knowledge of musical narrative conventions (and a willingness to follow

23
these conventions) compared to those with less experience of narrative media. One of the
conclusions of this part of the study is that the software tool used has allowed the
participants to achieve meaning through ‘musical action’, which is different from using
language. The interface has thus made visible implicit knowledge about musical narrative
functions and conventions.

Article 3
The third article, Making Meaning of Media Music: Expressions of Knowledge about
Musical Narrative Functions (Wingstedt, Brändström & Berg, 2008a), describes the second
part of the study. After having completed the test trials, described in article 2, each
participant was interviewed in a stimulated recall situation where they commented and
reflected on their own musical expressions as their completed trials were played back to
them. Audio recordings of the interviews (and the associated music examples) – where the
trials were discussed, compared and evaluated – were then transcribed, analysed and
matched with the corresponding visuals. Two categories of participants were selected as
being of special interest for the analysis: Conformers (the participants whose musical
expressions most closely conformed to the in-group convention) and Non-conformers (the
participants whose musical expressions deviated the most from the in-group convention).

Halliday’s (1978) metafunctions of communication were taken as a starting point for

making the analysis. It was apparent how the individual participants all established distinct
strategies when they discussed the musical and narrative features of each trial scene. In all,
five different types of verbal statements could be discerned: the Unclear, Intuitive,
Associative, Analytical and Transformative types. The Unclear type reflects the speaker’s
hesitation or uncertainty about the music’s function or impression, while the Intuitive type
is constituted by distinct evaluative statements about aesthetical or narrative functions of
the music, concerning how well the it ‘works’ – however without any further explanations
or descriptions. With these two types of statements, the interpersonal relationship of the
speaker towards the music and narrative situation is emphasized. In comments of the
Associative type, associations to observed emotional content or descriptions of narrative
events are offered. In this type of statements, the ideational (content) meaning of the music
and scene is foregrounded. Analytical statements include observations of musical structures
or performance factors contributing narrative meaning. This type of comments emphasizes
the intramodal aspects of the music, i.e. the inner structures of the musical sound.

24
Transformative statements typically suggest alternative structural or narrative possibilities
for the music, emphasizing the textually intermodal aspects of the music.

Statements of the Unclear and Intuitive types are seen as reflecting Swanwick’s (1994)
concept of intuitive knowledge. The Analytical and Transformative comment types are
primarily reflecting what Swanwick calls analytical knowledge. Associative statements
may be seen as essentially expressing intuitive knowledge, but are also in a way bridging
and connecting the intuitive and analytical domains. Swanwick describes intuitive and
analytical knowledge as interdependent and interwoven, as mutually reinforcing energies.

As most of the participants were fairly consistent in their types of comments used, they can
be looked at as showing either primarily intuitive or analytical profiles. Combining these
verbally based profiles with the Conformer and Non-conformer categories, which are based
on the musical expressions of the participants, results in four positions of ‘multimodally
joined’ statements: Analytical-Conventional, Analytical-Unconventional, Intuitive-
Conventional and Intuitive-Unconventional. As the different modes of speech and musical
sound express different aspects of knowledge and awareness, when put together they
provide a fuller and more nuanced manifestation of the participants’ expressed knowledge.

The design of the study does not provide access to the actual learning process of the
participants. What is available for study is rather what Jewitt (2006) refers to as ‘evidence
of learning’. Given the design and learning opportunities offered by new narrative media,
its abundance and the amount of time spent using it on a daily basis in contemporary
society, it seems likely that some learning about its functions (including musical functions)
is achieved by simply using it. It is concluded that we need, both as educators, researchers
and users of music, to relate to the impact of the escalating media society. In order to find
strategies for this, more knowledge is required.

25
26
6. THEORETICAL PERSPECTIVES

This thesis is about music as communication, socially and culturally situated. It deals with
aspects of musical meaning-making, looks at music as a mode of representation and
examines the narrative roles of music in multimodal interplay with other expressional
modes. It discusses aspects of musical knowledge and learning in informal settings and also
explores what can be seen as indication or ‘evidence’ of learning (Jewitt, 2006, p. 28),
expressed verbally as well as through musical sound.

Social Semiotics and Multimodality

As the three articles of this thesis provide limited room for in-depth theoretical discussions,
in this chapter the theoretical perspectives of the thesis will be more thoroughly examined.
In exploring the themes and topics mentioned above, theories of social semiotics and
multimodality, as formulated by Kress, van Leeuwen and others, are taken as a starting
point (e.g. Hodge & Kress, 1988; Kress & van Leeuwen, 2001, 2006; Jewitt & Kress, 2003;
van Leeuwen, 2005). This is already to some extent made apparent in articles 1 and 3,
where certain aspects of these theoretical perspectives are being discussed. The format of
Article 2, mainly focusing on the statistical analysis of quantitative data of the study,
provides less opportunity for an extended theoretical discussion. The exploration of musical
expression by adapting musical parameters was here seen as a useful way to describe
textual1 aspects of the musical sound. By the subsequent integration of the statistical results
into article 3, the issues explored in article 2 are seen as aptly contributing to the overall
theoretical perspective. The theoretical framework to be discussed here is therefore to be
seen as relevant to the entire thesis. In order to view the theoretical concepts in an
appropriate contextual perspective, they will in the following be illustrated with examples
relating to the articles and to various aspects of music.

Van Leeuwen (2005) describes social semiotic theory as a form of enquiry, offering ideas
for formulating questions rather than offering ready-made answers. At the same time it has
been described as being simultaneously a theory of communication, a theory of
representation and a social theory (Lindstrand, 2006). Since these three dimensions of
music are of particular interest for this thesis, social semiotics has emerged as a pertinent

1
The textual metafunction will be discussed later in the section ‘Metafunctions of communication’.

27
and useful choice of theoretical perspective. Another central topic of this thesis is that of
knowledge and learning. Viewing learning as socially situated and closely related to aspects
of representation, multimodality and meaning-making, social semiotics provides a starting
point for explorations and enquiry in ways that will be further discussed below. Social
semiotics as discussed here is founded on Halliday’s social semiotic theory of language
(Halliday, 1978, 1985). Its central idea is that language is social, it is the way it is because
of the social functions it has come to serve in people’s lives. Jewitt (2006) formulates it:
‘Rather than seeing language as a ready-made code, social semiotics understands language
as a result of people’s constant social and cultural work’ (p. 3).

‘Multimodal social semiotics’, extends Halliday’s theory of language into a semiotic theory
where meaning is understood to be realized not only in language but in a range of
representational modes, such as for example image, gesture or music. A mode is broadly
described as the effect of the work of culture in shaping material into resources for
representation and communication. Jewitt and Kress (2003) describe multimodality as the
field in which semiotic theory is applied – ‘its field of application’ (p. 9). Looked at in this
way, social semiotics and multimodality will be closely related and interdependent.
Hereafter, when either multimodality or social semiotics is referred to, this interdependent
relationship will be implicit. In the following sections, there will first be a description of
some key concepts of social semiotics and multimodality. After that, a multimodal view of
learning will be discussed, including the concept of learning as motivated design.

Some Key Concepts

The notion of modes, as mentioned above, is central to multimodality. Jewitt and Kress
(2003) describe it as follows:

Modes (of representation) carry the meanings of material affordance shaped by generations
of the work of people in their social lives. Over time this gives rise to a resource with
regularities shaped by ‘convention’, understood by members of a culture, and useable by
them for representation and communication (p. 13).

Each mode has its own distinctive affordances and constraints for representation, which are
both socially, culturally and materially determined. Thinking of music as a mode, its
endless manifestations of dynamically shifting appearances, genres, functions and uses in
different cultures and eras are to be seen as expressions of this kind of cultural work. Each

28
mode individually bears meaning, but in the communicational process meaning typically
emerges from the interweaving between and across modes within a multimodal system. In
article 1, it is explored how meaning emerges from the multimodal interplay of music and
image in film. Modes can be realised in more than one production medium. The concept of
media is in this context referring to technologies for making and distributing meanings as
messages, for example recordings of sound distributed on CD.

All modes are seen as being shaped through their social use into a range of semiotic
resources of meaning. Van Leeuwen (2005) describes semiotic resources as having ‘a
meaning potential based on their past uses, and a set of affordances based on their possible
uses, and these will be actualized in concrete social contexts’ (p. 285, my italics). Jewitt
(2006) points out how the focus on semiotic resource rather than code is a key distinction
between social semiotics and traditional semiotics. The notion of a code implies a pre-
existing and fixed set of rules for associating signs and meanings. The concept of semiotic
resources offers a different approach of thinking about semiotic systems, where social
processes are emphasized. People express and make meanings ‘through their selection from
the semiotic resources that are available to them in a particular moment’ (ibid. p. 18). As
illustrated in article 1, musical meanings can be more or less specific depending on the
ambiguity of the context. For example in the relatively distinct narrative context of the
movie Jaws, using a low register for the leitmotif is likely to imply ‘large size’, which
might not at all be apparent in a more ambiguous context. Also, potential meanings will
shift depending on the contexts involved. When the Jaws’ leitmotif is used in another
movie, the effect can be experienced as contrasting or humorous.

A sign, according to social semiotics, is seen as an instance of the use of a semiotic

resource for purposes of communication (van Leeuwen, 2005). In traditional semiotics, the
relationship between form (signifier) and meaning (signified) is described as arbitrary and
sustained by convention (Saussure, 1986). In a social semiotic perspective however, the
assumption is that this relationship is never arbitrary but always motivated by the interests
of the sign-maker to find the best possible form for expression of meaning (Kress, 1993).
People use the resources that are available to them in the specific social and cultural
environments in which they act to create signs.

29
The producer of the physical sign as well as anyone who engages with a sign are seen as
sign-makers – in ‘reading’ (or listening), the task is also to make signs. In the process of
communicating, signs are thought as being constantly newly made. The sign-maker is
always involved in finding the most apt signifier for what she or he wants to express or
represent in a particular context, rather than just applying pre-set rules (Jewitt, 2006). In
using the resources, they are also changed.

Representation is always considered to be partial, as it is that which is central to the sign-

maker at a particular moment (that of interest) that is represented. Signs as well as the
interests of people are seen as socially situated. Convention is realized as ‘people work with
the semiotic resources that are available to them in the social contexts that they live in’
(Jewitt, 2006, p. 21). Thus, the agency of the sign-maker is highlighted at the same time as
the choices made are socially constrained. What is represented is a result of past uses
(meaning potential) as well as possible uses (affordances).

These uses are determined by social, cultural and historical aspects – but also of the
materiality of a mode. Materials have inherent qualities or affordances that suggest cultural
and semiotic use. Materiality is in itself a potent factor in meaning. Van Leeuwen (1999)
describes how one fundamental aspect of the materiality of sound is how it is a ‘wrap-
around’ medium. We don’t have to turn towards a sound source to hear it, like we need to
do to see an image. Sound comes to us from all sides (even if we can often discern the
direction of the source) it is immersive and unifying. We also know from experience that all
sounds are the result of some kind of immediate physical action. The making of sound
requires activity. Air has to be moved in order for sound to be produced and thus physical
action is required, if only to move a loudspeaker’s membrane (vision of course works
differently, light itself taking care of the ‘movement’). We are therefore disposed to
associating sound with its origin, which affects the meaning potential. For example, we
know from experience that a loud sound requires more physical energy to produce
compared to a quiet sound. We intuitively understand that small objects cannot produce
sounds in a low register. It can aurally be determined if the sound is produced by a hard or
soft object and so on. In this sense, there is always a multimodal aspect to sound, which
affects how we make meaning from it. A meaning potential derived from what we
physically do when we produce a sign is referred to as experiential meaning potential
(Kress & van Leeuwen, 2001, p. 10).

30
Metafunctions of Communication
‘Semiotic resources of modes are shaped by how people use them to make meaning – the
social functions that they are put to’ (Jewitt, 2006, p. 18). Halliday (1978) classified these
social functions into three metafunctions, to be seen as three different kinds of meaning.
They are, according to Halliday and Matthiessen (2004), based on the simple notion that (1)
every message is about something, (2) it addresses someone, and (3) it does so through the
construction of a ‘text’. Using Halliday’s terminology, the three kinds of meaning are
known as the ideational, the interpersonal and the textual metafunctions, respectively.
These metafunctions are described, and used as a point of departure for the analyses made
in articles 1 and 3. They will also be described below, to facilitate the understanding of
their relationship to other aspects of social semiotic theory. In multimodal theory, this
system of metafunctional meaning is seen as being applicable to all modes, not only
language. The metafunctions can be applied to different modes individually as well as to
multimodal ensembles of several modes available in a communicative situation (Kress et al.,
2001). A modal expression (a ‘text’ or actualized meaning potential) is an instantiation of
all three metafunctions simultaneously and dynamically interwoven. According to Kress et
al. (2001) they are to be seen as ‘general requirements of any human communicational
system’ (p. 4).

The ideational metafunction is about the part of communication that tells us something
about the world – the world we live in or a fictional world, such as in a movie. According
to Halliday (1978) the ideational metafunction represents the communicator’s meaning
potential as an observer. It is the content function of communication, expressing
phenomena of the environment: the things, creatures, objects, actions, events, qualities,
states and observed relations – of the world and of our own consciousness. Kress et al.
(2001) further describe the ideational metafunction as representing material, verbal, mental
and (observed) relational processes – ‘who does what, with or to whom and where’ (p 13).
In the case of narrative media music, as described in article 1, ideational meaning is for
example salient in how music contributes information. This information can be about an
implied time period, cultural setting or social status – e.g. by the use of associated musical
genres. Music can also use leitmotifs to ideationally represent characters, relationships or
other phenomena of a story. It can provide information about (observed) emotions, clarify

31
visually ambiguous situations and so on. Narrative music can also be used to more actively
describe physical attributes and settings, it can illustrate physical movement etc.

The interpersonal metafunction is about the part of communication that positions us in

relation to someone or something (Jewitt, 2006). It is the participatory function of
communication, communication as doing something. It establishes and specifies
relationships, addresses solidarity and power, performs statements (declaratives),
imperatives and questions (interrogatives) etc. It is the component through which the
speaker intrudes herself into the context of situation, both expressing her own attitudes and
judgments – and seeking to influence the attitudes and behaviour of others (Halliday, 1978).
It establishes, maintains and specifies relationships between members of societies or groups
through expression of social relations, interrelations of power and knowledge (Kress et al.,
2001). Narrative media music articulates interpersonal meaning by for example indicative
functions (exemplified in article 1), where the use of expressional devices like accents
(‘stingers’) or phrasing can ‘point out’ certain visual details when synchronized to the
moving image. In computer game music, imperative functions are also often prominent,
where musical cues are used to initiate physical action by the user. Similarly, in music used
for advertising, the music is operating to convince the listener to seek out certain products
or services. The engagement of the viewer can be addressed by (induced or experienced)
emotional expressions. Different degrees of social distance can be established by varying
the size of instrumentation (i.e. solo instruments vs. large orchestra), or by using different
recording techniques (close microphone placement vs. long distance). By establishing
expressional contrast between music and visuals (or other involved modes), or by
intertextually referring to culturally well-known musical material, music can also be used to
rhetorically comment on narrative events in a multimodally told story. The scene from The
Secret of My Success, described in article 1, gives example of this function.

The textual metafunction represents the communicator’s text-forming potential. It is the

component which provides the texture, the organizing of a text (in a broader sense) as a
coherent message through textual resources of a mode. It expresses the relation of the
communicational act to its environment, including both the modal environment (including
what has been communicated before) – and the situational environment. Textual features of
music are constituted by material aspects and social use of the musical sound, such as the
employment and design of rhythm, melody, harmony and instrumentation for meaning-

32
making functions. Looking at music as a mode of representation, such attributes can be
seen as intramodal textual features available for making meaning. The textual component
has an enabling function with respect to the other two metafunctions. ‘It is only in
combination with textual meanings that ideational and interpersonal meanings are
actualized’ (Halliday, 1978, p. 113). In article 2, this is evident in how the participants
modify intramodal textual features (musical parameters, such as tempo, rhythm, register or
articulation) of the basic music examples – and in doing so also affect ideational and
interpersonal meanings of the given scenes. Different degrees of awareness of this are
apparent from the interviews made, as described in article 3.

It is also apparent how the different meanings, discussed by the participants in article 3,
relate to the multimodal interaction of music and image (which can partly be seen as
intermodal textual functions). As mentioned, Kress et. al (2001) point out how
metafunctional meanings can be ascribed to individual modes, as well as to the joint
expression of a multimodal ensemble, reflecting the interplay of several simultaneous
modes. The available modes in a certain statement can then individually express meaning
that can be seen as equivalent, complementary or contradictory in relation to each other
(ibid., p. 14). Different modes, however, tend to take on different tasks broadly along the
lines of their respective affordances and functional specializations (inherent material
attributes shaped by social processes). In combining music and moving image, both being
predominantly temporal modes of expression, important intermodal textual features will be
relative placement, timing and synchronization.

Other textual features are those of continuity, form and intertextuality. In music, what has
been played or heard before, will be decisive in the process of meaning-making. This
involves the immediate situation as well as the larger socio-cultural context. In the
immediate situation, music’s ability to contribute continuity and form will depend on
factors such as temporal cohesion and contrast, repetition and variety. A sense of cohesion
is established based on a combination of what was just played, and on our expectations of
what is to come next. In a larger cultural context, meaning is also established by our ability
to recognize specific compositions, songs or recordings, as well as certain styles and genres.
The use of the Jaws’ leitmotif in the movie The Secret of My Success (described in article 1)
is an example of the role of intertextuality for making meaning.

33
In the different articles of this thesis, metafunctional expressions of meaning are explored
in different ways. In article 1, the interplay of music and image in three different film
scenes is examined from the perspectives of ideational, interpersonal and textual meanings.
Article 2 has, as mentioned, its focus on the textual features of music – where musical
parameters are manipulated in order to express musical narrative functions. The third article
uses the concept of metafunctional meanings as a starting point for examining how verbal
statements reflect different kinds of knowledge. In multimodal social semiotics,
communication, meaning-making, learning and knowledge are seen as closely linked
processes. In the next section, a multimodal and social semiotic view of learning will be
further discussed.

Multimodality and Learning

As discussed above, according to social semiotic theory, signs are viewed as constantly
newly made (Jewitt & Kress, 2003). Signs are thus not seen as arbitrary ready-made objects
available for ‘use’, but as involved in motivated transformative processes of meaning-
making. These transformative processes constantly reshape the culturally available
resources of representation. Individuals use the resources available to them in their specific
socio-cultural environment and act to create signs. In using them, they change the resources.
All participants in the communicational act, producers or readers, are to be seen as sign-
makers. In the process of making signs, meaning is always in some sense transformed
according to the interests of the involved individuals. Signs can be made outwardly or
inwardly, always involving a transformative process which is also viewed as a determining
condition for learning:

A sign outwardly made changes the cultural resource for meaning making. The potential
for the cultural resource has been changed by the sign-maker’s action (even if in
infinitesimally slight ways). A sign inwardly made changes the sign-maker’s inwardly held
resources, both in this sign and it its interaction with all other signs in their inner resource.
The sign maker’s potential for meaning is changed. That change to a person’s inner
resource, both through representation to the outer world and through the representation to
their inner world, through interpretation, can be thought of as learning (Jewitt & Kress,
2003, p. 13).

Looked at in this way, learning is not about the acquisition and accumulation of given
semiotic meanings, but is seen as a dynamic process of sign-making, as a series of
processes of transformation and internalisation of signs. For assessment and research, only

34
outwardly made signs are directly available for study. Outwardly made signs are however
always to some degree reflecting (as well as affecting) signs made inwardly. Outwardly
made signs can therefore to some extent also be seen as ‘evidence of learning’ (Jewitt, 2006,
p. 28). The software tool REMUPP (described in article 2) was designed for making
possible the outwardly representation of knowledge and creative choices relating to musical
narrative functions through musical sound. In article 3, the interviews are in a different way
outwardly (through the different modal configurations of speech) expressing knowledge
and attitudes which also address the notion of musical narrativity.

According to multimodal theory, we learn from all available modes, not only from what is
written or said. As argued earlier, each mode offers its own distinct affordances and
constraints for representation. Also, meaning is usually established through the multimodal
interaction between modes. This gives rise to questions about the constitution of knowledge,
how it is formed through the different semiotic potentials of different modes. Does
knowledge remain the same when it is represented through different modes and modal
combinations? Selander & Rostvall (2008) describe how school textbooks have changed
over the years. Up until the 1940’s the dominating trend was to use written narrative texts
with pictures representing facts. In the 1960’s this relationship was turned around, the
pictures became more narratively oriented while the written text was more factual. Starting
in the 1990’s a different kind of layout became common, where written text and pictures
were more integrated, with the text commenting the pictures. Questions can be asked about
how these different ways of presenting a subject will affect the learning process – and how
does it affect the assessment of knowledge? When tested for knowledge, when will a
student want to use written text, image or both, to demonstrate the knowledge gained from
the books?

Jewitt (2006) points out how students are often involved in ‘translating’ across modes – a
process referred to as transduction. Transduction is seen as a transformative process across
communicational modes. What has been configured in one mode is in the transductive
process reconfigured according to the affordances of another mode. Kress and van Leeuwen
(2001) point out that the transductive process is not simply a matter of ‘translating’ between
modes, but is in itself transformative (p. 51). A shift in mode involves deep reshaping of
what is represented. This will lead to knowledge being expressed differently, depending on
what mode of representation is being used. What might have been learned through a variety

35
of modal expressions – not only from textbooks, but also from the teacher speaking,
gesturing, presenting models, organizing the classroom layout, showing videos etc – is in
assessment of the students often restricted to writing. Jewitt (2006) warns about the risk of
missing much of what the students do and the meaning they make, by only looking at what
they write and say. This is also true of learning taking place through new technologies:

In the case of new technologies the need to look beyond language is highlighted by the
silent activity that accompanies much technology-mediated learning. To continue to
connect learning with speech and writing at the exclusion of all other modes leads to a
restrictive concept of learning (Jewitt, 2006, p. 31).

In music education, the use and importance of other modes than language would seem self-
evident. As shown by Rostvall and West (2001), however (as described in chapter 4), a
focus on the coherent musical sound or expressive musical qualities is not always to be
taken for granted even during formal instrumental lessons.

Learning as Motivated Design

Sign-making has here been discussed as active and motivated processes where people (the
sign-makers) bring together form (signifiers) and meanings (signified) based on their
socially situated interests. This way of looking at it ‘suggests that learning is a process of
multimodal design’ (Jewitt, 2006, p. 30). Jewitt argues that viewing signs as motivated and
always transformative leads to looking at the texts students make as a trace of their
designed interests. In this perspective, the different musical results of the participants,
described in article 2, can be seen as designed texts reflecting their different interests as
well as knowledge of musical narrative functions.

Jewitt (2006) describes not only learning, but also pedagogy in terms of design: ‘pedagogy
is the designed textual realisation of interpersonal meaning – the social relations of the
classroom and the ideational shaping of knowledge – curriculum content’ (p. 140).
Selander and Rostvall (2008) identify three significations of the design concept, when put
in relation to learning: (a) media and modal expressions that have been designed for
learning, (b) the way that a learner forms conditions for her or his own learning, (c) a way
to analyze learning processes in relation to different conditions. Looking at the three
articles of this thesis from a design perspective suggests possible expansions of the above
notions of the design concept. Learning opportunities available in informal situations

36
involving watching films or playing computer games are not typically based on media and
modal expressions that have been explicitly designed with the intention of learning specific
topics, such as musical narrative functions. Educational or instructional media covering
various topics of course do exist, and are being used, but the overwhelming majority of
narrative media that we encounter on a daily basis do not have such a purpose. Film and
computer games, as we experience them in everyday life, are usually rather designed with
an aim to entertain, to tell a story and to make profit.

Nevertheless, film and games are however usually produced as a result of complex and
careful design – and the making of meaning when watching or playing them is also the
result of complex and involved design strategies and choices of the user. Gee (2007) points
out that in order to be able to play a computer game it must be learnt:

You cannot play a game if you cannot learn it. If no one plays a game, it does not sell, and the
company that makes it goes broke. Of course, designers could make the games shorter and
simpler. That’s often what schools do with their curriculums. But gamers won’t accept short
or easy games. So game designers keep making long and challenging games and still manage
to get them learned. How? [...] If a game, for whatever reason, has good principles for
learning built into its design – that is, if it facilitates learning in good ways – then it gets
played and can sell well, if it is otherwise a good game. (Gee, 2007, pp. 3-4).

Viewed this way, computer games can then actually be seen as being designed for learning.
The user must be able to learn the rules of the game, how to interact with it, how to make
sense of the plot etc. Part of this learning takes place through interaction with other gamers,
by participating in game-related communities, reading the game manual etc. But most
important is the design of the game itself, how it communicates through multimodal
configurations of image, animations, written text, dialogue, sound effects, music and
various forms of game controllers. To make sense of this kind of multimodal text, to be
able to make use of the opportunities for learning offered, the user must be able to ‘read’ it.
And to be able to read it, knowledge is required. Gee (ibid.) describes how learning
principles for acquiring this kind of knowledge need to be built into the games themselves.
We learn how to read the games (as well as the involved modes) by using them. Games
with poor learning designs won’t get played (or learned) and will not sell. Gee proposes
that producing, as well as playing, computer games this way represent processes leading to
increasingly better designs for good learning. This in turn makes possible the design of
more complex and challenging games. The learning principles thus described can be related

37
to Jorgensen’s (1997) concept of eduction, described in chapter 4. The games provide
designed learning environments, opportunities for learning, which draw or bring out the
user’s potential to learn.

Not only games, but all sorts of narrative multimedia can be seen from a similar
perspective, as offering designed opportunities for learning – enabling the user to make
sense from various forms of dynamic and multimodally achieved representations. Gee
(2007) speak of semiotic domains, any set of practices that recruits on or more mode of
representation to communicate distinctive types of meanings. Video gaming, anime, rap
music – as well as for example bird watching, physics, midwifery, wine connoisseurship –
can all be thought of as semiotic domains (that can in turn include distinctive sub-domains).
To be ‘literate’ in a specific domain includes also knowledge about the social practices of
which that literacy is but a part. Through social and cultural processes, media makers and
users continually explore semiotic resources of available media. They establish and
develop conventions for how to design and make meaning out of different forms of
expression, making possible complex and nuanced communicational practices. Jorgensen
(1997) describe the musical learning aspects involved in such processes by the concepts of
socialization and enculturation.

Applying the idea of design to education, the New London Group (2000) proposes a
pedagogical approach they refer to as a pedagogy of Multiliteracies. The New London
Group is an international group of scholars, including among others Bill Cope, Norman
Fairclough, James Paul Gee, Mary Kalantzis and Gunther Kress. Their basic concern is that
of literacy education, but their outspoken aim is to broaden the conventional ‘understanding
of literacy and literacy teaching and learning to include negotiating a multiplicity of
discourses’ (New London Group, 2000, p. 9). In doing that, they primarily address two
principal aspects of the rapidly changing cultural, institutional and global order. The first
aspect engages with the multiplicity of communications channels and media: ‘meaning is
made in ways that are increasingly multimodal [...] to find our way around this emerging
world of meaning requires a new, mulitmodal literacy’ (Cope & Kalantzis, 2000b, pp. 5-6).
The second aspect engages with the increasing salience of cultural and linguistic diversity
in contemporary societies. The fundamental questions addressed concern what these
changes mean for pedagogy, how content and methods of pedagogy can be conceptualized.
‘The key concept we developed to do this is that of Design, in which we are both inheritors

38
of patterns and conventions of meaning while at the same time active designers of meaning.
And, as designers of meaning, we are designers of social futures’ (ibid, p. 7). The
pedagogical approach proposed by the New London Group (2000) is based on four
different components: Situated Practice, Overt Instruction, Critical Framing and
Transformed Practice. These components will be discussed more in detail in the next
chapter.

Looked at from a design perspective, article 1 of this thesis can be described as exploring
the multimodal and meaning-making design of music together with image in film. Articles
2 and 3 can be seen as exploring the motivated design choices of the participants (both as
producers and readers), and also as discussing the designs used to explore their musical and
verbal expressions.

39
40
7. DISCUSSION

This thesis discusses music as a mode of communication, affording a range of semiotic

resources that we engage with to make meaning from. More specifically it focuses on
music with explicit narrative intensions and functions, achieving meaning in multimodal
interplay with other modes of representation – based on the interest and motivated design of
the producers as well as the viewers and listeners situated in specific social and cultural
contexts. This kind of music is referred to as narrative media music, or sometimes simply
narrative music or media music. The three articles of the thesis explore different
perspectives of narrative media music. While discussions of their various results and
methods can be found in each of the articles, the purpose of this chapter is to attempt a
deeper discussion of some of the topics, to put the articles in a clearer relation to each other
and to some of the central concepts used in this thesis. Also, inevitably, some new
questions will be raised.

Methodological Considerations
The three articles give examples of some quite different methodological approaches. In
article 1, detailed analyses are made of the musical narrative functions in several film
scenes. In the context of the entire thesis, the article aims to provide a referential,
conceptual and theoretical framework of narrative musical functions. This framework is to
serve as a foundation for the study presented in articles 2 and 3, where different aspects of
knowledge of narrative music are explored. In article 2, statistical methods are used to
examine quantitative data representing musical structure and performance. Quantitatively
based studies are relatively unusual in Scandinavian music education research. In this case
it was however considered to be a useful and interesting way to explore otherwise fleeting
and elusive aspects of musical sound and expression. Also, the game-like environment of
REMUPP seemed to be a suitable environment from an ecological point of view. Article 3
presents the second part of the study, analyzing stimulated recall interviews with the
participants verbally reflecting on their musical designs.

The different methodological approaches can be seen as each providing their own and
complementing ‘functional specializations’. One aim was to see how these specializations

41
each would contribute different kinds of results – and how these results could be combined
to complement or contrast each other.

Evidence of Learning
The notion of the participants of the study expressing knowledge as ’evidence of learning’
(Jewitt, 2006) has been referred to recurrently in this thesis. This is bound to the view of
learning as a series of processes of transformation. In interpreting the surrounding world,
the individual is guided by her or his socially situated interests. New signs are produced
‘inwardly’. Kress et al. (2001) describes the expression of knowledge (the re-presentation
of that what has been learned) as follows:

In re-presentation, a new, transformed inner configuration is the basis on which the new,
outwardly made sign (in a combination of writing, gesture, image and speech) is made in the
light of the individual’s interest at that moment, which includes an assessment of the external
environment in which this message/sign as communication is shaped. Consequently, sign-
making is always transformative, always the making of a new sign, always changing both the
shape of the resources and the disposition of the individual human subject. (Kress et al., 2001,
p. 28).

In chapter 6 it was discussed how only outwardly made signs are directly available for
study. Swanwick (1994) expresses a similar view, emphasizing that music teachers can
only observe and assess the work or results of the students rather than their actual process
of learning. He concludes that ‘perhaps the most significant learning experiences are always
tacit, unspeakable, incommunicable and therefore out of range of assessment’ (Swanwick,
1994, p. 103). In discussing ‘evidence of learning’, the word ‘evidence’ is therefore to be
understood as connoting ‘indication’, ‘sign’ or ‘token’ – rather than the more categorical
‘proof’ or ‘confirmation’.

On Causality and Sources of Knowledge

The study described in articles 2 and 3 explores expressions of knowledge, manifested by
the participants through the modes of musical sound and speech, as evidence or indication
of learning. The design and nature of the study do not however make available explicit
accounts of when and how this learning has taken place. The strong indications of group
consensus regarding musical narrative conventions, reported in article 2, are seen as signs
of learning – as knowledge of available semiotic resources and conventions of narrative
media music. It is assumed that learning has taken place, at least partly, through the

42
activities of watching movies and playing computer games. The clear, coherent and
contextualized designs of such narrative media are here regarded as providing good
opportunities for learning. This was discussed in chapter 6 (especially referring to Gee,
2007) and will be discussed more in detail later in this chapter. Also, the large amount of
time often spent by young people using various forms of narrative media on a daily basis
(as was also evident from the results of the questionnaires used in the study), positions such
media as likely for being influential sites of learning.

There can however also be other, additional, sources for this kind of knowledge. Music is
used for narrative purposes in many different kinds of situations, for example in children’s
games or play-songs. Activities such as creating mp3 playlists or using music for dance,
workout or various other musical situations, can be seen as including and expressing
narrative functions. In school however, there generally doesn’t appear to be much emphasis
on the notion of musical narrativity, at least not in Scandinavian music education – except
for in the lower grades by the use of play-songs and musical games.

The results presented in article 2 also indicate more specific significant relationships
between expressions of knowledge of musical narrative functions and the media habits and
amount of musical training of the participants. Participants playing comparatively much
computer games and watching more movies were generally, through musical sound,
expressing a higher degree of musical narrative conventions (as defined by the results of the
group). Participants learning to play an instrument, or doing comparatively much
recreational music listening, generally preferred more complex and expressive musical
structures. Similarly, as discussed in article 3, verbal statements of the Analytical type
dominated with participants who had a background of formal musical training. Those who
had no or little musical training expressed predominantly the Intuitive type of statements.

However, as discussed in article 3, these results do not allow for any unambiguous claims
of causality to be made. As the expressions of knowledge were only observed on one single
occasion, no observations of changes in knowledge are available. No claims of the causes
of the expressed knowledge can be made – other than as assumptions. As for the correlation
between high experience of narrative media and expressed knowledge of narrative
conventions, one interpretation could be that more time of engagement with narrative media
has resulted in more knowledge about musical narrative functions. Another interpretation

43
could be that people with a better understanding (and acceptance) of narrative functions and
conventions are more interested in spending time watching movies and playing games.
Also, the correlation between having received musical training and preferring a higher level
of musical complexity could be understood as people having developed a taste for musical
complexity from taking music lessons. Similarly, it could be assumed that participating in
music lessons results in an ability to verbally express musical knowledge in an analytical
way. But again, it could also indicate that people with a taste for higher musical complexity,
or with abilities to verbally and analytically express musical knowledge, are more interested
in taking music lessons.

It can also be speculated that the different causal possibilities are simultaneously available,
as a kind of self-reinforcing cycle. For example, people with a good understanding of
musical narrative conventions might be attracted to spend much time engaging with
narrative media, which provides opportunities for more learning about such matters which
in turn reinforces the interest, establishing new conditions for learning and so on. Such a
self-reinforcing process could be seen as an example of the mutual interaction of intuitive
and analytical knowledge described by Swanwick (1994).

As mentioned in article 3, there is an inherent contradiction and paradox in formally

attempting to study processes of casual and unconscious learning in informal situations.
This poses some interesting methodological challenges. As discussed elsewhere in this
thesis, narrative media music is typically experienced on an unconscious and unreflecting
level. If learning about musical narrative functions is occurring while playing computer
games or watching movies, it may be difficult to study such learning processes without
drawing attention to the music or otherwise interfering with the process. A formal study
might easily change the basic conditions for the participants’ engagement with media music.
To attempt a study of causal relationships of learning in such situations would require a
study carefully designed to address such challenges.

That said, it should be emphasized that the purpose of the study presented in articles 2 and
3 is to explore aspects of how musical knowledge is expressed, rather than to establish
causal relationships of the learning process. The focus is thereby put on features of the
expressed knowledge, on attitudes and awareness of the participants – as made available
through musical sound and speech – and also on the functions of narrative media music.

44
The indications of correlations between expressions of knowledge and media experience
and musical training are here seen as occasion for raising questions and discussing
relationships of informal and formal learning situations and opportunities.

Aspects of Transformation and Transduction

Transformative processes across different modes are discussed in chapter 6 as transduction.
In the study (articles 2 and 3), relating the music to the visual scenes involves complex
processes of transduction. The entire process of the participants can here be seen as being
constituted of several different transductive steps, where a shift was made from visuals, to
music, to audio-visuals, to spoken language. In the first part of the trial session, the starting
point was to view, interpret and relate to the visually represented scenes. Then, in response
to this, the object was to make a ‘fitting’ musical expression, relating closely to the visuals.
Later, during the interview, the task was once again to respond the scenes – but this time
relating verbally to the multimodal ensemble of visuals and music, with a certain emphasis
towards the musical expression.

The process of using the REMUPP software was not a strictly sequential process, but
involved dynamic interactions of different processes. When using the interface the
participant’s ‘reading’ of the scene would be affected by the continual changes made by the
musical parameters. The meaning of the scene would be defined by the multimodal
interweaved interaction of the visuals and the music. When altering the musical parameters,
the meaning of the combined multimodal ensemble would be transformed and the
conditions for making the next transformative decision would thus constantly be changed
and affected by the manipulations as they were made. Transformation took place not only
within each separate mode, but can be seen as involving the total multimodal ensemble.

Similarly, the research process of this study can be seen as transductive and transformative,
involving a number and variety of methodological, interpretative and communicative steps
– with the empirical data being represented as speech, writing and music (including its
relation to the 3D-animated movies). The music was, besides being represented as sound,
also described mathematically and statistically.

45
Musical Narrative Functions
A central concept of this thesis is the notion of musical narrative functions. These functions
have been proposed, discussed and listed in the first article of the licentiate thesis
(Wingstedt, 2004, 2005) and also in article 1 of this book. In short, they are categorized as
the emotional, informative, descriptive, guiding, temporal and rhetorical functions. In
previous publications these narrative functions have been referred to as classes or
categories. However, according to classical Aristotelian view, categories are to be defined
as discrete and mutually exclusive entities. The narrative functions are rather to be seen as
dynamic properties or functions of narrative media music, simultaneously active on
different levels. To avoid confusion, they will therefore henceforth be referred to simply as
functions, rather than as categories or classes.

In addition to these narrative functions there are of course also many other sorts of
functions that can be associated with music (including music used for narrative purposes),
ways of employing music which are related to a variety of socially and culturally available
musical practices. To attempt a description of the vast array of musical practices available
only in Western contemporary society is not possible within the scope of this thesis.
Traditionally, functions of musical aesthetics have been an important topic of discussion,
especially in Western classical tradition. A few examples of such discussions are Adorno
(2002), Dahlhaus (1982) and Scruton (1999). In recent years, there has also been a marked
interest in exploring musical functions related to our defining, maintaining and expression
of identity. For example DeNora (2000), Folkestad (2002), and Hargreaves, Miell and
MacDonald (2002) have discussed this topic. Related to discussions of identity is that of
music as a vehicle for collective experiences. Adorno and Eisler (1947/1994) suggest that
motion-picture music, among other things, may serve a social function that cements and
holds together the spectators.

Functions related to aesthetics, identity and collective experiences are indeed to be

considered as highly operational and significant functions of narrative media music (as well
as of any type of music). Looked upon as dimensions of communicational processes, such
functions operate principally on an interpersonal level in the interplay between the listener
and the music – and also within communities constituting several listeners. In discussing
narrative semiotic resources of media music, these functions have however not been
emphasized as they are here not considered as being of primarily narrative nature (even if

46
they in a sense can be seen as contributing to self-narratives or collective narratives in
various situations). They are here rather seen as parallel functions that in turn may influence,
and dynamically interact with, the functions that are here viewed as more specifically
narrative.

Narrative Functions and Communicational Metafunctions

In article 1 of this book, the musical narrative functions are put in relation to Halliday’s
(1978) metafunctions of communication.2 This places the musical narrative functions in a
wider communicational perspective, making it possible to look at them in terms similar to
for example language or image. By doing this, it becomes clearer how narrative music can
be seen as a full communicational mode, achieving meaning on ideational, interpersonal
and textual levels. Relating the musical narrative functions to Halliday’s metafunctions will
result in a setup looking as in Figure 1.

Narrative Functions of Media Music

IDEATIONAL MEANING
Emotive function (observed): As information and description...
Informative function (other than emotions): Providing facts, explaining,
establishing cultural setting, time period, status...
Descriptive function: describing physical attributes, environment, movement...

INTERPERSONAL MEANING
Emotive function (experienced, induced)...
Guiding function: indicative, imperative...
Rhetorical function: commenting, contrasting...

TEXTUAL MEANING
Temporal function: providing continuity, defining form...
Intermodal function: placement, timing, synchronicity...

FIGURE 1: Narrative functions of media music related to Halliday’s metafunctions of communication.

Compared to the original musical narrative functions (Wingstedt, 2005), the textual
intermodal function has been added in Figure 1. This is to better clarify how the musical

2
See chapter 6 for a discussion of Halliday’s metafunctions.

47
meaning-making functions are qualified in multimodal interplay with other modes of
representation. This is discussed more in detail in article 1.

Interactivity and Musical Functions

In the three articles, film and computer games are frequently given as examples of media
using narrative media music. Music in ‘linear’ media such as film and in interactive media
such as games is similar when it comes to the narrative functions and principles discussed.
There is however an obvious difference between them in how the audience is involved. In
games, rather than being described as a viewer or listener, the common concept is rather
that of a user. The difference is that of agency. According to social semiotic view, sign-
making, including watching movies as well as playing computer games, is seen as always
involving fundamentally agentive processes. Looking at learning as a transformative
process is based on a view of the agency of the learner. However, in games the physical
involvement becomes greater and more direct. Kress and van Leeuwen (2001) discuss how
‘modern computer interfaces technologies try to make perception and reading more
physically interactive’ (p. 68), hence potentially more transformative – allowing the user to
not only read but also to rewrite and adapt.

The REMUPP software (described in article 2), in enabling the adaption of musical
expression by manipulation of musical parameters, is an example of physical
transformation of meaning. The functional specializations of interactive interfaces offer
their own potentials and opportunities for how musical narrative functions can be employed
and achieved. For example, interpersonal guiding functions (such as imperative functions
prompting the actions of the user), or confirmative functions (interpersonal and ideationally
informative functions confirming or rejecting the user’s actions) tend to be salient in game
music. Aspects of consequences of interactivity for musical implementation in new media
are briefly discussed in the licentiate thesis (Wingstedt, 2005) and also for example by
Buttram (2004). General discussions of interactivity in games and other new media can be
found in Raessens and Goldstein (2005) and Salen and Zimmerman (2004).

Conditions and Opportunities for Learning

Many of the concepts that are central to this thesis relate to conditions and opportunities for
learning that are available in informal as well as formal learning situations. This includes
the concepts of intuitive and analytical knowledge, metafunctions of meaning (or

48
communication), transformation, interest, motivated design and multimodality. Some of
these relationships will be discussed in the following sections.

Metafunctional Configurations versus Intuitive and Analytical Knowledge

As mentioned earlier, Halliday (1978) states that the textual metafunction has an enabling
function with respect to the other two metafunctions of communication. The understanding
of the ideational content of a message and the establishing of an interpersonal relation
towards it relies on a textual structure that is perceived as coherent. This is illustrated by the
process of working with the REMUPP software, as described in article 2. By modifying
textual aspects of the music, i.e. altering the values of the given musical parameters, the
participants were able to express and shape ideational and interpersonal aspects of the
musical meaning. This way of looking at it puts the actual message (or musical sound) in
focus.

If instead putting the learner in focus, looking at expressions of awareness and knowledge
as evidence or indication of learning (as discussed in article 3), the configuration of
metafunctional interdependences can be described differently. Starting out from a view of
learning as socially and culturally situated, makes it possible to instead see the interpersonal
understanding as a point of departure – in a sense enabling the ideational and textual
dimensions. Expressing knowledge of musical content and structure (associational and
analytical aspects) implies that the participant has already established an interpersonal
relation, and a fundamental and intuitive awareness, towards the subject matter. The
associative descriptions of the different scenes, or the analytical explanations of the musical
structure at play, rest on the intuitive and interpersonal involvement with the situation and
matter at hand – in this case on the participant’s evaluation of the narrative relevance and
aesthetical functions of the music, and of their own perceived agency towards the musical
result and the situation. It was apparent from the interviews that all the participants, even
those with relatively little to say, based their statements on this kind of interpersonally
based engagement with the subject matter (their music and its narrative relevance). This is
in congruence with Swanwick’s (1994) view of intuitive knowledge as making ‘possible all
other ways of knowing’ (p. 28) – and also of his view of intuitive and analytical forms of
knowledge as being interwoven and interdependent dimensions that together lead to
understanding.

49
The concepts of intuitive and analytical forms of knowledge are not commonly included in
social semiotic theory. In the analyses of the interviews (article 3), building on Halliday’s
(1978) metafunctions of communication, Swanwick’s (1994) concepts however emerge as
useful for describing aspects of knowledge as expressed by the participants. Verbal
statements of the Unclear and Intuitive types foreground interpersonal aspects of
communication, expressing opinions and evaluations of the music heard. These kinds of
statements interpersonally emphasize the speaker’s position in relation to the music (at the
same time as it also in a sense ideationally describes interpersonal features of the music).
These expressions reflect a type of knowledge that fit well with Swanwick’s notion of
intuitive knowledge, to which he associates concepts such as appearance, impressions and
aesthetics.

Statements of the Analytical and Transformative types foreground the content aspects of
communication, as ideationally constituted meaning. As noted in article 3, the ideational
content of these statements predominantly focus on information and descriptions of textual
features and uses of the music. These statements reflect what Swanwick (ibid.) describes as
analytical knowledge, which he for example relates to aspects of underlying form,
relationships and conceptions. Statements of the Associative type, describing narrative
situations or events associated with the music, fit ‘the creative forming of images’ (ibid., p.
29) that is by Swanwick described as processes associated with intuitive knowledge. These
statements again make ideational aspects of the music and the narrative situation salient.
The ideational content can here be thought of as ‘imagined’, interpreted or created
involving the interest of the participants as well as their knowledge of musical narrative
functions.

It should be noted that using Halliday’s metafunctions of communication as an analytical

tool in the way described here, and in article 3, is somewhat different from more traditional
approaches to text analysis. This includes also common methods of Systemic Functional
Grammar, building on Halliday’s theories of language (e.g. Halliday & Matthiessen, 2004;
Thompson, 2004). Rather than looking at the spoken statements as self-contained units of
text, or examining the communication taking place between the interlocutors (here, the
participant and the interviewer), it is here instead the participant’s relation to the narrative
context (the musical expressions combined with the visuals) that is spotlighted. That is, the

50
metafunctional meanings are made as much relevant to the (verbally expressed
apprehension of the) use of music as to the use of language.

In article 3 it was pointed out how statements of the Associative, Analytical and
Transformative types usually were combined with comments of intuitive nature. This can
be seen as illustrating Swanwicks (1994) description of intuitive and analytical knowledge
as being mutually intertwined and interacting (discussed in chapter 4). Swanwick’s model
of learning as a spiral oscillating between the two poles of intuitive and analytical
knowledge is also compatible with the social semiotic view of learning as a continually
transformative process. Both these views emphasize how coming to knowledge is a
dynamic process, involving continuous change regarding what is learned as well as for the
learner. Swanwick describes his view as ‘that the growth of knowledge at any level
emerges intuitively and is nourished and channelled by analysis’ (p. 86).

Aspects of ‘Interest’
In chapter 6 it was discussed how viewing signs as motivated and always transformative
leads to looking at the texts students make as a trace of their designed interests. People’s
interests are seen as socially and culturally situated and are related to factors such as past
experiences, present situation and role, expectations etc, of the individual.

As mentioned earlier, the different musical results of the participants described in article 2,
can be seen as texts reflecting their interests as well as their knowledge and skills. Also, the
results of the interviews, described in article 3, can be looked at in a similar way. For
example, two of the participants (Jenny and Viktor), had produced rather similar musical
results for the Lake movie. When choosing their respective favourite versions however,
Viktor preferred the one closest to the narrative convention while Jenny chose a version
very different from the convention. This illustrates how not only their abilities, but their
different interests motivated their design choices. This is in support of a view where signs
are not seen as arbitrary (which would imply learning and reproduction of pre-set codes),
but where learning is seen as involving the active and motivated design of meanings.

Learning and Design

Selander and Rostvall (2008) argue that when designing educational texts and other tools
intended for communication and learning, it is important to provide material that is relevant

51
and functional. It should be well-arranged with a clear structure. A good understanding of
the users’ interests, disposition and opportunities to use the material is necessary. Also,
interactivity, playfulness and aesthetics are important aspects to consider. As it happens,
these are not only requirements for how to design learning tools, they are also features that
characterize a good film or computer game. In article 3 it is discussed how much of the
expressed knowledge of the narrative musical aspects in the study seems to be related to the
everyday use and experience of the media involved – film, video, television, computer
games, the Internet etc. In such multimodal settings opportunities are available for learning
not only thru multimodal texts but also about the modes themselves.

As discussed earlier, when watching a movie or playing a game, conscious attention is

usually not on the music itself. This does not however necessarily mean that the activity of
watching films and playing games are generally unfocused. The situation may be casual –
and focus is not on the music or on learning – but the situation often involves a strong focus
on other things, such as on the plot, emotions or actions in a film or on specific tasks to be
accomplished in a game. These kinds of activities tend to involve a high degree of
engagement, motivation, attention and concentration from the audience or user. And even if
music in such situations does not attain a high degree of conscious salience, it is clear that it
often contributes meaning that is essential for the understanding of the overall narrative – as
is discussed in article 1. To understand a film or game, it is necessary for the audience to
also understand the expressed musical narrative functions, even if this is maybe not
consciously reflected on in the process of experiencing a multimodally told story.

The music in such environments is typically composed and performed with utmost care
taken to detail and musical expression. Its relation to image, dialogue, sound effects and the
overall narrative is usually highly functional and coherent. At the same time meaning is
often achieved through dynamic and elaborate expressional means, requiring a certain level
of musical responsiveness from the listener. In short, music in narrative media is often
performed as, and contributing to, a complex, meaningful and coherent musical and
narrative ‘whole’. Selander and Rostvall (2008) stress how such a situation should be
considered and seen as beneficial for learning. This appears to be somewhat different from
how Rostvall and West (2001, 2006; Rostvall, 2008, West, 2007) found that formal
instrumental music teaching is often carried out. As described earlier, their video recordings
of instrumental lessons showed how music was often not addressed, by the teacher, as

52
coherent phrases, rhythms or melodies but that focus was put on isolated notes, fingerings
etc. Also, expressive qualities of the music were not addressed. Interestingly then, it seems
that a film or game situation in some respects can be seen as offering more coherent and
structured designs and opportunities for learning music than in the kind of music lessons
found by Rostvall and West.

Learning Principles of Narrative Multimedia

Gee (2007), in discussing video games, emphasizes that in the modern world language is
not the only important communicational system. He proposes that when people learn to
play video games they are learning a new literacy. That is – by learning to make meaning
from the multimodal ensemble that constitutes a game, they are learning to ‘read games’.
This implies viewing games as texts. A text, in multimodal sense, is any expressional
instantiation of a mode – or combination of modes – that we read and make meaning from.
It is thereby also an instantiation of available meaning potentials in a given social, cultural
and historical context. As mentioned in chapter 6, Gee (ibid.) uses the concept semiotic
domains – any set of practices that recruits one or more modes of representation to
communicate distinctive types of meanings. He points out that ‘literacy in any domain is
actually not worth much if one knows nothing about the social practices of which that
literacy is but a part’ (p. 18). Viewing musical narrative functions from a semiotic and
communicational perspective makes it possible to also, in a similar vein, speak of a certain
kind of musical literacy – referring to the ability to learn and read musical meanings in a
narrative and multimodal setting. This kind of musical literacy is thus different in meaning
from when we traditionally talk about ‘reading music’, which usually rather refers to the
ability to read standard music notation.

In chapter 6, Gee’s (2007) view of how learning opportunities are built into computer
games is discussed. Gee argues that learning principles by necessity need to be part of
successful game design, to make it possible for users to acquire knowledge about how to
read, understand and play the games. A basic reason for this is that if a game is too difficult
to learn, it does not sell. Gee discusses the conditions and nature of this kind of game-
related learning. Thirty-six general learning principles are proposed that he sees at work
when we play computer games. These have to do with how we learn to solve problems,
take cues, form an identity, connect different sign systems and so on. Gee focuses explicitly
on the semiotic domain of computer games, but much of his arguments can also be applied

53
to film or other narrative multimodal expressions. Since music can be considered to
contribute essential functions to the meaning-making design of narrative media, many of
these principles may also be discussed in relation to features more specifically related to
narrative media music. Using a few of Gee’s learning principles as a point of departure –
and borrowing some of his headings – reflections will here be made that concerns learning
opportunities of narrative music by use of games, film and other narrative media in
informal situations.

Design Principle. By watching movies and playing games etc, we learn that music ‘works’
and that it is available as a mode of representation in the design of multimodally told
narratives. The use of non-diegetic music3, as well as diegetic, is recognized as being part
of the convention of how to make films or games. In article 3, it is apparent how it came
naturally for the participants to treat music as a narrative and representational mode. We
also learn to appreciate and form opinions about such design. In the study, it was evident
how most of the participants could express such opinions, generally of intuitive character
but at times also of a more analytical nature.

Semiotic Principle. We learn not only that music works in narrative media but also how it
works. We learn to attribute musical meaning to certain types of musical expressions. We
learn how to mean, musically. This includes knowledge about specific culturally available
clichés, conventions and meaning potentials as well as more general musical narrative
functions. Such knowledge was evident in the expressions of musical as well as verbal
statements of the participants of the study, for example in how they musically related to
convention, and how they verbally associated to (by Intuitive or Associative statements) or
reflected on (by Analytical or Transformative types of statements) the narrative aspects of
the music.

Multimodal Principle. We learn that meaning and knowledge are built up through various
modes of expression – one of those being music. We learn how meaning is achieved
through the dynamic interaction of music in interplay with other modes, as is described in
article 1. This kind of knowledge was made apparent in the study, where the relation of the

3
Non-diegetic music is not ‘heard’ by the characters of the narrative, only by the audience. Diegetic music
(also known as ‘source music’) is music that is part of the spatio-temporal world of the story (the diegesis),
e.g. music coming from a radio, an on-screen band, a jukebox – or any other music that is apparently heard by
the characters of the story. This is discussed more in article 3.

54
musical expression to the visuals was carefully considered by the participants in their
musical as well as verbal statements.

Semiotic Domains Principle. We learn not only to ‘read’ narrative media or music, but also
to participate in practices related to various domains. We gain the potential to join related
social groups, ‘affinity groups’ as Gee calls it. These groups can be connected to larger
domains of which music is part, such as ‘strategy games’ or ‘anime’ – but also to domains
that more specifically relate to music, maybe focusing on specific narrative music genres,
certain media composers or such.

Metalevel Thinking about Semiotic Domains Principle. Learning involves active and
critical thinking about the relationships of the semiotic domain being learned to other
semiotic domains. This can for instance be done by reflecting on how music is used
(similarly or differently) for narrative purposes in different media: e.g. video games, film,
opera, ballet, game shows on television etc. Or comparisons of narrative music can be made
to music that is not specifically used for narrative purposes. Another example is to reflect
on how music originally created for other purposes appears and is used in films or games.
In the study, metalevel reflection was approached by several of the participants – for
example in explaining that ‘normally I don’t like this kind of music, but here (used in this
kind of narrative situation) it is good’. The Transformative statements described in article 3,
repositioning the music examples to new functions or situations, are also examples of
metalevel thinking about semiotic domains.

Situated meaning principle. The meanings of music as a semiotic resource in narrative

multimedia are experienced and situated in ‘situation-specific’ contexts. Meanings are not
general or decontextualized. Whatever generality meanings come to have is discovered
bottom up via embodied experiences in specific situations. Wider and more general
meaning potentials of musical expressions become specific in situational and narrative
contexts, as discussed in article 1.

Intertextual Principle. The learner understands narrative media, as well as media music, ‘as
a family (“genre”) of related texts and understands any one such text in relation to others in
the family, but only after having achieved embodied understandings of some texts’ (Gee,
2007, p. 110). Film and games abound with such families or genres, including various
practices of employing narrative music. Understanding a group of texts as a family (genre)

55
of texts is a large part of what helps the learner make sense of such texts. Narrative media
also often include specific references to other works as part of the narratives. Using the
Jaws’ theme in The Secret of My Success (article 1) is one such example.

Identity Principle. Music plays an important part in how games and movies offer the
audience or users opportunities to explore identity issues by identifying with characters of
the plot, trying out virtual identities in games and so on. Music’s powerful emotive
functions, abilities to express dimensions of social status, style, ethnicity, subcultures etc,
contribute to this. In engaging with identity issues offered by narrative media, the user will
get opportunities to experience and learn musical matters from various perspectives. An
example of this is how certain musical genres, otherwise experienced strange or difficult,
may become (at least temporally) accepted when experienced from the point of view of
‘someone else’.

Practice principle. When playing games and watching movies, learners get lots of
experience and practice in a context where the practice is not boring. In games, we spend
lots of time on a task, often repeating certain actions. Game music is also often repetitious,
facilitating an increasing familiarity with the musical structure and narrative functions. The
repeating music is an example of a design principle characteristic for games, which Viktor
expressed knowledge of in one of his comments in article 3. Repetition is also an important
factor with advertising music, typically recurring during frequent commercial breaks on
television. Also, by being in private possession of movies in various video formats, it is
today not uncommon to watch the same movie quite a few times – thus getting opportunity
to turn attention to the musical score in more analytical ways than is usually done the first
time.

For knowledge to deepen and evolve, to not remain predominantly intuitive, the learner
needs to be enabled, motivated and challenged to critically reflect and to redesign the
knowledge to work in other contexts or cultural sites. To use the concepts of Swanwick
(1994), the spiral of intuitive and analytical knowledge needs to be set in motion. Gee
(2007) argues that this to a large extent is achieved by engaging with narrative media, as
described above. The results of the study, described in articles 2 and 3, however raise
questions about the role and relationship of formal education to the expanded opportunities

56
for learning offered by new media. Some issues relating to this will be discussed in the
following sections.

Educational Implications
As described in chapter 4, Folkestad (2006) points out the importance of defining how the
concepts of formal and informal learning are being used in specific writings and contexts.
He identifies four common uses of these concepts, each focusing on different aspects of
learning: (1) The situation; (2) Learning style; (3) Ownership; and (4) Intentionality. The
learning opportunities offered by the activities of watching movies and playing computer
games have so far in this thesis mostly been described simply as being part of ‘informal
situations’ or settings. The above aspects, in this case especially those of situation and
intentionality, can be used to further define the character of these informal situations.

The place or physical context of casual media use is normally located outside of school, at
one’s own or at friends’ homes, in cinemas, at gaming venues – and increasingly also in
various other public spaces through the use of portable media players, laptop computers or
mobile phones. Such settings, in discussing relations to school and formal education, are
typically to be considered as informal venues for learning. It should however be kept in
mind that the film and game industries are at the same time to be viewed as well-
established institutions in their own right. In that sense, the learning here discussed can also
be thought of as taking place within other institutions than school.

The activities of watching film or playing computer games are normally to be considered as
self-chosen and the ‘learning style’ is to be seen as aurally based. The intention, however, is
usually not to learn – at least not to learn about narrative functions of media music. It was
earlier in this chapter discussed how the situation of engaging with narrative media can be
considered as focused, even if the focus of attention is not typically on the music. The
music is rather experienced through what Green (2002) refers to as distracted listening or
hearing, or by Ericsson’s (2002) concept of ‘preoccupied assimilation’. Swanwick (1994)
uses the term osmosis to describe how learning takes place in such situations. Also with
respect to learning style, ownership and intentionality, this kind of learning situation is
primarily to be considered as informal.

57
It has been argued throughout this thesis how the expanded opportunities for engaging with
media and media music involve expanded opportunities for musical learning in informal
situations. The notion of musical learning as we are used to think about it in formal
education is thereby also changed and broadened – and in the process the concept may
come to appear slightly blurred and difficult to grasp. As noted in article 3, Folkestad (2006;
using the terms of Ziehe, 1986) remarks that the learning taking place informally outside of
school is now experienced as the common while learning in school appears as the
uncommon learning practices. Viewed this way, it can be alleged that the majority of
musical learning takes place outside schools. As children start school, ‘the ways in which
learning is organized and approached there become the alternative ways of gaining
knowledge compared to what they have already experienced’ (Folkestad, 2006, p. 144).

If this is the case, several questions emerge. One question, which was also asked in article 3,
is about what implications the evolving situation has for formal music education. Another
question concerns what kind of knowledge it is that we get through learning in informal
settings and how it might differ from knowledge we get from music education in school.
Looking at how the design of new communication media offers opportunities of learning
can serve as a starting point for reflecting on the relationship of learning in informal and
formal situations. Can something be gained from combining learning practices and
opportunities of the different situations? If so, how can it be done? One recent example of
addressing such questions is Green (2008), mentioned in chapter 4, who proposes methods
for bringing informal learning practices of popular musicians into the classroom.

Folkestad (2006) recommends to not look at formal – informal as a dichotomy, but rather as
the two poles of a continuum – as ‘in most learning situations, both these aspects of
learning are in various degrees present and interacting in the learning process’ (p. 141). As
mentioned earlier, he also proposes that Jorgensen’s (1997) concept of eduction can be seen
as a meeting place for formal and informal learning. A learning environment, organised,
designed and led by the teacher, where the potential, capacities, abilities and aptitudes that
the student already possess are enabled, brought out and developed. That is, an environment
where opportunity is given for the transformative work of the student to be encouraged and
challenged.

58
Swanwick (1994) speaks of education as ‘a meeting place for intuition and analysis’ (p.
117). It has earlier (and in article 3) been discussed how he emphasizes that intuitive and
analytical kinds of knowledge are not to be seen as two distinct types of knowledge, but as
interdependent and intertwined. They are interacting in a continual and dynamic process,
described as an ascending learning spiral oscillating between the poles of intuitive and
analytical knowledge. As we gain new analytical knowledge, our intuitive knowledge is
simultaneously changed and re-defined and the spiral is ready for another turn. This can be
understood as a transformative process of coming to knowledge. Swanwick (ibid.) argues
that it is not possible to proceed up one side or the other of the spiral – to rely only on either
intuition or analysis. Some instances of formal music education are however at risk for
attempting this at times.

Pedagogy as Design for Social Futures

The New London Group (2000) in proposing their ‘pedagogy of Multiliteracies’, discussed
in chapter 6, recognize the importance of recruiting ‘the different subjectivities, interests,
intentions, commitments and purposes that students bring to learning’ (p. 18). They
describe how the ‘languages’ needed to make meaning in modern society are radically
changing in three realms of our existence: our working lives, our public lives, and our
personal lives. As they see curriculum as a ‘design for social futures’ (p. 19), the New
London Group argue for the notion of pedagogy as Design. Teachers and managers are seen
as Designers of learning processes and environments, where different curricular,
pedagogical and classroom designs motivate and achieve different sorts of learning.

The processes of learning involve transformative re-presentation and recontextualisation of

meaning. Reading, seeing and listening, as well as writing, speaking and other modal
expressions, are all viewed as productive activities, forms of Designing. ‘Through these
processes of Design, moreover, meaning-makers remake themselves’ (ibid. p. 23). As
mentioned earlier, the proposed pedagogical approach is based on four different
components: Situated Practice, Overt Instruction, Critical Framing and Transformed
Practice. These four components do not constitute a linear hierarchy, nor do they represent
stages. They are to be understood as being related in complex ways, where elements of each
may occur simultaneously, or be salient at different times. All of them are in the
educational process repeatedly revisited at different levels. As these components have a

59
certain relevance to the results of the study described in article 3, they will be further
discussed below.

Situated Practice is, as a pedagogical approach, constituted by immersion in meaningful

practices based on the world of the learners. It starts out from the view that ‘human
knowledge is primarily situated in sociocultural settings and heavily contextualized in
specific knowledge domains and practices’ (New London Group, 2000, p. 31). As an
integrated part of a formalized school curriculum, there will be a need to recruit learners’
previous and current experiences, including extra-school communities, practices and
discourses, as an integral part of the learning experience. The issues of musical learning in
informal settings can be seen as examples of such practices and discourses. However, as the
New London Group sees it, certain limitations can be discerned by using the concept of
Situated Practice as the sole basis for pedagogy. One concern is that Situated Practice does
not necessarily lead to conscious control and awareness of what one knows and does. Also,
it does not necessarily nurture learners or communities who can critique what they are
learning. There is also the question of putting knowledge into action. ‘People may be able
to articulate their knowledge in words [...] yet they might still be incapable of reflexively
enacting their knowledge in practice’ (ibid., p. 32).

The learning conditions and outcomes of Situated Practice, as described here, may be
compared to the ‘intuitive-associative’ type of verbal statements discussed in article 3.
Characteristic for these statements, were that they expressed knowledge intuitively, not
involving reflection, conscious awareness or critique of the music commented on. This is
similar to the described learning outcomes of Situated Practice. These statements however
expressed clear assessments, evaluative opinions and narrative associations – which can be
seen as necessary requirements for subsequent development of reflected awareness, critical
perspective and redesigned transformative practice (be it expressed verbally or musically).
The participants with a predominantly intuitive profile were typically not strongly
associated with previous music education. It can be assumed that most of their musical
learning had taken place as situated practice in informal settings outside of school. This is
in line with the concern expressed by the New London Group that Situated Practice alone
does not necessarily lead to conscious control and awareness of what one knows and does.

60
Overt Instruction is therefore suggested as a necessary pedagogical component, primarily
employing scaffolding learning techniques rather than ‘direct transmission’ and rote
memorization. Based on the ideas of Vygotsky (1987), Overt Instruction is intended to
facilitate conscious awareness and control over what is being learned – ‘over the intra-
systematic relations of the domain being practiced’ (New London Group, 2000, p. 33). One
important aspect here is the development and use of metalanguages, to describe the form,
content and function of the discourses of practice.

In the study, statements of the ‘analytic-associative’ type seem to fit well with the above
described learning outcomes. A characteristic feature of this statement type was how the
intuitively associative comments were combined with analytical descriptions of musical
(intra-modal) features contributing to the musical expression and narrative functions.

However, according to the New London Group (ibid.) neither immersion in Situated
Practice nor Overt Instruction necessarily gives rise to critical or cultural understanding.
Through methods of Critical Framing, learners can gain the necessary personal and
theoretical distance towards what they have learned; constructively critique it; account for
its cultural location; creatively extend and apply it – ‘interpreting the social context and
purpose of particular Designs of meaning’ (Kalantzis & Cope, 2000, p. 247). This involves
learners standing back from what they are studying and viewing it critically in relation to its
context – to make strange again what they have learned and mastered. This is also the basis
for Transformed Practice.

Transformed Practice is described as a ‘transfer in meaning-making practice, which puts

the transformed meaning to work in other contexts or cultural sites’ (New London Group,
2000, p. 35). This includes innovation, in old communities and in new ones. The process of
learning and transformation leads to re-practice, where theory becomes reflective practice –
which in turn takes us back to where we started, to Situated Practice, to where we always
need to return. Through Transformed practice students can implement understanding
acquired through Overt Instruction and Critical Framing in practices that help them
simultaneously to apply and revise what they have learned. This process is analogous to the
process of coming to knowledge described by Swanwick (1994), as the turning of the
intuitive-analytical spiral. He states that as we gain new analytical knowledge ‘we come to
see things differently as intuitive understanding is re-defined’ (p. 43).

61
The verbal statements of the Transformative type are in article 3 described as repositioning
the music to new situations or functions. Alternative uses, placements and possibilities for
the music are suggested. Similarly it can be said that the musical examples described as
‘non-conforming’, deviating from the convention, are expressions including innovation and
re-practice – at least in cases where this results from the participants making ‘informed
choices’. These verbal and musical examples can be seen as expressing a kind of
knowledge that is akin to how the outcomes of Transformed Practice are described. As
discussed earlier, the study does not provide results showing how and where this kind of
knowledge is achieved. The results however illustrate different aspects of musical
knowledge, emphasizing how they interrelate, interact and interdepend. The educational
model proposed by the New London Group (2000) suggests a framework for
acknowledging and bringing together the different aspects of knowledge.

Conclusion
As the title implies, and as was discussed in the introduction, making music mean is that
which is of central concern for this thesis. Throughout the book, two different connotations
of the word ‘mean’ have been of special interest – that of ‘representation’ and that of
‘importance’.

The use of music as a representational mode impacts powerfully on our experience and
understanding of narrative situations and processes of film, computer games and other
narrative multimedia. It is apparent that music actively and efficiently provides far more
than just ‘mood’ or mere ‘niceness’ to a multimodally told story. The many functions of
narrative media music can however be elusive. It is not surprising how a deeper and more
reflective knowledge of these functions can be hard to achieve. As discussed before, much
of the time this music goes unnoticed on a conscious level. When I lecture for students in
film scoring classes at college, we spend much time analyzing the musical functions of
various film scenes. During such sessions it frequently happens that the students ask to see
a certain sequence once (or several times) more, since they ‘forgot to listen to the music
because the scene was so dramatically engaging’. On repeated listening we can in such
cases usually establish that the music paradoxically appears to be one of the main reasons
the scene is so engaging.

62
It has almost become a cliché to claim that a sign of good media music is that it is
‘invisible’ or ‘unheard’. It is not my view that this is always the case. As the above example
illustrates, it does however seem that narrative media music often have a tendency to
become transparent as the audience gets narratively involved. As much of our experience of
narrative media music thus is of intuitive nature, maybe also much of our knowledge of the
functions of this kind of music tend to remain intuitive in character. But even so, as the
results of the study indicate, it was apparent how the participants took the musical narrative
functions for granted, as something obvious and self-evident. They naturally approached
music as a representational mode, with the potential to express ideational, interpersonal and
textual meanings.

As music contributes meaning as a representational mode, it also becomes important as an

element in the design of multimodal narratives. And as narrative media become
increasingly available in modern society, narrative media music gradually becomes more
socially and culturally important, taking space in our daily lives. As argued earlier,
narrative media music is often, especially for young people, the largest source of daily
musical experience. This music communicates on many different levels, often bypassing
the listener’s resources for conscious and critical reflection. Narrative music will, on a daily
basis, affect how we feel, think and act. Not only in film and computer games – musical
narrative functions are also in operation for example at work places, in public spaces, in
advertising, in news programs or documentaries on television and so on. Music will
contribute pleasure and joy, aesthetical experiences, energy and purpose. But music is also
active and operative in the continual negotiating of cultural values, truth and credibility,
status and power that is part of the contemporary media flow. As a consequence of this, an
important and desirable prospect for music education and research should be to contribute
to a fuller and deeper understanding, awareness and critical reflection concerning music’s
communicational and narrative roles and functions. If what we (think we) see is to a large
degree determined by what we hear, it is clear that more knowledge is needed about what it
is we hear.

Given the communicational resources and learning opportunities of new communication

media, it has to be asked what is communicated and learned – and how it is done. The
multimodal and interactive facilities of new media invite certain kinds of transformative
work by the user. It can be asked if we learn different things about music when we listen to

63
a CD, go to a concert, watch a movie or play a computer game. Does knowledge remain the
same if we learn it from a book or from a game? What is the role of the modal and
situational contexts involved in the communicational process and how are interrelations of
power and knowledge enacted through available modal configurations? As such questions
become increasingly important in society, they are also of increasing relevance for music
education and research. As stated in the introduction, making music mean is an ongoing
process. It is hoped that this thesis can contribute at least a small turn of the intuitive-
analytical spiral of musical knowledge.

64
REFERENCES

Adorno, T.W. (2002). Essays on Music. Berkeley: University of California Press.

Adorno, T. & Eisler, H. (1994). Composing for the Films. London: The Athlone Press.
(Orig. work published 1947).

Bratt-Rawden, K. & DeNora, T. (2005). ‘Music and informal learning in everyday life’, in
Music Education Research, Vol. 7, No. 3, pp. 289-304.

Brown, R. S. (1994). Overtones and Undertones: Reading Film Music. Berkeley:

University of California Press.

Buckingham, D. & Willett, R. (Eds.). (2006). Digital Generations: Children, Young People,
and New Media. Mahwah, NJ: Lawrence Erlbaum Associates.

Buttram, T. (2004). ‘Beyond Games: Bringing DirectMusic into the Living Room’, in T. M.
Fay (Ed.) DirectX 9 Audio Exposed: Interactive Audio Development (pp. 501-515).
Plano, TX: Wordware Publishing Inc.

Burn, A. & Parker, D. (2003). ‘Tiger’s Big Plan: Multimodality and the Moving Image’, in
C. Jewitt, & G. Kress (Eds.). Multimodal Literacy (pp. 56-72). New York: Peter Lang
Inc.

Chion, M. (1994). Audio-Vision: Sound on Screen. New York: Columbia University Press.

Cohen, A. J. (2001). ‘Music as a Source of Emotion in Film’, in P.N. Juslin & J.A. Sloboda
(Eds.) Music and Emotion: Theory and Research (pp. 249-272). Oxford: Oxford
University Press.

Cook, N. (1998). Analysing Musical Multimedia. Oxford: Oxford University Press.

Cope, B. & Kalantzis, M. (2000a). ‘Designs for Social Futures’, in B. Cope & M. Kalantzis
(Eds.) Multiliteracies: Literacy Learning and the Design of Social Futures (pp. 203-234).
London: Routledge.

Cope, B. & Kalantzis, M. (2000b). ‘Introduction: Multiliteracies: The Beginnings of an

Idea’, in B. Cope & M. Kalantzis (Eds.) Multiliteracies: Literacy Learning and the
Design of Social Futures (pp. 3-8). London: Routledge.

Croce, B. (1992). The Aesthetic as the Science of Expression and of the Linguistic in
General (C. Lyas, Trans.). Cambridge: Cambridge University Press. (Original work
published 1902).

Dahlhaus, C. (1982). Esthetics of Music. Cambridge: Cambridge University Press.

Darby, W. & Du Bois, J. (1999). American Film Music: Major Composers, Techniques,
Trends, 1915-1990. Jefferson, NC: McFarland Classics.

Davison, A. (2004). Hollywood Theory, Non-Hollywood Practice: Cinema Soundtracks in

the 1980s and 1990s. Aldershot: Ashgate.

65
DeNora, T. (2000). Music in Everyday Life. Cambridge, UK: Cambridge University Press.

Donnelly, K.J. (2001). Film Music: Critical Approaches. Edinburgh: Edinburgh University
Press.

Donnelly, K.J. (Ed.). (2005). The Spectre of Sound: Music in Film and Television. London:
BFI Publishing.

Ericsson, C. (2002) Från guidad visning till shopping och förströdd tillägnelse:
Moderniserade villkor för ungdomars musikaliska lärande [From guided exhibition to
shopping and preoccupied assimilation: Modernised conditions for adolescents’ musical
learning] Malmö, Sweden: Malmö Academy of Music: Studies in Music and Music
Education no 4. (Doctoral diss.).

Folkestad, G. (1996). Computer Based Creative Music Making: Young People’s Music in
the Digital Age. Göteborg: Acta Universitatis Gothoburgensis. (Doctoral diss.).

Folkestad, G. (2002). ‘National Identity and Music’, in R. Macdonald, D. Hargreaves & D.

Miell (Eds.) Musical Identities (pp. 151-162). Oxford: Oxford University Press.

Folkestad, G. (2006). ‘Formal and informal learning situations or practices vs formal and
informal ways of learning’, in British Journal of Music Education, 23:2, pp. 135-145.

Gee, J. P. (2007). What Video Games Have to Teach Us about Learning and Literacy
(Revised and updated ed.). New York: Palgrave Macmillan.

Gorbman, C. (1987). Unheard Melodies: Narrative Film Music. Bloomington, Indiana:

Indiana University Press.

Green, L. (2002). How Popular Musicians Learn: A Way Ahead for Music Education.
Aldershot: Ashgate.

Green, L. (2008). Music, Informal Learning and the School: A New Classroom Pedagogy.
Aldershot: Ashgate.

Gullberg, A.-K. (2002). Skolvägen eller garagevägen: Studier av musikalisk socialisation

[The School Way or the Garage Way: Studies on musical socialization] (PhD diss.).
Piteå: Musikhögskolan i Piteå, Luleå University of Technology.

Halliday, M.A.K. (1978). Language as Social Semiotic. London: Edward Arnold.

Halliday, M.A.K. (1985). An Introduction to Functional Grammar. London: Arnold.

Halliday, M.A.K. & Matthiessen, C. (2004). An Introduction to Functional Grammar (3rd

Edition). London: Hodder Arnold.

Hargreaves, D.J., Miell, D. & MacDonald, R. (2002). ‘What are Musical Identities, and
why are they Important?’, in R. Macdonald, D. Hargreaves & D. Miell (Eds.) Musical
Identities (pp. 1-20). Oxford: Oxford University Press

Hickman, R. (2006). Reel Music: Exploring 100 Years of Film Music. New York: Norton

66
Hodge, R. and Kress, G. (1988). Social Semiotics. Cambridge, UK: Polity Press.

Jewitt, C. (2003). ‘Computer-Mediated Learning: The Multimodal Construction of

Mathematical Entities on Screen’, in C. Jewitt, & G. Kress (Eds.) Multimodal Literacy
(pp. 34-55). New York: Peter Lang Inc.

Jewitt, C. (2006). Technology, Literacy and Learning: A Multimodal Approach. London:

Routledge.

Jewitt, C. & Kress, G. (Eds.). (2003). Multimodal Literacy. New York: Peter Lang Inc.

Johansson, K.-G. (2002). Can You Hear What They’re Playing?: A Study of Strategies
among Ear Players in Rock Music. Piteå: Musikhögskolan i Piteå, Luleå University of
Technology. (Doctoral diss.)

Jorgensen, E.R. (1997). In Search of Music Education. Chicago: University of Illinois Press.

Juslin, P.N. & Sloboda, J.A. (2001). (Eds.). Music and Emotion: Theory and Research.
Oxford: Oxford University Press.

Kalantzis, M. & Cope, B. (2000). ‘A Multiliteracies Pedagogy: A Pedagogical Supplement’,

in B. Cope & M. Kalantzis (Eds.) Multiliteracies: Literacy Learning and the Design of
Social Futures (pp. 239-248). London: Routledge.

Kalinak, K. (1992) Settling the Score: Music and the Classical Hollywood Film. Madison,
WI: The University of Wisconsin Press.

Karlin, F. & Wright, R. (2004). On the Track: A Guide to Contemporary Film Scoring (Rev.
2nd edition). New York: Routledge.

Karlsen, S. (2007). The Music Festival as an Arena for Learning: Festspel i Pite Älvdal and
Matters of Identity. Piteå: Luleå University of Technology, Department of Music and
Media. (Doctoral diss.).

Kenner, C. (2003). ‘Embodied Knowledges: Young Children’s Engagement with the Act of
Writing’, in C. Jewitt & G. Kress (Eds.) Multimodal Literacy (pp. 88-106). New York:
Peter Lang Inc.

Kress, G (1993). ‘Against Arbitrariness: The Social Production of the Sign as a

Foundational Issue in Critical Discourse Analysis’. Discourse and Society, vol. 4 (2):
169-191.

Kress, G. (2003). Literacy in the New Media Age. London: Routledge.

Kress, G., Jewitt, C., Bourne, J., Franks, A., Hardcastle, J., Jones, K. & Reid, E. (2005).
English in Urban Classrooms: A Multimodal Perspective on Teaching and Learning.
Oxon: RoutledgeFalmer.

Kress, G., Jewitt, C., Ogborn, J. & Tsatsarelis, C. (2001). Multimodal Teaching and
Learning: The Rhetorics of the Science Classroom. London: Continuum.

67
Kress, G. & Van Leeuwen, T. (2001). Multimodal Discourse: The Modes and Media of
Contemporary Communication. London: Arnold.

Kress, G. & Van Leeuwen, T. (2006). Reading Images: the Grammar of Visual Design, (2nd
edition). London: Routledge.

Kulturrådet (2008). Statistik om kultur och medievanor. [Swedish Arts Council: Statistics
on Culture and Media Habits]. Retrieved July 26, 2008, from http://www.kulturradet.se.

Lindstrand, F. (2006). Att göra skillnad: Representation, identitet och lärande i ungdomars
arbete och berättande med film [Making Difference: Representation, Identity and
Learning in Teenagers’ Work and Communication with Film]. Stockholm: HLS Förlag.
(Doctoral diss).

Lipscomb, S.D. & Kendall, R.A. (1994). ‘Perceptual Judgment of the Relationship Between
Musical and Visual Components in Film’. Psychomusicology, 13, 60-98.

Lipscomb, S.D. & Tolchinsky, D.E. (2005). ‘The Role of Music Communication in
Cinema’, in D. Miell, R. MacDonald & D.J. Hargreaves (Eds.) Musical Communication
(pp. 383-404). Oxford: Oxford University Press.

Macdonald, R., Hargreaves, D. & Miell, D. (Eds.). (2002). Musical Identities. Oxford:
Oxford University Press.

Marshall, S. & Cohen, A. J. (1988). ‘Effects of Musical Soundtracks on Attitudes to

Geometric Figures’. Music Perception, 6, 95-112.

Martin, P. J. (1995). Sounds and Society: Themes in the Sociology of Music. Manchester:
Manchester University Press.

Mera, M. & Burnand, D. (Eds.). (2006). European Film Music. Aldershot: Ashgate.

New London Group (2000). ‘A Pedagogy of Multiliteracies’, in B. Cope & M. Kalantzis

(Eds.) Multiliteracies: Literacy Learning and the Design of Social Futures (pp. 9-37).
London: Routledge.

New Oxford Thesaurus of English (2000). Oxford: Oxford University Press.

Nilsson, B. (2002). ‘Jag kan göra hundra låtar’: Barns musikskapande med digitala
verktyg [‘I can Make a Hundred Songs’: Children’s Creative Music Making with Digital
Tools]. Studies in Music and Music Education No. 5. Malmö: Malmö Academy of
Music. (Doctoral diss.).

North, A. C. & Hargreaves, D. J. (1997). ‘Music and Consumer Behaviour’, in D.J.

Hargreaves & A. C. North (Eds.) The Social Psychology of Music (pp. 268-289). Oxford:
Oxford University Press.

North, A. C. & Hargreaves, D. J. (2005). ‘Musical Communication in Commercial

Contexts’, in D. Miell, R. MacDonald & D.J. Hargreaves (Eds.) Musical Communication
(pp. 405-422). Oxford: Oxford University Press.

68
Prendergast, R.M. (1992). Film Music: A Neglected Art (Rev. 2nd edition). New York:
Norton.

Prensky, M. (2006). ‘Don’t Bother Me Mom – I’m Learning!’. St. Paul, MN: Paragon
House.

Raessens, J. & Goldstein, J. (Eds.). (2005). Handbook of Computer Game Studies.

Cambridge: MIT Press.

Rostvall, A.-L. (2008). ‘Möten och metaforer’ [Meetings and Metaphores], in A.-L.
Rostvall & S. Selander (Eds.) Design för Lärande [Design for Learning] (pp.143-156).
Stockholm: Norstedts Akademiska Förlag.

Rostvall, A.-L. & Selander, S. (Eds.). (2008). Design för Lärande [Design for Learning].
Stockholm: Norstedts Akademiska Förlag.

Rostvall, A.-L. and West, T. (2001). Interaktion och kunskapsutveckling: En studie av

frivillig musikundervisning [Interaction and Learning: A Study of Music Instrument
Teaching]. Stockholm: KMH Förlaget. (Doctoral diss.).

Rostvall, A.-L., & West, T. (2005). ‘Theoretical and methodological perspectives on

designing video studies of interaction’. International Journal of Qualitative Methods,
4(4), Article 6. Retrieved May 2, 2007, from
http://www.ualberta.ca/~iiqm/backissues/4_4/pdf/rostvall.pdf.

Rostvall, A.-L., & West, T. (2006). ‘Multimodality and Designs for Learning’, in J.
Allwood, B. Dorriots & S. Nicholson (Eds.) Papers from the Second Nordic Conference
on Multimodal Communication (Gothenburg Papers in Theoretical Linguistics 92), pp.
273-282. Gothenburg, Sweden: Göteborg University, Department of Linguistics.

Salen, K. & Zimmerman, E. (2004). Rules of Play: Game Design Fundamentals.

Cambridge: MIT Press.

Saussure, F. de (1986). Course in General Linguistics (transl. by Roy Harris). Peru, IL:
Open Court.

Scruton, R. (1999). The Aesthetics of Music. Oxford: Oxford University Press.

Selander, S. & Rostvall A.-L. (2008). ‘Design och Meningsskapande – en inledning’

[Design and Meaning Making – An Introduction], in A.-L. Rostvall & S. Selander (Eds.)
Design för Lärande [Design for Learning] (pp. 13-27). Stockholm: Norstedts
Akademiska Förlag.

Shaffer, D. W. (2006). How Computer Games Help Children Learn. New York: Palgrave
Macmillan.

Small, C. (1998). Musicking: The Meanings of Performing and Listening. Middletown:

Wesleyan University Press.

Swanwick, K. (1994). Musical Knowledge: Intuition, Analysis and Music Education.

London: Routledge.

69
Söderman, J. (2007). Rap(p) i käften: Hiphopmusikers konstnärliga och pedagogiska
strategier [Verbally fa(s)t. Hip-hop musicians’ artistic and educational strategies].
Malmö: Malmö Academy of Music, Studies in Music and Music Education No 10.
(Doctoral diss.).

Tagg, P. (2000). Kojak – Fifty Seconds of Television Music: Towards the Analysis of Affect
in Popular Music. New York: The Mass Media Music Scholars’ Press (orig. 1979).

Tagg, P. & Clarida, B. (2003). Ten Little Title Tunes: Towards a Musicology of the Mass
Media. New York: The Mass Media Music Scholars’ Press.

Thompson, G. (2004). Introducing Functional Grammar (2nd edition). London: Hodder

Arnold.

Van Leeuwen, T. (1999). Speech, Sound and Music. London: Macmillan.

Van Leeuwen, T. (2005). Introducing Social Semiotics. London: Routledge.

Vygotsky, L. S. (1987). The Collected Works of L.S. Vygotsky: Vol. 1, Problems of General
Psychology, Including the Volume Thinking and Speech. New York: Plenum.

West, T. (2007). ‘Multi-layered Analysis of Teacher-student interactions: Concepts and

perspectives guiding video analysis with Tattoo, The Analytic Transcription Tool’.
Pedagogies: An International Journal, Volume 2, No. 3, pp. 139-150.

Wingstedt, J. (2004). ‘Narrative Functions of Film Music in a Relational Perspective’,

paper presented at ISME – Sound Worlds to Discover, 14-16 July, Santa Cruz, Tenerife,
Spain.

Wingstedt, J. (2005). Narrative Music: Towards an Understanding of Musical Narrative

Functions in Multimedia. Luleå: School of Music, Luleå University of Technology,
Sweden. (Licentiate thesis). (Available at http://pure.ltu.se/ws/fbspretrieve/199292)

Wingstedt, J., Brändström, S. & Berg, J. (2008a). Making Meaning of Media Music:
Expressions of Knowledge about Musical Narrative Functions. Manuscript submitted for
publication.

Wingstedt, J., Brändström, S. and Berg, J. (2008b). ‘Young Adolescents’ Usage of

Narrative Functions of Media Music by Manipulation of Musical Expression’.
Psychology of Music, Vol. 36 (No. 2): pp. 193-214.

Wingstedt, J., Brändström, S. & Berg, J. (in press). ‘Narrative Music, Visuals and Meaning
in Film’. Visual Communication.

Wingstedt, J., Liljedahl, M., Lindberg, S. & Berg, J. (2005). ‘REMUPP – An Interactive
Tool for Investigating Musical Properties and Relations’, in S. Fels & T. Blaine (Eds.),
The International Conference on New Interfaces for Musical Expression, NIME-05 (pp.
232-235). Vancouver: University of British Columbia.

Ziehe, T. (1986). Ny Ungdom: Om Ovanliga Läroprocesser [New Youth: On Uncommon

Learning Processes]. Stockholm: Norstedts.

70
 Article 1

Wingstedt, J., Brändström, S. & Berg, J. (in press):

‘Narrative Music, Visuals and Meaning in Film’. Visual Communication.

Manuscript accepted for publication.

Narrative Music, Visuals and Meaning in Film
Johnny Wingstedt
Royal College of Music, Stockholm, Sweden

Sture Brändström and Jan Berg

Luleå University of Technology, Dept. of Music and Media, Piteå, Sweden

Abstract
Narrative media music, music used for narrative purposes in multimedia
such as film, television or computer games, is becoming one of the largest
sources of musical experience in our daily lives. Though typically
experienced on an unconscious and unreflected level, this kind of music
actively contributes narrative meaning in multimodal interplay with image,
speech and sound effects. Often, what we (think we) see is to a large degree
determined by what we hear. Using Halliday’s (1978) metafunctions of
communication as a starting point, three short film scenes (from Jaws, The
Secret of My Success and The Birds) are examined, with a focus on the
intermodal relationships of music and image. The examples illustrate how
musical and visual expressions combine to form multimodal statements
where the whole is certainly different than the sum of the parts.

Keywords: Film music, Metafunctions, Multimodality, Musical semiotics, Narrative music.

The emergence of new digital media is having a profound effect on how we communicate –
and thereby on how we make meaning and perceive the world. Kress (2003) describes how
the screen is replacing the book as the dominant medium for communication. The centuries-
long dominance of writing is giving way to a new dominance of the image, which in recent
years has led to an increased interest in exploring principles of visual meaning-making,
literacy and learning through visual means (e.g. Kress & van Leeuwen, 2006; Lindstrand,
2006; van Leeuwen & Jewitt, 2001). The new conditions for communication via the screen is
however not restricted only to visual modes of communication. Today, the screen is usually
not silent. Besides image, written language, layout, video, animation etc, we are also making
sense – or trying to make sense – out of the intricate interplay with aural modes such as
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

spoken language, sound effects and music. Each mode individually bears meaning, but above
all meaning emerges from the complex interplay of the different modes involved. As Walter
Murch remarks, talking about film sound in his foreword to Michel Chions book Audio-Vision
(1994): ‘Despite all appearances, we do not see and hear a film, we hear/see it’ (p. xxi).

This article will take a closer look at some of the functions of narrative media music, music
used for narrative purposes in film, television and computer games etc – and discuss how
meaning is achieved in interplay with the visuals. This will be done by examining the
multimodal interplay of music and image in three short movie-scenes, using Halliday’s (1978)
metafunctions of communication as a point of departure. Narrative media music is becoming
one of the largest sources of musical experiences in our daily lives. Even if this kind of music
tends to be transparent and is often processed by the audience on an unreflecting level, it
seems to actively contribute to how we make meaning from a multimodally told story.
Gorbman (1987) describes it: ‘[the music] guides the spectator’s vision both literally and
figuratively’ (p. 11). In other words – what (we think) we see is to a large degree determined
by what we hear.

It of course also works the other way around. Just as the music will affect how we see things,
the visuals will also determine how we hear the music. Murch (in Chion, 1994) describes a
phenomenon he calls conceptual resonance between image and sound, where the sound
makes us see the image differently, and then this new image makes us hear the sound
differently, which in turn makes us see something else in the image and so on. As audience
however, our conscious attention is usually on the visuals. We tend to interpret the events on
film or television as something we see – even if we in fact actually ‘hear/see’ it. This is
reflected also in how we talk about media experiences: we go to see a movie and we watch
television. For the sake of analysis however, emphasis will in the following primarily be put
on music’s contribution to how meaning is established in the multimodal interplay of the
filmic narrative. A social semiotic perspective will be used, and especially Kress and van
Leeuwen’s (2006) book Reading Images: The Grammar of Visual Design has provided
inspiration, fundamental concepts and terminology.

2
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

A musical starting point has been Wingstedt (2004, 2005) who suggests a categorization of
musical narrative functions such as they appear in film and other multimedia. Six classes are
presented, which will here simply be referred to as functions:

• The Emotive function refers to music’s ability to communicate emotive qualities, either
experienced by the audience (induced) or just cognitively identified (represented; Juslin,
2001). The expressed emotions may be attributed to individual characters of the story or
represent relationships or events – they can also describe overall emotive aspects of
situations or forebode future implications of the plot.

• The Informative function comprises situations where music expresses or ‘explains’

phenomena or events by communicating information on a cognitive rather than on an
emotional level. Music can for example evoke a certain cultural settings or time periods,
clarify ambiguous situations, indicate social status or simply represent a character or
phenomenon, for instance by the use of a leitmotif (this will be further discussed later).

• The Descriptive function is related to the informative function in certain aspects, but differs
in that the music is actively (or programmatically) describing something rather than more
passively representing certain values. It is usually a matter of describing the physical world,
such as physical setting, appearance or movement.

• The Guiding function includes musical functions that, so to speak, turn directly to the
audience aiming to ‘direct’ the eye, thought and mind. This could include indicative or
imperative functions. The latter function is prominent in computer games or advertising,
where the purpose is to bring the audience to perform specific actions.

• The Temporal function foregrounds the time dimension of music. Especially important is
music’s ability to provide continuity (immediate, longer or overall) as well as how music
can contribute and define structure and form.

• The Rhetorical function refers to how music sometimes ‘steps forward’ to comment the
narrative events or situation. This is often achieved by having the musical expression
contrast the visuals or by referring to well-known musical material.

In a given situation, different musical narrative functions typically operate simultaneously on

several different levels; the salient functions will quickly and dynamically change.

3
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

Metafunctions
In order to make it possible to examine the interactions between music and visuals in film
more closely, and to put the musical narrative functions into a wider communicational
perspective, Halliday’s (1978) metafunctions of communication will be used as a starting
point. Simply put, the metafunctions are based on the notion that: (1) Every sign tells us
something about the world; (2) It positions us in relation to someone or something; (3) It
produces a structured text (Jewitt, 2006). Using Halliday’s terminology, these three basic
functions are known as the ideational, the interpersonal and the textual metafunctions
respectively. According to Kress, Jewitt, Ogborn and Tsatsarelis (2001) the metafunctions are
to be seen as ‘general requirements of any human communicational system’ (p. 4). To
elaborate a little further on what they imply:

The ideational metafunction is the content function of communication. Kress et al. (2001)
describe it as representing what goes on in the world; material, verbal, mental and relational
processes – ‘who does what, with or to whom and where’ (p 13). The interpersonal
metafunction is the participatory function of communication, communication as doing
something. It is the component through which the communicator expresses her own attitudes
and judgments – and seeks to influence the attitudes and behaviour of others. It establishes,
maintains and specifies relationships between members of societies or groups through
expression of social relations, interrelations of power and knowledge (ibid.). The textual
metafunction is the component which provides the texture, the organizing of a text (in a
broader sense) as a coherent message through textual resources of a mode (in relation to the
environment). The textual component has an enabling function with respect to the other two.
It is only in combination with textual meanings that ideational and interpersonal meanings are
actualized. (Halliday, 1978: 113). A modal expression (a text, or actualized meaning potential)
is an instantiation of all three functions interwoven.

The issue of musical meaning has been controversial and much discussed over the years,
especially when it comes to music’s ability to represent ideational meaning. Advocates of the
idea of ‘absolute’ or ‘autonomous’ music have argued that (especially instrumental) music is
not capable of expressing any specific meaning, except the meaning of the musical sound
itself (e.g. Hanslick 1955/1854). It can however be asked if music is ever by ‘itself’. The
context may be more or less clear, but from a social semiotic point of view it is impossible to
think about music without a social, cultural and situational setting including also the

4
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

multimodal interplay involved in any musical performance – be it live or recorded. This has
also been discussed in depth by Cook (1998).

Van Leeuwen (1999) has pointed out the difficulties in applying Halliday’s metafunctions to
sound as a general mode. He suggests that this is maybe because the resources of sound does
not seem as specialized as those of language and vision, maybe because sound is not yet to be
considered a communicational mode. Trying to look at sound as one distinct mode of
expression seems to be an overwhelming challenge, since the wide domain of sound can
encompass so many different forms of expressions. However, looking at the much narrower
domain of narrative media music, the communicative aspects become clearer. This has several
reasons. Firstly, the historical, social and cultural practices connected to the use of music in
narrative multimedia such as film are well established in the Western society. Also, the
typical musical underscore for film is composed with an explicit narrative purpose.

Furthermore, the situational, narrative and multimodal contexts in film are usually relatively
clear. According to social semiotic theory, an expression’s contextual relationship is
inseparably connected to how we make meaning. Hodge and Kress (1988) for example, state
that ‘the context, both the physical referents and the social conditions of semiosis, is decisive
for communication to occur’ (p. 39). If the context in which music occurs (and contributes) is
perceived as being vague and ambiguous, the meaning-making process will be similarly
vague and ambiguous. In film however, the musical expression occurs in a relatively well-
defined and distinct multimodal context. The interplay of the music with visuals, dialogue,
sound effects etc, provides conditions for the music to actively and concretely contribute to
the narrative. Also, the typical filmic narrative relies to a large degree on socially and
culturally established conventions, which contributes to making the musical narrative
functions clear and ‘readable’ to members of that culture (i.e. general conditions for meaning-
making are relatively clear, even if specific meanings may be realized differently for
individual members).

Wingstedt, Brändström and Berg (2008) have studied how, among other factors, previous
media habits are related to knowledge and understanding of musical narrative functions. The
participants of the study demonstrated strong consensus regarding knowledge of general
musical narrative functions in multimedia, which were clearly evident and readable to them.
They also shared a view of music as being an important and highly relevant narrative mode in

5
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

multimedia. At the same time they however expressed notable individual variations
concerning specific narrative meanings, treating meaning potentials and affordances of music
in ways related to their individual backgrounds and interests. This suggests that even when the
general narrative context seems relatively clear, potential meanings as realized for individual
listeners/viewers are still open to negotiation.

Jaws Example
A short scene from the movie Jaws (1975), directed by Steven Spielberg, will here be used to
illustrate how the concept of metafunctions can be applied as a point of departure when
analyzing a musical dramatic underscore. When analyzing a multimodal text, such as film, the
metafunctions can be applied to the different modes individually, as well as to the whole
multimodal ensemble. In this case it is primarily the latter approach that will be taken, in
order to explore the interactions between the music and the other modes involved. The
following scene takes place about 25 minutes into the movie, and lasts for ca 90 seconds:

It is a quiet summer night and we see two men standing on a small wooden pier by the sea.
Musically, in the underscore, a mildly dissonant chord is quietly building. The men are
fishing, and we understand they are going for a big fish since they are using a car tire for a
float and a thick metal chain as a fishing-line. Suddenly something takes the bait and we can
see the chain disappear into the water as the float is dragged away. At the same time a
musical motif is heard in the underscore. It is a melodic interval of a minor second, played by
low strings, repeating relentlessly. The float and the chain keep moving further out into the
water. The men get excited and shout: ‘He’s taking it, he’s taking it!’ The chain is attached to
the pier, and suddenly the wooden construction collapses from the strain put on it. One of the
men falls into the water and part of the pier is dragged outwards following the chain. After a
short while we can see the floating pier suddenly turning around. It is now coming towards
the man swimming in the water. As this happens, the musical expression changes, the
downbeats are heavily accented and the music gets louder. The man swims for his life. The
tempo of the music speeds up, and the image intercuts between shots of his struggle and the
pier getting closer to him. He manages to reach safe ground just before the creature dragging
the pier gets to him. The music slows down, the rhythmic activity evens out, it gets softer and
ends on a long note. The man is now safe.

6
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

The musical content for this entire scene is based on the (now) famous Jaws leitmotif,
composed by John Williams. In its most basic form it is a two-note motif of an alternating
minor second interval that is used to represent the shark. By the time this scene appears in the
movie, a connection has been well established between the motif and the creature, since the
shark has already appeared several times – each time accompanied by the same motif. The
motif has played only when the shark is directly referred to. Through consistent repetition and
association it has gradually acquired a certain meaning, and this way of using it is what makes
it a ‘true’ leitmotif: a recurring musical motif that is associated with a particular character,
object, relationship, place or idea etc. The concept of leitmotif was developed and refined
primarily by the German composer Richard Wagner (1813-1883), and in order for a musical
theme in film to be considered a leitmotif ‘one must observe a clear and consistent
relationship between a musical idea and its onscreen counterpart’ (Hickman, 2006: 43).

Jaws: Music’s Ideational Aspects

A basic characteristic of any leitmotif is that its meaning could be described as being
‘cumulative’. The first time the Jaws leitmotif is heard in the film its meaning is not yet
established. For each recurrence its meaning will however become clearer – the motif
gradually turns into a sign that explicitly represents the shark. The statement of the motif in
the scene described above thus communicates ideational information necessary for this scene:
‘Jaws is present’. Interestingly, the shark is not even visible in this scene, only what might be
interpreted as signs of its actions – which the leitmotif helps to make clear. The leitmotif
coming to represent Jaws (and subsequently ‘danger’, ‘evil’ etc) is an example of a musical
symbolic attributive process and can also be ascribed to the Informative function of narrative
music mentioned above. Kress and van Leeuwen (2006) list several characteristics of
symbolic attributes, one being that ‘they are made salient in the representation’, another being
that ‘they are pointed at’ (p. 105). A leitmotif is made salient by its repeated use and
association. The recurrent use of the same musical material makes it gradually easier to
recognize and thereby to notice. The leitmotif is ‘pointed at’ by its consistent placement
together with the phenomenon it is set to represent, in this case Jaws (at the same time as it is
also ‘pointing to’ the phenomenon). Through this, a ‘relation of identity’ is established. The
meaning-making processes of the leitmotif are in this way similar to how we learn to associate
a visual logo with a certain product, brand or institution. But just as a logo is not arbitrarily
chosen but is also carefully designed to express certain values regarded to be appropriate or

7
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

desirable associations, so is a leitmotif usually composed to as ably as possible describe or

characterize the person, object or other phenomenon it is representing.

The Jaws motif contributes such descriptive meanings through sound, as analytical processes,
by associating certain possessive attributes with the beast (the carrier). The low register of the
motif suggests large size and power in a way that a higher register would not. By experience
we know that large objects are required to produce sounds in the bass register, an example of
what van Leeuwen (2005) refers to as experiential metaphor. The low register can also be
associated with the physical position of the shark lurking beneath the surface of the water.
Still another association would build on a convention of film music that low notes express
danger, violence and menace. The repeating rhythmic movement of the music describes
another attribute of the shark – the relentless movement of its tail, which Williams himself has
said to be the initial idea behind this motif (Bouzereau, 2000). The instrumentation of bowed
strings suggests a gliding motion through the water, providing a dark and glum timbre. The
rhythmic movement also stresses the dissonant melodic interval of a minor second. A
dissonant interval is usually defined as being unstable, harsh and unresolved. Dissonance does
in itself suggest potential movement towards resolution, and in this case it also seems to
emphasize the unstable, primitive and ruthless nature of the beast.

At the point where the pier turns and start coming towards the swimmer, the simultaneous
changes in the music (accented downbeats, louder dynamics and increased tempo) lets us
know about Jaws’ determination and intention. We can hear how the shark is firmly
determined to catch its prey, which can be understood as a ‘mental attribute’ of Jaws. It can
also be seen as what Kress and van Leeuwen (2006) call a mental process, which is a ‘vector
formed by a “thought bubble” or a similar conventional device [that] connects two
participants, the Senser and the Phenomenon’ (p. 75, our italics). The vector in this case
would be replaced by the synchronization between the music and the image. The Senser is
represented by Jaws, in turn jointly represented visually by the pier being dragged and
musically by the leitmotif. The Phenomenon is represented musically by the accented
downbeats, dynamics and increased tempo.

Kress and van Leeuwen (ibid.) also describe how the ideational relation between an Actor and
a Goal in an image is established through a vector – and how in film the role of the vector is
taken over by movement. A complication is however that the relation between Actors and

8
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

Goals may be represented in several subsequent shots, each showing only one of the
participants, resulting in a disconnection typical for contemporary ‘film language’. In the
Jaws scene the sequence after the pier turns, when the shots alternate between showing the
swimming man and the approaching pier, is an example of this. One function of the music in
this case is that it contributes to somewhat bridging the disconnection. It establishes a
pronounced sense of forward motion, attributed to the shark as well as to the overall situation
(we can still hear the shark even when we see the shots showing only the swimming man).
This makes it reinforce the vector between the Actor and the Goal in the separate shots as it
connects them and adds direction and momentum. The accented downbeats and the
increasingly faster tempo of the music further underline this function.

It seems clear that the music is not only capable of expressing ideational meanings, but also
does this on several simultaneous (and dynamically shifting) levels. What Kress and van
Leeuwen (2006) call narrative processes – participants, such as Actor and Goal, interacting
connected by vectors – are here partly being expressed musically, as described above. The
Actor (Jaws) is in this scene visually deleted but musically represented by the leitmotif in
interplay with the image of the moving piece of the pier (and the float). At the same time the
music is active in defining conceptual symbolic as well as analytical processes.

This scene illustrates some important aspects of the interplay of visuals and music, which
provide conditions for the music to actively and expressively contribute to the overall
narrative. The initial meaning potential of the music is relatively open, determined by the
material affordances and constraints of the musical sound as well as social and cultural
conditions. Through the multimodal interplay of music and image the musical meaning
becomes more specific, which enables it to actively and expressively contribute to the
narrative. It becomes apparent how the narrative meaning emerges from the ‘intermodal’
processes in interplay with the contexts involved.

Jaws: Interpersonal Aspects

The interpersonal metafunction, the participatory function of communication, involves issues
of ‘who’ and ‘why’. It addresses questions of the relationships and purposes of the
participants involved in the communicational act and thereby also aspects of power, modality
(degrees of truth), attitudes, agency etc. Kress and van Leeuwen (2006) distinguish between
interactive participants and represented participants:

9
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

The former are the participants in the act of communication – the participants who speak and
listen or write and read, make images or view them, whereas the latter are the participants who
constitute the subject matter of the communication; that is, the people, places and things
(including abstract ‘things’) represented in and by the speech or writing or image, the
participants about whom or which we are speaking or writing or producing images (p. 48).

The interpersonal processes of the musical narrative functions discussed here can be seen as
primarily taking place between interactive and represented participants – between the
audience member and the film. Before looking closer at the musical narrative functions at
work at the interpersonal level, it first has to be discussed who the involved interactive
participants are in film communication. The viewers (or rather listeners/viewers) can maybe
be loosely defined as ‘the audience’, even if that is both a hazy and complex definition. From
the producers’ side there might be attempts at defining a ‘target audience’ or ‘imagined
audience’ but the viewer will still remain a relatively anonymous concept from this point of
view. Chatman (1980) suggests the term ‘implied reader’ as well as ‘implied author’. Various
studies exploring different aspects of audience responses also illuminate the complexity of the
audience concept, showing how factors such as age, gender, ethnicity, schooling, family
situation, media habits, interests etc relates to attitudes and meaning-making processes of the
individual listener/viewer (e.g. Barker & Brooks, 1998; Hodge & Tripp, 1986; Wingstedt,
Brändström & Berg, 2008).

Equally complex and hazy is usually the viewer’s idea about who is the producer(s) of the
musical score in a film. Seldom is the musical underscore distinguished from the other aspects
of the movie, making the issue of determining the musical communicator(s) vague and
unreflected – just as the music itself is often experienced on an unconscious and unreflected
level, de-emphasizing the listener’s awareness of any specific ‘musical communicator’. The
implied author might be – more or less consciously – associated with different participants
(one specific or a combination of several), such as the film director, the movie company or
television network, the executive producer, the writer, the composer, the performing actors,
even characters of the narrative or the rather indistinct notion of ‘the film itself’.

Looking at some of the basic intentions of most films, the purpose to tell a story is usually
evident. In the case with the Jaws scene it seems likely that the aim is to express suspense, to

10
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

entertain, to achieve immersion or to frighten the audience. To do this, music’s emotive

dimension is important and it is also made use of here, as described above. The emotive
dimension however works on different levels. Juslin (2001) makes a distinction between
emotions being experienced by the listener (induced emotions) or emotions simply being
recognized (represented or observed) as such on a more cognitive level. In the latter case,
we’re speaking of information about, or description of, the emotions as an ideational function
– in the Jaws scene as attributes of the shark or of the general scene. If the feeling of danger is
induced and felt by the audience, this function will also be partly interpersonal, an emotive
process going on between the film (and its makers) and the audience member. In a viewing
situation however, when experiencing a scene, it is not usually a simple matter of the emotion
being either cognitively understood or emotionally felt. Rather, it is typically a mixture of the
two – but often with one of the functions being more salient at a given moment.

At the very start of the Jaws scene, there is a short moment of ‘introductory’ music, a
dissonant chord quietly building. At first the music is kept in the background, partly masked
by other sounds. The foreboding nature of this music subtly prepares the audience, in a
manner that can be understood as providing a prediction using future tense – an ‘offer of
information’ of what is to come (Kress & van Leeuwen, 2006: 123). As soon as the thumping
leitmotif starts, it is however clearly audible getting more into the aural foreground – and it
stays that way for the rest of the scene. This is done without really having to increase the
volume very much, since the frequency spectrum of the music is now not colliding with the
other sounds, not even when the men start shouting – a technique sometimes referred to as
audio interleaving (Jägerskogh, 2002; also personal communication, April 4, 2008). It is as if
the music is discretely making itself available for listening – in a sense corresponding to what
Kress and van Leeuwen (2006) call an offer. By doing this at the same time as the float starts
moving, the music also helps to call attention to what is going on visually. It not only
ideationally represents the shark, but also at the same time performs an indicative function –
on an interpersonal level alerting the audience to look.

Despite the strong emotive impact of this music, we never really get ‘inside’ or close to the
shark. The music is not used to portray the inner mental processes of Jaws. We are not to
identify with the beast (as for example Max Steiner’s music in the 1933 version of King Kong
makes us do; Palmer, 1990), its psychological processes remain obscure. Rather than inner
attributes, the music emphasizes the outer (the ‘what’ rather than the ‘why’) – letting us feel

11
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

the size, power, movement, unpredictability and vicious intentions of the shark. These
attributes are then what the audience is placed in relation to. This, in combination with Jaws’
invisibility, contributes to installing a fear of the unknown, of the uncontrollable – where the
audience has to imagine what is not represented.

At the same time the musical expression is congruent with the overall narrative. The level of
involvement is high, the music almost naturalistically and with close attention to detail setting
out to paint every nuance of Jaws’ physical attributes, changes in movement and temperament.
This heightens immersion as well as modality. The music convinces us to accept and believe
in what is happening. Actually, the visuals alone in this scene are not really expressing much
excitement or suspense, but in combination with the music and the overall narrative the
expressed events become immersive and believable. In film, music is often used this way, to
achieve high modality, to heighten the emotional credibility – what Gorbman (1987) describes
as lowering the audience member’s ‘thresholds of belief’ (p. 6)

Jaws: Textual Aspects

When it comes to the textual aspects of music much has been written over the years, not least
in fields such as musicology and music cognition (e.g. Meyer, 1956; Cooke, 1959; Temperley,
2001). Most often though, the focus has been on musical structure with little concern to social,
cultural, situational or multimodal contexts – a notable exception of course being van
Leeuwen’s (1999) exploration of the semiotic potentials of speech, music and sound. In
discussing the interactions between music and visuals, it is however not primarily music’s
internal (‘intramodal’) structures that will be of interest here, as much as the intermodal
processes at play.

A fundamental feature of the multimodal design in the Jaws scene is the placement of the
musical sound in relation to the overall narrative structure. This has several implications. To
start with, the musical underscore in the scene is nondiegetic, meaning that it is not part of the
spatio-temporal world of the story told – not heard by the represented participants but only by
the audience. This is, as we know, common practice of the musical underscore in film, the
accepted convention. Still, as Donnelly (2005) remarks, this practice is remarkable and ‘a
most notable anomaly in the system’ (p. 12). Using the music as something that is ‘not-real’
in relation to the diegesis, however opens up for certain meaning-making narrative practices
not available with diegetic music. In a scene like this for example, with the use of an

12
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

orchestral leitmotif representing the shark, making the music diegetic would not be an option
– if nothing else so for reasons of narrative realism. Donnelly (ibid.) suggests that nondiegetic
music is given ‘something of a supernatural character, where it is closer to “the voice of God”,
the transcendent or the supernatural’ (p. 13). The nondiegetic position of the music makes it
possible to aurally communicate and illustrate phenomena, both physical and abstract, that
don’t actually make sounds. In this case we can hear Jaws’ movements and physical as well as
mental attributes without having to consider the diegetic source of the aural representation.

The placement of the music in a world different from the world depicted in the story becomes
a framing device that marks a disconnection of the music from the other narrative modes
representing the told world (image, dialogue, sound effects). But as the individual modes are
narratively disconnected, they are at the same time temporally simultaneous. Through editing
as well as musical composition, the music is carefully placed and timed with the visuals and
other aural elements. The temporal alignment between modes becomes a crucial element of
establishing meaning – including whether the music at all is present or not, also making the
‘musical silence’ a carrier of meaning (Lipscomb & Tolchinsky, 2005). Kress and van
Leeuwen (2006) describe how in film the synchronization between the dialogue track and lip
movements on screen takes the role of a vector, connecting Speech and Speaker, similarly to
how dialogue balloons represent speech in still images. In a similar way, the synchronization
between music and image functions as a vector connecting the visuals with the musical sound
– establishing a joint ‘meaning potential basis’. Chion (1990) relates to this phenomenon
using the term synchresis (combining the two words ‘synchronicity’ and ‘synthesis’),
referring to how visual and auditory elements that are synchronized will be perceived as one
unity: ‘The spontaneous and irresistible weld produced between a particular auditory
phenomenon and visual phenomenon when they occur at the same time [...] independently of
any rational logic’ (p. 63). It is this kind of textual weld that enables the ideationally
informative and descriptive representations of Jaws, as well as the interpersonal processes
described above. While Chion mainly refers to relatively exact instances of synchronization,
such as footsteps and lip synch, it is apparent that simultaneity also in a looser sense sparks
persuasive meaning-making mechanisms.

The technique of audio interleaving, making room for the leitmotif, was mentioned earlier.
This technique as well as the changes in volume, the accented downbeats and increased tempo
described – are different ways of achieving musical salience. Not necessarily to draw

13
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

conscious attention to the music itself, but to enhance musical clarity and thereby making the
relations to the visuals and other narrative elements more articulate. Salient aural elements
will typically be associated with simultaneously salient visual elements and form multimodal
units of joint meaning.

An important textual meaning-making function of music is how it establishes continuity – the

ability of music to provide a sense of temporal coherence to the multimodal weave. Through
rhythmic, harmonic, melodic and timbral devices, music contributes a temporal flow, which
forms rhythmical relationships on many simultaneous levels and between the different
semiotic modes available. Music makes time audible. Visually, the Jaws scene is represented
through a collection of shots containing various kinds of movement; participants falling,
floating, breaking, swimming – as well as fast camera movements in various directions. The
individual shots are all from different angles with different size of frame – and cutting
between the shots is made in a relatively high tempo. The musical sound provides an
immediate continual structure that connects the individual cuts and helps the viewer to make
sense of the visual jumble. A longer continuity is also established that keeps the entire scene
together, giving it a heightened sense of unity. Discussions of how rhythmical relationships
contribute cohesion within and between semiotic modes, and are articulated on several
simultaneous levels, can be found in for example Martinec (2000) and van Leeuwen (1985,
1999, 2005).

The music in the Jaws scene is also textually expressing a formal shape that can roughly be
described as: intro, exposition, climax and coda. This musical form is enhancing and defining
the dramatic form for the whole scene – which is like a very short Aristotelian drama (or
maybe drama according to Freytag) containing exposition, climax and resolution. Another
way to see this can be to look at the exposition, represented musically by the basic leitmotif,
as the Given – according to Kress and van Leeuwen (2006), ‘what we already know’. From
there we are taken to the climax (the accented downbeats), the New (that ‘which is not yet
known’). This New turns into another Given, taking us further to a second New, the slowing
down and ending on a safe long note.

Intermodal textuality is of course not only about simultaneity. Coherence is also achieved in
relation to what has happened before, to intertextuality, the relationship between separate
utterances. If, within a film, each scene is looked at as somewhat separate texts, the

14
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

cumulative meaning-making process characteristic for a leitmotif is depending on the

intertextual progression of the scenes containing the motif. Furthermore, the repeating of the
leitmotif throughout the film also has the textual function of contributing overall continuity
and unity to the entire production. If the Jaws leitmotif later is heard in different situations or
in other movies, this will also have consequences for the meaning-making processes for those
who have seen the original movie Jaws (see the next example). Similarly – a reflection often
heard about the Jaws leitmotif is that it bears a striking resemblance to Igor Stravinsky’s ballet
music The Rite of Spring. The Jaws motif may because of this be associated with aspects more
or less consciously attributed to Stravinsky’s music, such as ‘primitive forces’ or ‘pagan
rituals’. Intertextual connotations like this will influence how the Jaws scene is experienced
and interpreted.

Musical Commenting
The use of music in the Jaws scene exemplifies a few of the manners in which music can
contribute meaning in interplay with visuals in a multimodally told story. The musical
functions include examples from the emotive, informative, descriptive, guiding and temporal
narrative functions suggested initially. These functions are simultaneous, but their relative
salience will continuously and dynamically shift in interplay with the other narrative modes
involved. The involved expressional resources offer a wide range of potential meanings that
can turn more or less specific according to the listeners’/viewers’ interests, and situational and
socio-cultural contexts. Meanings will also dynamically transform according to the
multimodal processes described. Replacing the music in this scene or even shifting the same
music by just a few frames in relation to the visuals will create new intermodal relationships,
suggesting different readings.

Similarly, using the same music in different visual and narrative contexts will uncover
entirely different potentials for meaning. To illustrate this, let’s look briefly at a scene from
the film The Secret of My Success (1987, directed by Herbert Ross):

The main character, Brantley Foster (played by Michael J. Fox) is about to be seduced by his
boss’s wife, Vera Prescott (Margaret Whitton). They are outdoors in her luxurious garden,
and the scene starts as Brantley, dressed in shorts, jumps into the swimming pool. He feels a
little uneasy about the situation. Vera, in a sexy bikini, follows with an elegant dive. The
musical underscore starts playing. It is a two-note motif played by low strings. As the couple

15
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

starts playing around in the water, the music develops. It’s the Jaws’ leitmotif. Brantley is
slightly bewildered when Very pulls off his shorts. The musical downbeats get heavily
accented. Brantley vainly tries to get his trunks back, as Vera mischievously teases him and
starts to remove her bikini top. The water sprays and splashes as they vigorously tumble
around. In the music a dramatic crescendo culminates with a low note ‘stinger’, synchronized
with a shot of the wet bikini hanging from a small cherub statue by the pool.

The music in this scene is virtually the same as the music in the Jaws’ scene described before,
but its meaning is totally different. One possible reading of the scene is that the contrast
between the playfulness of the situation and the serious and dramatic music creates a
humorous effect. The use of the Jaws leitmotif here relies on the assumption that the audience
knows the movie Jaws and is familiar with the music. The intertextual associations make us
compare Vera to the ruthless shark, which can be seen as contributing humour to the scene
(one of several available readings). On the interpersonal level the music is here given a
commenting role, producing a distancing perspective. It is almost as if someone was looking
into the camera giving the audience a knowing and ironic wink. In this sense the music is
doing the equivalent of what Kress and van Leeuwen (2006) describe as a demand –
demanding ‘that the viewer enters into some kind of imaginary relation’ (p. 118), in this case
with the ‘implied producer’ of the film. This is also an example of the rhetorical function
mentioned earlier. Creating intermodal contrast is an often used method to achieve
rhetorically commenting functions. Lipscomb & Tolchinsky (2005: 396) point out how the
use of musical contrast can invite intellectual processing and active participation from the
audience. The scene also clearly illustrates how the meaning of the music changes depending
on the context.

On the ideational level the music supports the physical movement and the bustling stir taking
place in the pool. It also adds a certain emotive nerve that may be especially noticeable for the
viewer who is not familiar with the Jaws leitmotif. After having shown this scene at a
conference, one of the authors was approached by a conference participant who explained: ‘I
haven’t seen the movie Jaws, but I think the music worked very well in this scene – it made
me really worry about whether the woman’s husband might arrive!’

The concluding ‘stinger’ (an accented singular note or harmony, usually synchronized with a
visual event or used to emphasize a reaction to a dramaturgical turn), is here used indicatively

16
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

to (interpersonally) point to the wet bikini. It ‘rhythmicizes’ the visuals at the same time as it
comments, tongue-in-cheek, the not-shown activities of the now bikini-less Vera.

Diegetic Music
In the examples discussed so far the music is used non-diegetically. Thus, there is no
narratively implied demand to account for its sound source. This is facilitated by music’s
generally high degree of representational abstraction. It can certainly be made to represent
quite specific and concrete phenomena, as we could see in the Jaws scene, but essentially
music is an abstract form of communication. A musical expression in film will seldom be
mistaken for concretely being a ‘genuine recording of reality’ in the way that a sound effect
may be perceived – except when the aim is to represent ‘someone playing music’. Kress and
van Leeuwen (2006) point out that ‘the more abstract the sign, the greater its semantic
extension [...] the greater its potential range of uses as a signifier in signs’ (p. 54). Maybe
herein lies much of the reason for music’s readiness to express such a wide and multilayered
spectrum of possible meanings. Music’s seemingly endless ability to combine and blend with
image and other forms of expression builds on its highly abstract affordances, which in film is
also what allows for its high extent of nondiegetic usage. A relatively abstract level of
expression makes more room for the listener’s creative meaning-making activity, providing a
higher potential for narrative immersion.

When the music in film is diegetic, sometimes referred to as source music (as the source of
the music is shown or implied as being present in the spatiotemporal world of the story) the
communicational conditions will somewhat change. There will be consequences for the
meaning-making processes on the ideational and interpersonal as well as the textual level.
Source music will typically be used in similar ways as ambient sound effects, to contribute
aural atmosphere or realism to a setting or situation in the story – such as a radio playing
music in the background or a band playing dance-music at a party.

Ideationally, the music will be perceived as simply being (part of) the physical environment,
rather than non-diegetically describing it. The informational value will concern the fact that
(in this situation/environment) there is music playing, what kind of music is playing, who or
what is playing etc. This will in turn contribute to our understanding of the situation, just as
other diegetic features such as dialogue, ambient or synchronized sound effects, physical
setting, clothes, hairstyles etc, will. Furthermore, the ideational and interpersonal emotive

17
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

function of establishing an overall mood will often be an important factor also with diegetic
music. An interpersonal aspect of diegetic music is how it has a tendency of becoming
transparent to the listener. Once established as being diegetic, the audience will be inclined to
take it for granted – and this will contribute to the music, and thus its narrative functions,
often being even less consciously prominent to the listener than nondiegetic music is.

On a textual level, diegetic music will introduce some new parameters – which in turn will
also affect ideational and interpersonal dimensions. One such textual parameter concerns the
music examples or genres that are used in specific situations. When composing or using music
for nondiegetic purposes, consideration is usually made with respect to the film’s overall style,
instrumentation, use of motifs etc. With diegetic music different considerations are made.
Important aspects will be how the music fills its diegetic purpose, aspects of authenticity and
situational relevance will usually be essential. This can for example result in the use of well-
known music pieces, which might not be appropriate if the music was nondiegetic.
Synchronization to movement is another textual aspect that will follow somewhat different
principles when the music is diegetic. The impression here will often be that the music
initiates movement, or that the movements follow the music (such as when we see people
dancing or marching to music) – while tightly synchronized nondiegetic music will rather be
perceived as expressing or describing movement, as when using mickey-mousing (musically
expressing or imitating physical movement, usually synchronized to onscreen action).

Diegetic music is however also often simultaneously used for more specific narrative
purposes, according to the narrative functions discussed with nondiegetic music – a technique
sometimes known as source scoring. At times music in film will glide between diegetic and
nondiegetic functions, blurring the line between the two.

The Birds
As an example of how diegetic music may work narratively, let’s look at a scene from Alfred
Hitchcock’s The Birds (1963). In the scene to be discussed, the music is clearly diegetic and is
at the same time demonstrating multi-layered narrative functions of vital dramaturgical
importance:

One of the main characters, Melanie Daniels (played by Tippi Hedren) is sitting outside a
small school house, smoking a cigarette. She is waiting for the class inside to finish. We can

18
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

hear the young schoolchildren inside the house singing an ‘a capella’ version of the lively and
cheerful children’s song ‘Risseldy, Rosseldy’. Behind Melanie’s back we can see a
playground with a climbing frame. During a long sequence we can follow how big black
crows, unnoticed by Melanie, gradually gather on the frame. After several minutes Melanie
eventually turns her head and is horrified to see the place swarmed with menacing black birds.
The children’s song, which has been going on during the entire scene, continues for a while
as Melanie hurries towards the house to warn the children and their teacher.

Textually, the song is (intramodally) structured so there is a seemingly endless row of verses,
and the last melodic line of each verse is repeated several times using nonsense lyrics. This
gives the song a repetitious but playful quality. The music is in a major key and performed at
a brisk tempo. Intermodally, the music is clearly diegetic as we get to see the singing children
right before the start of this scene. That this scene has music at all makes it stand out, since
there is otherwise no nondiegetic music soundtrack in this movie – which is quite unusual for
a horror movie. In the scenes where the birds attack however, the sound of the birds is
carefully designed using, among other things, tape loop techniques of ‘musique concrète’. The
sound of the children’s song is not loud, coming from inside the school house. Still, it is
relatively aurally salient since there are almost no other sounds in this scene. However, maybe
partly resulting from the transparency of source music, it is experienced as dramaturgically
being in the background while the visual narrative achieves salience.

Ideationally and symbolically, the music represents the schoolchildren. Being just aurally
represented they are continually present throughout the scene, while the image intercuts
between shots of Melanie and the birds – resulting in a two-part audio-visual counterpoint
(with the music textually framing, and providing continuity to, the scene). No individual voice
is discernable, the children are depicted as a group – with the a cappella sound emphasizing
their vulnerability. The voices project possessive attributes such as their young age,
togetherness, the playful mood and their unknowing innocence.

The ‘aural size of frame’ interpersonally marks ‘close to medium social distance’ rather than
‘personal social distance’, emphasizing the sense of togetherness. The contrast between what
we hear and what we see is striking on several levels. The purity of the children’s voices
presents a stark contrast to the menacing threat of the birds, establishing a dissonant
intermodal tension between music and visuals, which on an interpersonal level serves as a

19
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

comment to the drama. This tension is further accentuated by the differences in dramaturgical
curve between the image and the music. The harmonic consonance and relatively low emotive
energy of the music, as well as its unchanging and static expression contrasts the gradually
increasing suspense of the visuals, culminating in Melanie’s sudden shock. Such use of
musical ‘detachment’ (Kress & van Leeuwen, 2006: 136), what Chion (1990) calls
anempathetic music, would be difficult to justify using nondiegetic music. With diegetic
music however it strengthens the dramaturgically important use of contrast, adding depth,
making room for reflection and effectively leaving much of the emotional work to the
audience – emphasizing the interpersonal dimension and heightening immersion.

Textually, the music is, as mentioned, characterized by persistent repetitions between and
within verses. This provides a strong sense of continuity and nerve to the scene. At the same
time the repetitious music, where one verse continually is added to the others and the repeated
lines at the end of each verse are extending the form even further, represents a structural
parallel to the visuals – a rhyme to the continual adding of birds to the playground. This
works, on an interpersonal level, as a wry comment to the dramatic situation at play – the sort
of sardonic irony that is often present in Hitchcock’s films. The unrelenting repetitions may
also, interpersonally, after a while generate a growing sense of uneasiness or even irritation
for the listener. This further contributes to heightening the experience of dramatic tension
within the scene.

The, at first, seemingly innocuous use of source music, at closer inspection turns out to be part
of a precise and careful design where it is employed as a narrative tool indispensable for the
scene.

Conclusion
Kress and van Leeuwen (2006) point out how our sense of sight is generally considered more
reliable than our sense of hearing and they comment on how ‘seeing has, in our culture,
become synonymous with understanding’ (p. 163). This view is of course quite pertinent and
true. As the above examples show however, there is in narrative multimedia more to see than
meets the eye. When image, dialogue, sound effects and music combine into multimodal texts,
a ‘chemical reaction’ seems to take place. The resulting whole is, if maybe not greater,
certainly different than the sum of the parts. The communicational act takes place on several
levels and through many simultaneous channels or modes, but our experience is perceived as

20
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

being one. Since such experiences so often are interpreted as being of primarily visual nature
the effect is, as stated initially, that what (we think) we see is to a large extent determined by
what we hear.

The emerging multimodal possibilities of new media affect most aspects of modern society,
including how we learn, how we work and how we play. New media also emphasize ‘the fact
that all meaning-making is in its nature multimodal’ (Cope & Kalantzis, 2000: 211).
Acknowledging the narrative impact of music in multimodal storytelling such as film,
television or computer games, it will be of increasing importance to further explore how
musical elements combine in aural statements that in turn combine with visuals and other
narrative modes to form multimodal expressions. Such explorations will surely find argument
to add, adjust and modify certain concepts and grammatical issues according to the cultural
and material properties unique for the modes and interactional processes in question. Looking
at narrative media music as maybe the largest source of musical experience in our daily lives,
it is clear that a better understanding of its meaning-making functions is of great importance.

References
Barker, M. & Brooks, K. (1998). Knowing Audiences: Judge Dredd, Its Friends, Fans and
Foes. Luton: University of Luton Press.
Bouzereau, L. (Writer/Director/Producer). (2000). ‘The Making of Jaws’ [DVD
Documentary]. On Jaws: Anniversary Collector’s Edition [DVD Motion Picture]. United
States: Universal Studios Home Video.
Chatman, S. (1980). Story and Discourse: Narrative Structure in Fiction and Film. Ithaka,
NY: Cornell University Press.
Chion, M. (1994). Audio-Vision: Sound on Screen. New York: Columbia University Press.
Cook, N. (1998). Analysing Musical Multimedia. Oxford: Oxford University Press.
Cooke, D. (1959). The Language of Music. Oxford: Oxford University Press.
Cope, B & Kalantzis, M (2000). ‘Designs for Social Futures’, in B. Cope & M. Kalantzis
(eds.) Multiliteracies: Literacy Learning and the Design of Social Futures, pp. 203-234.
London: Routledge.
Donnelly, K.J. (2005). The Spectre of Sound: Music in Film and Television. London: BFI
Publishing.
Gorbman, C. (1987). Unheard Melodies: Narrative Film Music. Bloomington, Indiana:
Indiana University Press.
Halliday, M.A.K. (1978). Language as a Social Semiotic. London: Edward Arnold.
Hanslick, Eduard (1955). Om det sköna i musiken [On the Musically Beautiful; orig. Vom
Musikalisch-Schönen, 1854]. Uppsala: Schismens skriftserie 1.

21
 Wingstedt et al.: Narrative Music, Visuals and Meaning in Film

Hickman, R. (2006). Reel Music: Exploring 100 Years of Film Music. New York: Norton.
Hodge, R. & Kress, G. (1988). Social Semiotics. Cambridge, UK: Polity Press.
Hodge, R. & Tripp, D. (1986). Children and Television: A Semiotic Approach. Cambridge:
Polity Press.
Jewitt, C. (2006). Technology, Literacy and Learning: A Multimodal Approach. London:
Routledge.
Juslin, P.N. (2001). ‘Communicating Emotion in Music Performance: A Review and a
Theoretical Framework’, in P.N. Juslin and J.A. Sloboda (eds.) Music and Emotion:
Theory and Research, pp. 309-337. Oxford: Oxford University Press.
Jägerskogh, A. (2002). Visst ska det höras vad dom säger och sjunger? [Of course what they
say and sing should be heard?]. Unpublished manuscript, Royal Institute of Technology,
Stockholm.
Kress, G. (2003). Literacy in the New Media Age. London: Routledge.
Kress, G. & Van Leeuwen, T. (2006). Reading Images: The Grammar of Visual Design, (2nd
edition). London: Routledge.
Lindstrand, F. (2006). Att göra skillnad: Representation, identitet och lärande i ungdomars
arbete och berättande med film [Making difference: Representation, identity and learning
in teenagers’ work and communication with film] (Doctoral dissertation). Stockholm: HLS
Förlag.
Lipscomb, S.D. & Tolchinsky, D.E. (2005). ‘The Role of Music Communication in Cinema’,
in D. Miell, R. MacDonald & D.J. Hargreaves (eds.) Musical Communication, pp. 383-404.
Oxford: Oxford University Press.
Martinec, R. (2000). ‘Rhythm in Multimodal Texts’. Leonardo, Vol. 33, No. 4, pp. 289-297.
Meyer, L.B. (1956). Emotion and Meaning in Music. Chicago: University of Chicago Press.
Palmer, C. (1990). The Composer in Hollywood. London: Marion Boyars Publishers.
Temperley, D. (2001). The Cognition of Basic Musical Structures. Cambridge: The MIT Press.
Van Leeuwen, T. (1985). ‘Rhythmic Structure of the Film Text’, in T.A. Van Dilk (ed.),
Discourse and Communication: New Approaches to the Analysis of Mass Media Discourse
and Communication, pp. 216-232. Berlin: De Gruyter.
Van Leeuwen, T. (1999). Speech, Sound and Music. London: Macmillan.
Van Leeuwen, T. (2005). Introducing Social Semiotics. London: Routledge.
Van Leeuwen, T. & Jewitt, C. (eds.) (2001). Handbook of Visual Analysis. London: Sage
Publications.
Wingstedt, J. (2004). ‘Narrative Functions of Film Music in a Relational Perspective’, paper
presented at ISME – Sound Worlds to Discover, Santa Cruz, Tenerife, 14-16 July.
Wingstedt, J. (2005). Narrative Music: Towards an Understanding of Musical Narrative
Functions in Multimedia (Licentiate thesis). Luleå: School of Music, Luleå University of
Technology, Sweden. (Available at http://pure.ltu.se/ws/fbspretrieve/199292)
Wingstedt, J., Brändström, S. & Berg, J. (2008). ‘Young Adolescents’ Usage of Narrative
Functions of Media Music by Manipulation of Musical Expression’. Psychology of Music,
Vol. 36 (No. 2): pp. 193-214.

22
 Article 2

Wingstedt, J., Brändström, S. and Berg, J. (2008):

‘Young Adolescents’ Usage of Narrative Functions of Media Music by
Manipulation of Musical Expression’. Psychology of Music, Vol. 36 (No. 2),
pp. 193-214.

© 2008 Society for Education, Music and Psychology Research.

A RT I C L E 193

Young adolescents’ usage of Psychology of Music

Psychology of Music
narrative functions of Copyright © 2008
Society for Education, Music
media music by manipulation and Psychology Research
vol 36(2): 193–214 [0305-7356

of musical expression (200804) 36:2; 193–214]

10.1177/0305735607085012
http://pom.sagepub.com

JOHNNY WINGSTEDT
R O YA L C O L L E G E O F M U S I C , S W E D E N
S T U R E B R Ä N D S T RÖ M A N D JA N B E RG
L U L E Å U N I V E R S I T Y O F T E C H N O L O G Y, S W E D E N

A B S T R A C T This study investigates usage and knowledge of musical narrative functions

in contemporary multimedia. A group of young adolescents were given the task of
adapting musical expression, using the non-verbal research tool REMUPP, to fit different
visual scenes shown on a computer screen. This was accomplished by manipulating
seven musical parameters: instrumentation, tempo, harmonic complexity, rhythmic
complexity, register, articulation and reverb. They also answered a questionnaire giving
information about their musical training and media habits. Numerical data from the
manipulation of the musical parameters were analysed to search for tendencies within
the group with regard to the musical expression in relation to the different visual scenes
shown. The results showed a large degree of in-group consensus regarding narrative
functions of music, indicating knowledge about musical narrative codes and conventions.
Also, the results were clearly influenced by factors such as the participants’ musical
training, gender and habits of music listening, playing computer games and watching
movies – highlighting the complexity of learning and pointing to the impact of the
increasing availability of narrative media on our attitudes and knowledge.

KEYWORDS: musical functions, musical parameters, narrative codes, narrative music

The new media and communication technologies are transforming the ways we meet
and interact with music. As an ingredient of communication media such as televi-
sion, film and computer games, music is gradually changing and expanding its roles
in our daily lives.
The study of the role of sound and music in multimedia settings is so far largely
a neglected field. Remarkably so, since music and sound are often important expres-
sive and narrative elements used in contemporary media such as film, television and
computer games. Considering the high degree of exposure to this kind of narrative
music (henceforth referred to as media music) in young people’s everyday lives, there
are good reasons to assume that it contributes to shaping attitudes, opinions and

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
194 Psychology of Music 36(2)

knowledge – including knowledge about musical narrative functions, codes and con-
ventions.
Tagg and Clarida (2003) discuss how institutionalized studies of western music,
and literature on European music history, traditionally have ignored the subject of
film music, its functions and composers. In Scandinavian formal music education,
media music is still typically a blind spot and is rarely discussed in depth. Slowly, how-
ever, the subject is gaining recognition. In some higher level study programmes (such
as the teacher education programme at the Royal College of Music in Stockholm),
film music is being offered on elective courses. In the UK, the GCSE and GCE A-level
exam syllabuses now include the formal study of set-work films and television pro-
grammes and practical composition for the moving image. However, there is still
much to be done to further develop knowledge about this growing area – and to pro-
vide teachers with knowledge and skills about the subject.
This article will describe the design and results of a study using the music soft-
ware tool REMUPP (Relations between Musical Parameters and Perceived Properties),
designed for controlling musical expression by real-time manipulation of musical
parameters such as tempo, instrumentation and articulation (Wingstedt, Liljedahl
et al., 2005). The overall purpose of the study is to investigate usage and knowledge –
concerning practices of convention as well as aspects of creativity – with children
and youth regarding musical narrative functions in contemporary multimedia.
A characteristic feature of modern society is the increased interaction between people
and technology. New technology requires new kinds of skills and knowledge – but is also
the source of new knowledge. This new knowledge concerns not only technology itself,
but also various societal and cultural phenomena related to the technological changes.
Kress (2003) has described how, in this new ‘age of media’, the book is being
replaced by the screen as the dominant medium for communication – changing the
basic conditions for the concept of literacy. The centuries-long dominance of writing
is giving way to a new dominance of the image. But, of course, this new literacy not
only involves visual communication; rather, in the new media today, we are making
sense, or trying to make sense, out of an intricate assortment and multimodal combin-
ation of different media: images, written and spoken text, video, animations, sound,
music and so on. What creates meaning is above all the complex interplay of the dif-
ferent modes of expression involved.
Even though a large part of the music we encounter on a daily basis can be at-
tributed to the media music category, paradoxically our relation to and conscious
knowledge about this form of musical expression is often on a rather unreflecting and
subconscious level. Our understanding about the communicative functions and
conventions of this kind of music is frequently of an intuitive and relatively unsophis-
ticated nature. Media music becomes a black box that is taken for granted and
becomes invisible – or unheard (Gorbman, 1987). Nevertheless, it ‘works’. As listen-
ers, we assimilate complex sets of musical codes and conventions, which enable us to
decode and interpret the various narrative functions performed by the music in inter-
play with the other narrative elements involved. This is a type of knowledge we nor-
mally do not get from formalized learning in music classes – but simply from
experiencing the mere presence of music in our lives and through the functional roles
that media such as film or computer games play in society. Learning typically takes

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
 Wingstedt et al.: Young adolescents’ usage of narrative functions of media music 195

place in informal situations where the act of learning, so to speak, becomes ‘unin-
tentional’ (Ericsson, 2002).
Many recent studies have explored different aspects of youth, popular culture and
music (e.g. DeNora, 2000; Folkestad, 1996; Goodwin, 1992; Gullberg, 1999; Nilsson,
2002), often focusing on popular music. However, general knowledge and conscious
opinions about popular music that is actively sought and discussed among peers dif-
fer in many respects from those about media music. Some of the characteristic traits
of media music have already been mentioned, such as its pronounced narrative func-
tion, the complex interplay with other narrative media elements and low degree of
conscious salience as the visual elements and narrative context take ‘priority of atten-
tion’. Also, especially in film (and increasingly in computer games), the music sound-
track often represents alternative musical genres compared to the music otherwise
listened to by the audience.
The present study explores how usage of, and knowledge about, functions and con-
ventions of narrative music are displayed within a group of young participants and
how different backgrounds in musical training and media habits might influence
their creative choices. The usability of the REMUPP interface as a means for study-
ing musical narrative functions is also examined.

Methodology
Twenty-three participants, 12–13 years old, were given the task of adjusting and
shaping music to make it match and enhance as much as possible the perceived
expression of three different 3D animations depicting various physical settings (‘envir-
onments’). The task was accomplished using the music manipulation tool REMUPP
(see ‘The REMUPP interface’ later). The manipulations were recorded as numerical
data, for statistical analysis. In addition, the participants also completed a question-
naire where they answered questions about their movie-watching, music-listening
and computer game-playing habits – and also about their musical training.
Although performed in what can be seen as a laboratory setting, the study to be
described does not include all the criteria associated with a strictly ‘true experiment’.
Most notably, the participants were not randomly allocated (see ‘Participant selection and
the questionnaire’ later), no control groups were used and several dependent variables
were in play (see ‘The parameters’ later). Also, the independent variables can be seen as
relatively complex (see ‘Animations in 3D’ and ‘Basic results of the questionnaire’ later).
A term sometimes used for this type of study, lacking some of the essential features of
a ‘true experiment’, is quasi-experiment (Cook and Campbell, 1979; Coolican, 2004).
However, for simplicity, the term ‘experiment’ will be used throughout this article.
Additionally, it should be mentioned that the entire study also includes a stimu-
lated recall interview session with each of the participants, carried out directly after
the completed experiment sessions. The analyses and results from these interviews
are not, however, reported here, but will be the subject of a future article.

T H E T E S T E N V I RO N M E N T
In planning the study, some basic questions were raised: How can the type of intu-
itive understanding that is typical for media music be studied? How can this implicit

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
196 Psychology of Music 36(2)

knowledge be turned into something explicit? The strategy chosen was to seek ways
to put the knowledge into action rather than words – or at least have action comple-
ment the words – and thereby make the knowledge visible. To accomplish this, the
musical analysis tool REMUPP was designed.

The REMUPP interface

REMUPP is a software-based tool designed for testing various musical functions, for
use within different disciplines of music research. It was developed by the Interactive
Institute’s studio Sonic in Piteå, Sweden and allows for investigating selected musical
parameters in interplay with a musical and situational context. The term musical
parameters here implies structural and performance-related elements such as tonality,
mode, harmonic and rhythmic complexity, register, instrumentation, tempo, timing
and articulation.
By manipulating controls presented graphically on a computer screen, participants
can change the expression of an ongoing musical piece by adjusting selected musical
parameters. In the present study, the music was combined with 3D animations to
provide opportunity for exploring the relational interplay of music with visual nar-
rative media.
Managing the parameter controls requires no previous musical training. Having
the participants manipulate the music makes REMUPP a non-verbal tool where the
user responds to the musical experience within ‘the medium of music’ itself, without
having to translate the response into other modes of expression such as words or
drawings. An overall aspiration of REMUPP is to try as much as possible to minimize
the negative ‘laboratory situation’ effect (Coolican, 2004: 105). The multimedia set-
ting and interactive control of the software make the use of REMUPP akin to playing
a computer game, which is intended to heighten the authenticity of the experience –
and also introduce a dimension of creativity into the test situation.
In REMUPP, the participant’s manipulations of the parameter controls are
recorded into the software and can be output in the form of numerical data, avail-
able for statistical analysis. Furthermore, the resulting music, including all the manip-
ulations on a time-axis, can be played back in real time, making it possible to study
the creative process as well as the aural end result. Further descriptions about the
design, possible uses and initial experiments with REMUPP can be found in several
separate articles (Berg and Wingstedt, 2005; Wingstedt, Berg et al., 2005).

Musical narrative function

The study takes as a starting point one of the musical narrative functions presented
in an earlier paper, which discusses various types, categories and classes of functions
that music has in multimedia narrative situations. The narrative function that will be
focused on here belongs to a category called Describing the physical setting (included
in the Descriptive class) – more specifically, the musical narrative function of describ-
ing a concrete physical setting, such as ‘the sea’, ‘a busy city street’, ‘outer space’, etc.
(Wingstedt, 2004).
This type of descriptive narrative function can be found in music from many dif-
ferent time periods and cultures. In traditional western classical music, this function
is most often used as an example when discussing what is known as programme music,

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
 Wingstedt et al.: Young adolescents’ usage of narrative functions of media music 197

i.e. musical compositions intended to depict or suggest non-musical incidents, ideas

or images.
Focusing on the description of a physical setting makes it possible to design situ-
ations that are relatively stable in a time-perspective. Spotlighting the ‘setting’, rather
than ‘action’ or ‘continuity’ functions, makes the scenes more constant and steady.
This makes the test easier to carry out – it becomes more straightforward for the par-
ticipant to handle, and the interpretation of the empirical data becomes more man-
ageable for the researcher. The ambition and expectation is that in this test,
experience can be gained upon which to build in later experiments where time-related
elements can successively be introduced.

Animations in 3D
The graphical interface of this version of REMUPP was designed so that, besides the
visually presented controls for manipulation of musical parameters (here using
‘faders’), there was also a 3D-animated sequence shown, depicting a physical setting
or location – a physical environment.
Three different visual settings were designed; they are here referred to as movies:

1. Movie 1 (‘City’): City night – unfriendly neighbourhood. A dark alley under an old
highway bridge. Broken windows on shabby buildings, a wrecked car in a cor-
ner, steam emerging from the ground, a fire in an old rusty barrel, blood on the
ground, a broken public phone. Dark colours, some objects barely visible, some
light is given from old neon signs and from the fire in the barrel (Figure 1).
2. Movie 2 (‘Space’): In space – looking at eternity. Inside a huge spaceship, looking
out to space through a giant window. A slowly rotating space station is in sight
as well as Saturn and a distant nebula – another spaceship passes by. Colours
are relatively mellow, a greyish dim light inside the spaceship, outside a black sky
with many stars, Saturn is yellow (Figure 2).
3. Movie 3 (‘Lake’): Sunny day – picnic by the lake. A picnic basket on a blanket on
the green grass by a small lake with water lilies. Butterflies flutter about, a big
tree is gently swayed by a slight breeze. Bright, warm green and blue colours
(Figure 3).

FIGURE 1 Movie 1: ‘City’.

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
198 Psychology of Music 36(2)

FIGURE 2 Movie 2: ‘Space’.

FIGURE 3 An example of REMUPP’s test interface – Movie 3 (‘Lake’) with faders controlling
musical parameters.

To keep the focus on the actual setting, there are no people in these environments.
Since people usually tend to attract attention and will be associated with emotional
attributes – and often become objects of personal identification, etc. – it was decided
to keep the environments empty of visible humans.
The graphics were realized as animations, but with the movements used spar-
ingly, so there is no visible plot or story – they can be thought of as ‘moving still
images’. The idea was to give an impression of these places being alive, ongoing, in

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
 Wingstedt et al.: Young adolescents’ usage of narrative functions of media music 199

process – representing ‘present tense’. A still picture would give a different impres-
sion, like a frozen moment in time – more of a ‘past tense’. The purpose was to try
to make the visual impression as absorbing as possible, to install a sense of partici-
pation and immersion – to inspire a good response on the music task and to try to
avoid an otherwise potentially hampering response to the ‘laboratory situation’.
One added advantage of using digital 3D graphics is that the technology allows
for detailed control over graphical parameters such as light and shadow, colour,
shape, object size and placement, textures and angles. The visual elements can
thereby be precisely defined, facilitating comparison with other visuals. When used
as a part of the REMUPP interface, the animations were played back as QuickTime®
(from Apple Computer, Inc.) movie files.

The basic music examples

Music (basic music examples) was composed for the participants to use as a fundamen-
tal material – to manipulate and shape in real time to the task given. Composing music
for use with REMUPP differs in many respects from composing for traditional settings.
The most important aspect is that the basic music should allow for the parameter
changes made by the participants. This has to work on both a technical and an artis-
tic level, so that the music clearly responds to the expressional changes made at the
same time as it maintains a musically adequate performance. A professional game and
media composer created three different basic music examples for use in the test.
The three basic pieces of music were actually composed with the different
movies in mind, so that their basic performance in the composer’s mind represented
(1) ‘City Night’, (2) ‘In Space’ and (3) ‘Picnic Lake’ respectively. However, when presented
to the participant, at the start of every trial, each of the seven individual parameters
(see below) was set to a random value. This way, any given music example would ini-
tially sound quite different in different trials, to avoid systematic errors stemming
from the influence of the initial music on the participant’s manipulations. At the
same time, the original intention of the composer would thus to a certain degree be
concealed, to leave more creative freedom to the participant.

The parameters
Seven different musical parameters were chosen to be available to the participants for
influencing and altering the musical expression. The parameters were chosen accord-
ing to previous research on how specific musical elements affect various perceived
expressional musical qualities (Berg et al., 2005; Gabrielsson and Lindström, 2001).
The following seven parameters were selected for the participants to use for alter-
ing the musical expression in real time:

1. Instrumentation (instrument set);

2. Tempo (beats per minute);
3. Harmonic complexity (degree of consonance–dissonance);
4. Rhythmic complexity (rhythmic activity);
5. Register (pitch level by octaves);
6. Articulation (staccato–legato);
7. Reverb (reverberation level).

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
200 Psychology of Music 36(2)

The Instrumentation parameter involves a complex set of sub-parameters that will

affect the perceived musical experience in different ways. One aspect is the actual tim-
bre of the individual instrument sound, including differences in overtone spectra, etc.
Another important factor is that a certain instrument, or group of instruments, is
often associated with a specific musical genre. The perception of an instrument’s
sound is therefore to a high degree determined by cultural conventions – and thus
also by personal taste.
For this experiment, three different instrumentation sets were designed:

● Instrument Set 1: ‘Rock’, with instruments such as electric distorted guitar, elec-
tric bass and Hammond organ.
● Instrument Set 2: ‘Electronic’, with pronounced synthetic sounds of ambient nature.
● Instrument Set 3: ‘Symphonic’, with traditional orchestra instruments, emphasiz-
ing woodwind and strings.

These three instrumentation sets were used because they were considered to represent
different characteristic and contrasting genre-specific sound worlds of common use in
narrative music situations. Since this parameter is nominal (rather than ordinal), the
three sets are presented in different positions on the controlling fader – randomized for
each trial – to avoid systematic error.
The Tempo parameter is a fundamental means of influencing the musical expres-
sion. It is measured in beats per minute (bpm) and in REMUPP the test administrator
sets the maximum and minimum tempo boundaries individually for each basic ex-
ample. Thus ‘absurd’ tempi (extremely slow or fast) can be avoided. The controlling
fader changes the tempo continually, with the lowest position being the slowest tempo.
The Harmonic Complexity parameter is in this test accomplished by adding or mut-
ing tracks for instruments carrying voices involved in the harmonic structure. This gives
the participant between three and five different degrees of consonance vs. dissonance,
depending on the basic music example used, to choose from. The most consonant ver-
sion (the lowest position on the fader ⫽value ‘1’) will have the music play nothing more
dissonant than major or minor triads. Higher positions on the fader successively alter
the harmonic structure into more complex and dissonant harmonic structures.
The Rhythmic Complexity parameter technically works similarly to the Harmonic
Complexity parameter, by adding or muting tracks. The fader changes the music from
simple rhythmic structures (lowest on the fader) to gradually more complex and
rhythmically dense and active structures, using three different levels.
The Register parameter changes the pitch of the music by transposing instrument
voices up or down in octaves, depending on the position of the fader – the lowest pos-
ition being the lowest available octave. This fader offers four or five different positions,
depending on the basic music example.
The Articulation parameter simply changes the length of the notes played, from
staccato (short notes with space between, lowest position on the fader, lowest value ⫽1)
to legato (long and tied notes, highest value ⫽120). It is presented as a continuous
parameter.
The Reverb parameter continuously changes the amount of the reverberation
effect with the lowest fader position providing no reverb, giving a dry sound. Values
range from 1 to 100.

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
 Wingstedt et al.: Young adolescents’ usage of narrative functions of media music 201

These seven parameters were presented in all the trials in the indicated order, from
left to right on the screen. The reason for their placement was to a certain degree
determined by how fundamental they were considered to be to the musical structure.
The first two were thought of as key parameters – both in function and in the actual
creative process of composing. The third, fourth and fifth are fundamental in estab-
lishing the inner musical structure on a compositional level in traditional western
music, while the sixth parameter is essentially a performance related-parameter. Finally,
the seventh parameter is typically a part of the mixing process in contemporary
music production, and is therefore one of the last features to be added to determine
the expression of the music. Of course, it is not assumed that all participants will ne-
cessarily always start from the left and work their way stepwise to the right. Many
would probably use the parameters in no set order and several times – and indeed
many did, especially after the first few trial rounds – but anyway it was considered
that an organized system might be helpful. Also, the parameters are arranged so that
every second fader (1, 3, 5 and 7) controls a parameter that is predominantly timbre-
related, while the others are mainly temporal-related. This is an attempt to provide
variety and contrast to the interface.

P RO C E D U R E
Music is a multidimensional phenomenon, including dimensions of the musical sound
and the individual person as well as societal, situational, aesthetic and functional
dimensions. In research as well as practice related to music education, of course, all
of these dimensions are important to consider – but it is a big challenge trying to
include more than just one of the dimensions in the same study. In an attempt to
approach this challenge, the current study makes use of several methodological tools.
Results from an experiment and a questionnaire are reported in this article.

Participant selection and the questionnaire

Teachers of six different elementary school classes (grades 6 and 7, in the Swedish
school system corresponding to ages 12–13) in the city of Piteå in the north of
Sweden were contacted and a suitable time for the researcher to come and visit each
class was arranged. When visiting the classroom, the researcher gave a short presen-
tation about the project and distributed a questionnaire to all the pupils. In the ques-
tionnaire, the children were asked about their habits listening to music, playing
computer games and watching movies on television or at the cinema. The amount of
time they spent doing these activities was indicated on a scale graded in six steps:
(a) Never; (b) A couple of times/year; (c) A couple of times/month; (d) A couple of
times/week; (e) Up to two hours/day; (f) More than two hours/day. Furthermore, they
were asked about their musical training: whether they could play an instrument (or
sing), and, if so, whether they had formally studied the instrument and for how long.
They were also asked if they wanted to participate in the experiment using REMUPP.

The experiment session

Children who had indicated that they wanted to participate in the experiment – in all
23 participants – were contacted and a session for them scheduled. In the test situa-
tion, they were introduced to the REMUPP interface via a short introduction by the

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
202 Psychology of Music 36(2)

test administrator, and by written instructions on the screen. Under the guidance of the
test administrator, they first completed a training trial, intended to familiarize them
with the task. They were then left on their own to complete the trial series.
The instruction given to the participants was:
Look at the picture presented on the screen and try to think about what impression it
gives you. When the music starts playing, use the controls to adjust the musical expres-
sion so that it fits the visual environment shown onscreen as well as possible.

They were also told that there could be no ‘wrong’ answers, and were encouraged to
play and experiment with the controlling faders to find a setting that best expressed
their personal interpretation of the visuals.
At the beginning of each trial, the movie was shown in silence for a few seconds.
Then the basic music started playing and the controls were made visible below the
picture. The controlling faders were presented without any written labels to make the
participant focus on their functions only by listening to their effect on the musical
expression when moved. The faders controlling discrete (rather than continuous)
values had small markings indicating the positions to which the fader would lock to
make its function clear to the user. The continuous controllers had no markings, but
otherwise all the faders were visually identical.
Each of the three movies was presented three times, each time in a new combin-
ation with one of the three available basic music examples – thus adding up to a total
of nine trials per session. The order of the trials was randomized for each session to
avoid order effects of the movies and the music examples.
The REMUPP software recorded the entire session including all the manipulations
made by the participant. This way the finished trial could be played back, the move-
ments of the faders could be watched and the resulting musical performance listened
to. The final result of each trial was also made available as numerical values. This
included the setting of each parameter, the total trial length and the number of times
each parameter was manipulated during a trial. This data thus became available for
statistical analysis. The participants spent on average 20 minutes (ranging from 10
to 42 minutes) completing the nine trials.
After every completed session, the participants got to rate a favourite version of
each of the three movies (‘which one are you most satisfied with?’), based on how
well they thought the music fitted the visuals. These selected versions will be referred
to as Preferred Trials (PT) when we discuss the results in the next section.

Results
The results presented here are mainly based on statistical analyses of the numerical
data from REMUPP – when appropriate, in combination with data retrieved from the
questionnaire. The objectives of the current statistical analyses were to:

1. Search for patterns within the group with regard to the set values of the musical
parameters in relation to the different movies (physical settings). Such intrinsic
patterns can be seen as indicators of the participants’ knowledge and usage of
existing extrinsic codes and conventions of narrative music, and also as express-
ing a willingness to follow these conventions.

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorize
distribution.
 Wingstedt et al.: Young adolescents’ usage of narrative functions of media music 203

2. Explore how the different musical backgrounds, gender and media habits of the
participants influence their creative choices when manipulating the musical
parameters.
3. Explore how narrative codes are expressed musically, i.e. how certain musical
properties or parameters are related to specific narrative expressions.
4. Explore the usability of the REMUPP interface as a means for studying musical
narrative functions.
Because of the complexity of the material, tentative interpretations of some of the
results will be attempted directly in connection to the presentations of the results below.

BA S I C R E S U LT S O F T H E Q U E S T I O N NA I R E
Five dichotomous categories were retrieved from the completed questionnaires:
Gender, Musical Training, Music Habits, Game Habits and Film Habits.
The Gender category showed a larger representation of males than females, with
16 boys and 7 girls participating in the study.
The ‘Yes’ subgroup (n ⫽13) in the Musical Training category had received three
years or more of formal training in playing an instrument; the ‘No’ subgroup (n ⫽10)
had little or no formal training or instrument skills. Three participants had previously
taken music lessons (more than two years ago), but stopped after less than a year.
They were attributed to the ‘No’ subgroup. (In what follows, when indicating the total
number of members in a sample uppercase ‘N’ is used – if a limited portion of the
total sample is indicated lowercase ‘n’ is used.)
The Habits subgroups (High and Low respectively, see later) were constructed from the
answers on a six-grade scale (as described in ‘Basic results of the questionnaire’ earlier).
The original intention was to consider the ones indicating ‘a couple of times/month’ or
less as Low ratings, and ‘a couple of times/week’ or more as High ratings. However, there
turned out to be very few answers in this lower range (Music Habit: one person, Game
Habit: three persons and Film Habit: two persons). In order not to make the subgroups
too unequal in size, it was therefore decided to put the dichotomous break point one step
higher, including ‘a couple of times/week’ in the Low subgroup.
The Habits categories represent the results from questions about the amount of
recreational music listening (Music Habits, High: n ⫽13, Low: n ⫽10), playing com-
puter games (Game Habits, High: n ⫽9, Low: n ⫽ 14) and watching movies (Movie
Habits, High: n ⫽8, Low: n ⫽ 15). There was considerable overlap between the differ-
ent category subgroups, meaning that participants belonging to the High subgroup
in one category would often belong to the Low subgroup in another category and vice
versa. Only three participants did not belong to the High subgroup in any category.
The REMUPP trial results of the category subgroups will be compared in ‘The
Categories’ section later.

C O R R E S P O N D E N C E S B E T W E E N M OV I E S A N D PA R A M E T E R S
In comparing the values of the seven musical parameters, in relation to (and
between) the three different 3D animations (here called movies) – and also later for
the different background and habit categories – the ␹2-test was used for nominal
data. The continuous parameter data showed weak correspondence to a normal dis-
tribution. As a consequence, these data – together with the ordinal parameters – were

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
204 Psychology of Music 36(2)

all treated as ordinal. Non-parametric analysis methods were therefore used on all the
data that were not nominal. For this, the Kruskal-Wallis test was used when compar-
ing cases of several different given conditions (i.e. musical parameters vs. movies or
music examples), the Mann-Whitney for comparing dichotomous subgroups and
occasionally Kendall’s tau-b with small data sets.
The parameter data were assigned to two different groups – All and PT, respect-
ively. The All group consisted of data drawn from all the performed trials (23 partici-
pants ⫻9 trials ⫽207 trials in total). Equally divided between the three movies, this
made 69 trials for each Movie Number. The PT group consisted of data drawn only
from the trials where the participants had rated the trial as ‘the preferred version’ of
each movie (see ‘The experiment session’ earlier). On a small number of occasions,
some participants could not decide which trial to rate as the most preferred for a cer-
tain movie. On these occasions, they were allowed to select two trials as top rate,
resulting in the total number of PTs equalling 73 (rather than the expected 69).

Movie number: The All group

Examining all parameters, comparing their values for the three different movies in
the All group (using a ␹2-test for Instrumentation and the Kruskal-Wallis test for the
other parameters), significant patterns were found for the parameters
Instrumentation (p ⬍ .001) and Reverb (␹2(2) ⫽13.82, p ⬍.001). The distribution of
the instrumentation sets chosen for each movie is shown in Table 1. Especially for the
‘Lake’ movie, there is notable consensus within the group for the use of the
Symphonic instrument set, which might be interpreted as an example of a narrative
convention (see ‘Discussion’ later for more on this).
The Reverb settings showed a smaller value for the bright outdoor ‘Lake’ setting
(median value ⫽34) than for the darker and more enclosed, and also more hostile or
mysterious, settings of ‘City’ and ‘Space’ (median ⫽54 for each movie). These differ-
ences in value can be seen as relevant from the perspective of the narrative, with
open outdoor settings naturally giving less reverberation – and the higher level of
reverb might also be seen as an example of a narrative convention contributing to
the drama of the ‘City’ and ‘Space’ movies.
When the All group was divided into three sections, corresponding to Music
Number (basic music example), and each section was examined separately – to com-
pensate for possible bias due to the different structures of each basic music example –
two more significant parameters were found: Rhythmic Complexity (␹2(2) ⫽9.77,
p ⬍.01) and Register (␹2(2) ⫽7.26, p ⬍.05). Both these parameters showed a lower
value for Movie 1 (‘City’) and a higher value for Movie 3 (‘Lake’) – with a value in

TA B L E 1 Distribution of the instrument sets chosen for movies in group All (N⫽207)

Instrument set
Movie number Rock Electronic Symphonic
Movie 1: ‘City’ 20 40 9
Movie 2: ‘Space’ 23 31 15
Movie 3: ‘Lake’ 5 13 51

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
 Wingstedt et al.: Young adolescents’ usage of narrative functions of media music 205

between for Movie 2 (‘Space’). The lower register for City might be interpreted as
corresponding to the darker colours and more hostile mood (compared to Lake). At
the same time the ‘Lake’ movie is the one containing the highest level of movement
of the three, with butterflies fluttering restlessly from flower to flower – the ‘Space’
movie could be considered to display a little less movement, with a huge space station
slowly rotating and a small spaceship sometimes idling past the window. Finally, the
‘City’ movie is the most static of the three. This relationship of relative movement
corresponds well with the different levels of Rhythmic Complexity (activity) chosen by
the participants.

Movie number: The PT group

Examining the parameter values for the three movies in the PT group showed simi-
lar significant patterns for the Instrumentation, Register and Rhythmic Complexity
parameters to those in the All group.
In addition to the seven musical parameters manipulated by the participants in each
trial, the PT group also makes an eighth parameter available: Music Number, which
is the basic music example (of three available) that is included in the preferred choice.
A ␹2-test of the correspondence of preferred Music Number to each Movie Number
showed a significance of p ⫽.008. The distribution of the music examples is seen in
Table 2. In the ‘City’ and especially the ‘Lake’ movies, the Music Numbers preferred by
the participants correspond with the intentions of the original music examples.

T H E CAT E G O R I E S
Looking at the dichotomous categories retrieved from the questionnaire (see ‘Basic
results of the questionnaire’ earlier), several significant differences were found
between the different subgroups with regard to the use of musical parameters for nar-
rative music purposes. In the following, a ␹2-test was used for nominal data, for ordinal
data the Mann-Whitney test was used if nothing else was indicated.

Gender
In the Gender category, significant differences were found between females and males
in the following parameters: Harmonic Complexity ( p ⫽.032), Rhythmic Complexity
( p ⫽.001) and Reverb ( p ⫽ .045). Interestingly, in all these cases, the girls were gen-
erally using higher values: more dissonance (or fuller harmonic content), more rhyth-
mic activity (especially in the ‘Space’ and ‘Lake’ movies) and more reverb than the
boys.

TA B L E 2 Distribution of Music Number preferred for each movie in group PT (n ⫽ 73)

Music number

Movie number ‘City Night’ ‘In Space’ ‘Picnic Lake’

Movie 1: ‘City’ 12 8 4
Movie 2: ‘Space’ 10 6 10
Movie 3: ‘Lake’ 5 2 16

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
206 Psychology of Music 36(2)

The reasons for these differences can only be speculated on at this stage. Part of the
explanation might depend on differences in maturity between girls and boys of this
age group (von Tetzchner, 2005). Studies showing a relationship between maturity
and perception of harmonic and rhythmic structures have been discussed by Sundin
(2001), for example. Another reason for the differences might be due to gender-
dependent media preferences. O’Neill (1997) and Russell (1997) discuss gender differ-
ences in music preferences as well as general attitudes towards the music subject. The
results of the questionnaires in this study suggest such gender differences also for com-
puter game preferences – where the girls generally have indicated preferring games
such as ‘Sims’ (where you build, design and decorate) whereas the boys more often
have specified ‘action and adventure’ oriented games. Differences in preference –
including the differences in the music used for these different game genres – might, at
least in part, explain some of the gender differences found.

Musical training
The Musical Training category showed significant differences between the Yes
(received training) and No (little or no training) subgroups in the Rhythmic
Complexity (p ⫽.045) and Articulation (p ⫽ .024) parameters belonging to the All
group. In both cases, the Yes subgroup preferred higher values than the No subgroup,
using more rhythmic activity and more legato articulations.
One reason for the differences might be that a higher level of training and experi-
ence results in a higher degree of sophistication or maturity (here relating to the sub-
ject of music), which in turn would accommodate a preference and taste for more
complex musical structures. The similar results for the Rhythmic Complexity param-
eter also by the Gender group (relating to the discussion of maturity) and the Music
Habits group (see next section) seem to support such an interpretation (the groups
have been cross-checked to eliminate spurious correlations). Furthermore, the
Articulation parameter is predominantly a performance-related parameter (rather
than structural) – the results here might be an indication of a greater awareness of
‘touch’ or ‘delivery’ by the group that is learning to play an instrument.

Music habits
Rhythmic Complexity (p ⫽ .003), Register (p ⫽ .041) and Reverb (p ⫽ .011) were
parameters where significant differences were found between the Low and High sub-
groups concerning musical listening habits. For these parameters, the general trend
was again that higher values were chosen by the High Habit subgroup – resulting in
more rhythmic activity, a higher register (especially for the ‘Space’ movie) and slightly
more reverb. Also, Instrumentation (p ⫽.018) showed significant differences between
the two subgroups. For the ‘Space’ movie, the High Habit subgroup preferred the Rock
instrument set, whereas the Low Habit subgroup primarily chose the Electronic
instrumentation (putting Rock as the least preferred). For the ‘City’ movie, the High
Habit subgroup again preferred the Rock instrumentation – together with the
Electronic instrumentation (a tied first place) – while the Low Habit subgroup pre-
ferred the Symphonic instrument set.
On the questionnaires, most participants had listed rock, pop and related genres
as their favourites. It can be speculated that the High Habit group has simply chosen

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
 Wingstedt et al.: Young adolescents’ usage of narrative functions of media music 207

the instrumentation that corresponds with their musical taste, having generally
shown a high interest in musical listening. The High Habits group’s higher settings
for the ordinal parameters show tendencies similar to those of the other ‘more experi-
enced’ groups, as discussed previously. Interestingly, a closer examination of the
material shows that the Low Habits group’s lower values of Rhythmic Complexity and
Register are especially noticeable for the ‘City’ and ‘Space’ movies. These settings can
be seen as narratively reflecting the dark, sombre and low activity qualities of these
movies. It thus seems that the music of the High Habit group in a sense ‘goes against’
the narrative conventions (which is also the case in their instrumentation for the
‘Space’ movie) – maybe reflecting that general taste-related choices are prevailing
over narrative considerations. The future analysis of the interviews might further
clarify some of the reasons for these results.

Game habits
In the Game Habit (PT) category, there is a significant tendency for the High Habit
subgroup in how they used Rhythmic Complexity for the different movies (Kendall
tau-b, used here because of small data set, ⫽.35, n ⫽ 30, p ⫽.019): the ‘Lake’ movie
got a value indicating higher activity than the other movies. For the Low Habit sub-
group there is no such significant tendency (Kendall tau-b ⫽.18, n ⫽43, p ⫽.201).
Assuming that high Rhythmic Complexity is a sign of a narrative convention for the
‘Lake’ movie, due to the higher amount of physical movement in this movie, this
result could be interpreted as the High Habit subgroup – because of their greater
experience of playing computer games – being more aware of this convention.

Film habits
The High and Low subgroups with differences in Film Habits showed significant
results in their different uses of Instrumentation (p ⫽.044), Articulation (p ⫽.001)
and Register (p ⫽.021). In Movie 3 (‘Lake’), the High Habit subgroup displayed a sig-
nificantly stronger concordance in the choice of the Symphonic instrument set
(91.7%) as their favourite choice compared to the Low Habit subgroup (64.4%).
Although both subgroups favoured the ‘convention’ in this case (see the Discussion
section later), the High Habit group was much more in accord, which might indicate
their higher awareness of musical narrative conventions, gained from more experi-
ence in watching movies. The High Habit subgroup generally also produced lower
values for Articulation (less legato) and for Register (darker sonorities, especially
pronounced for the ‘Space’ movie, suggesting narrative conventions).

C O R R E S P O N D E N C E S B E T W E E N M U S I C N U M B E R A N D PA R A M E T E R S
Besides the different movies, several other features might be considered as influenc-
ing the settings of the musical parameters. One such factor would be the basic music
example (Music Number) being used. To examine the effect of Music Number on the
settings of the musical parameters, an analysis was carried out similar to the one
done for the relations to Movie Number.
Using the Kruskal-Wallis test, the musical parameter showing the highest degree
of significant correspondence to the choice of Music Number was Tempo (␹2(2) ⫽38.98,

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
208 Psychology of Music 36(2)

p ⬍.001), with especially the Music Number 1 (‘City Night’) being associated with a
markedly faster tempo than the other two music examples. Interestingly, Tempo is the
single parameter showing no significant correspondences to the choice of Movie
Number or background/habit categories. The strong correspondence of the Tempo
parameter to the Music Number parameter may at least in part explain the weaker
correspondence of Tempo to other factors.
Some weaker correspondences were also found between Music Number and the
parameters Register, Rhythmic Complexity and Reverb – but only after first having
divided the trials according to Movie Number and then comparing the effects of Music
Number on the parameters. Generally speaking, it seems clear that the effects of
Music Number on the different parameters were weaker than the effects of Movie
Number or the different backgrounds or habit categories.

SUMMARY OF PREFERRED TRIALS AND COMPARISON WITH THE ORIGINAL MUSIC

In this section follows a summary of the parameter values set for each of the movies
in the PT group. There is also comparison made of these values with the original set-
tings made by the composer of the basic music examples. As mentioned earlier, these
original settings were concealed from the participants by randomizing the parameter
values at the start of each trial. This comparison is made viewing the original settings
as including suggestions of ‘narrative conventions’ by the original composer – besides
being the result of the composer’s creative work. As the three basic music examples
were composed with each of the movies in mind, the original music makes use of –
or at least forms a relationship to – more general extrinsic conventions of narrative
music that are part of western media culture. The musical expression suggested by the
participants as a group can be thought of as another – in-group (intrinsic) – conven-
tion, which also forms a relationship to more general cultural/societal conventions.
Studying the correspondences between the music of the PT group and the original
music examples might then serve as a tool for making these general cultural conven-
tions more visible.
An overview of the parameter settings made by the PT group and the original set-
tings can be seen in Table 3. For the PT group’s values of the nominal parameters
(Instrument Set and Music Number) the mode values are used. As the rest of the
parameters are being treated as ordinal, the median values are displayed for them.
For the parameters Harmonic Complexity, Rhythmic Complexity and Register, the
ordinal values indicated can broadly be interpreted as: 1 ⫽low (as in low complexity
or low register), 2 ⫽medium, 3 ⫽high. A pair-wise comparison for each movie shows
good correspondence between the pairs, especially for Movie 3 (‘Lake’). The largest
disparities are found in the Music Number parameter of Movie 2 (‘Space’) and the
Instrument Set parameter of Movie 1 (‘City’). In the ‘City’ movie, the ‘City Night’
music example is preferred but with the Electronic instrument set (rather than the
Rock instrumentation suggested by the original music); in the ‘Space’ movie, the PT
group prefers the same instrument set as the original (Electronic), but does not favour
the original ‘Space’ music example. The Music Number parameter is naturally funda-
mental to the musical structure, and the Instrumentation parameter has also been
found to be perceived to dramatically change the musical sound when compared
to other parameters (Berg et al., 2005). In all, there seem in this comparison to be

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
 Wingstedt et al.: Young adolescents’ usage of narrative functions of media music 209

TA B L E3 A summary of the parameter values of the Preferred Trial settings (mode or median
values; n ⫽ 73) and a comparison with the settings of the original music
Movie 1 (‘City’) Movie 2 (‘Space’) Movie 3 (‘Lake’)

PT values PT values PT values

(mode or (Original (mode or (Original (mode or (Original
Parameter median ) values) median) values) median) values)
Instrument 2 1 2 2 3 3
Set (mode) (Electronic) (Rock) (Electronic) (Electr.) (Symphonic) (Symph.)
Music number 1 1 1&3 2 3 3
(mode) (‘City (‘City (‘City (‘In Space’) (‘Picnic (‘Picnic
Night’) Night’) Night’ & Lake’) Lake’)
‘Picnic
Lake’)
Tempo (bpm) 110.5 130 88.5 68 89 70
Harmonic 2 2 2 3 2 2
Complexity
Rhythmic 2 3 2 2 3 3
Complexity
Register 2 2 2.5 3 3 3
Articulation 81.5 80 75.5 80 64 80
(0–120)
Reverb 50 0 46 25 37 20
(0–100)

certain common features suggesting commonalities in the expression of musical nar-

rative functions between the PT settings and the original settings of the basic music
examples. An attempt to interpret these values in terms of musical narrative func-
tions will be made in the Discussion section.

Discussion
The results reported so far indicate that the participants to a large degree display a
collective consensus about certain narrative musical functions. This intrinsic consen-
sus can, in turn, be interpreted as mirroring extrinsic norms – existing conventions
that we encounter in film, computer games and other narrative multimedia.
An attempt to interpret some of the musical narrative codes and conventions con-
veyed in this study by means of manipulating musical parameters (see Table 3) can
be summarized: the pastoral scene by the lake is expressed by the group of partici-
pants by the use of the ‘symphonic’ instrumentation consisting primarily of flute,
strings and harp – a classic cliché for expressing pastoral settings in western musical
tradition. The darker and more hostile urban ‘City’ scene, as well as the more high-
tech and mysterious ‘Space’ scene, are portrayed using electronic instruments. In the
two latter scenes the register is also generally lower, producing darker and more
sombre sonorities than in the brighter ‘Lake’ scene. The basic tempi of the ‘Space’ and
‘Lake’ scenes are kept relatively low, reflecting the tranquillity of these situations –
although the rhythmic activity in the ‘Lake’ scene is higher, maybe expressing the

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
210 Psychology of Music 36(2)

movements of the fluttering butterflies. The tempo of the ‘City’ scene is slightly
higher, although with a low rhythmic activity, which can be seen as reflecting a
higher degree of suspense. The more confined locations of the ‘Space’ and ‘City’
scenes are portrayed by the use of more reverb than the open air, and less dramatic,
‘Lake’ scene. The articulation of the music for the ‘Lake’ scene is also shorter, although
not down to a full staccato, providing an airy quality allowing more ‘breathing’ into
the musical phrasings.
According to the above interpretations, it seems that the narrative functions sug-
gested are well in line with the intention of this experiment to focus on the descrip-
tive qualities of the music. The descriptive narrative category (Wingstedt, 2004)
includes functions such as describing physical atmosphere (descriptions of abstract
character, such as ‘the time of day’), describing physical setting (more concretely
describing an environment) and describing physical movement. The above interpret-
ations include these functions, with tempo, register and articulation here contributing
to atmospheric descriptions – instrumentation and reverb more concretely describing
the physical settings – and rhythmic complexity providing physical movement.
Emotional narrative qualities are at the same time emphasized by (primarily) the
Instrumentation, Tempo and Register parameters.
The musical parameter that is kept most constant between all three movies is
Harmonic Complexity, which is set to medium values for all scenes. This setting pro-
vides a rich but not overly dissonant harmonic texture. There are several possible rea-
sons that could explain the lack of variety in these values. One reason might be that
the three different visual scenes simply provided stimuli that were perceived as equal
with regard to the harmonic expressiveness. Another reason might be related to the
general music taste of the participants – the medium setting might be perceived as
the generally most pleasant and therefore preferred. Still another reason might be
that the harmonic qualities of the music were relatively difficult to perceive for the
participants of this age-group and background – or simply regarded as less important.
This could be due to the perceptional abilities of the participants – or to the construc-
tion of the REMUPP interface, including the basic music examples. Future reports of
the interviews conducted may shed more light on this.
One factor that may partly explain the relatively strong group consensus – other
than as a display of conventions – is the homogeneity of the participants taking part
in this experiment. They all come from the same small northern Swedish city of Piteå,
sharing many common societal values and being of the same age group. An inter-
esting continuation of this study would be to perform comparative research with par-
ticipants from other cultural settings.
Also, the non-randomized allocation of the volunteering participants can be a fac-
tor contributing to the demonstrated patterns. It is possible that the children who
chose to participate have a greater interest in the topic of narrative media, and thus
have a higher degree of familiarity with narrative conventions.
Offering several musical parameters for manipulation at the same time introduces
a potential risk for interaction effects between the parameters, affecting the result.
Examples of such interaction effects are, for example, whether a certain choice of instru-
mentation affects the perceived register – or whether different tempo settings affect the
perceived rhythmic complexity, or vice versa. This could lead to certain parameter

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
 Wingstedt et al.: Young adolescents’ usage of narrative functions of media music 211

settings determining the settings of other parameters, rather than the settings being
determined by the different movies (a similar situation to the potential impact of the
basic music examples discussed in ‘Correspondences between music number and
parameters’ earlier). To check for this, relevant statistical tests (see ‘Correspondences
between movies and parameters’ earlier) have been repeated while isolating param-
eter settings thought to be potential interaction hazards, such as the ones mentioned.
The results have proven to be quite stable, but since the interaction combinations
quickly get more complex with an increased number of parameters and settings, this
factor should not be underestimated.
Considering the potential risks and challenges, the study has been designed in an
attempt to meet and reflect the complexity of the music topic. An effort has been
made to provide in different ways an interesting, absorbing and stimulating test envir-
onment. Making several musical parameters available, rather than just one, con-
tributes to the game-like creative agency, serving to heighten the ‘experimental realism’
(Coolican, 2004: 107, emphasis added) – which in turn will lessen the participant’s
evaluation apprehension (concern about being tested) and strengthen the ecological
validity. The simultaneous manipulation of several parameters also weakens partici-
pant expectancy, making it less likely that the participants will try and guess what
would ‘please the experimenter’.
The various results gained in this study indicate the usefulness of the REMUPP
interface as a tool for exploring musical narrative functions. In manipulating the
musical parameter controls, the participants achieve meaning through ‘musical
actions’, which is different from using language. For example, to just say that a visual
setting is ‘scary’ is not the same as expressing it musically. To determine ‘scary’ by
(for example) assigning a low register, setting a certain degree of harmonic disson-
ance and rhythmic activity, adding more reverberation and slowing down the tempo,
demands a commitment to a higher degree than just saying the word. Not only the
music, but the interweaving between different modes – in this case especially visuals
and music – is what creates meaning in the multimodal ensemble (Kress et al.,
2001: 25). REMUPP provides conditions for such a kind of interweaving. As argued
in the introduction above, there is a tendency for the audience to treat media music
on a relatively subconscious and unreflecting level because the visuals tend to achieve
salience. Working with the REMUPP interface has made it possible to bring the music
to the front, to make visible the implicit knowledge about musical narrative functions.
By communicating narratives musically, the participants draw from their experience
of historically and culturally established conventions and codes – resulting in an over-
all consensus about musical narrative functions. This consensus can be seen as a result
of knowledge, as ‘evidence of learning’ (Kress et al., 2001: 143). The results of the
study hint at the complexity of the issue of musical narrative functions and also high-
light the complexity of learning in general. Not only do differences in musical training
seem to influence the creative musical choices, but so also do gender and differences in
habits of listening to music, watching movies and playing computer games. These lat-
ter factors point to the impact of the increasing availability of narrative media, not only
on our attitudes and opinions, but also on our skills and knowledge.
The results strengthen the assumption that high exposure to media and its asso-
ciated music contributes to the shaping of knowledge of and attitudes to media music.

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
212 Psychology of Music 36(2)

We learn, not only through the ‘multimodal texts’, but also about the modes them-
selves from simply using media in informal situations. This gives rise to questions
about how learning takes place in pronounced multimodal settings, how we become
‘multimodally literate’ by using the various modes – and the role of music in such
situations. The results also fuel the continually important questions about what kind
of knowledge musical knowledge is. Given the multimodal opportunities of new com-
munication media, further questions arise about how the emergence of new media is
changing the conditions for learning in informal settings and situations, and what
consequences the changing conditions for informal learning bring to traditional
learning situations in school.
Future reports from the interviews conducted will hopefully add substance to some
of the results and tentative interpretations of the results presented in this report. At this
point, the statistical material can mainly indicate answers to the ‘what’ questions – the
aim of the interviews will be to also contribute some answers to the ‘why’ questions
and to include matters related to creative issues, including choices made (conscious
or intuitive) in order to follow or deviate from narrative and expressional codes and
conventions. The given communicative modes of the new narrative media obviously
influence our knowledge as well as the process through which we achieve meaning.
An important prospective task for contemporary and future research is to investigate
how meaning is constituted in connection to the new media – and the role of music
in this process. The discourse comprising ‘the school subject music’ has everything to
gain in relating in a holistic way to our everyday musical realities. In the emerging
new prospects for media and entertainment, music is poised to take on new respon-
sibilities. The more music plays an active part in our daily lives, the more important
it becomes to study its functions, possibilities and potential.

AC K N OW L E D G E M E N T S

The authors wish to thank Mats Liljedahl for programming the REMUPP interface, Stefan
Lindberg for composing the basic music examples and Jacob Svensson for creating the 3D ani-
mations. Thanks also to the participants of the study.

REFERENCES

Berg, J. and Wingstedt, J. (2005) ‘Relations between Musical Parameters and Expressed
Emotions – Extending the Potential of Computer Entertainment’, in S. Zhou, Z. Ying and
L.S. Ping (eds) Proceedings of ACM SIGCHI International Conference on Advances in Computer
Entertainment Technology ACE 2005, Valencia, Spain, 15–17 June, pp. 164–71. New York:
ACM.
Berg, J., Wingstedt, J., Liljedahl, M. and Lindberg, S. (2005) ‘Perceived Properties of
Parameterised Music for Interactive Applications’, paper presented at the 9th World
Multi-Conference on Systemics, Cybernetics and Informatics WMSCI, 10–13 July, Orlando,
FL, USA.
Cook, T.D. and Campbell, D.T. (1979) Quasi-Experimentation: Design & Analysis Issues for Field
Settings. Boston, MA: Houghton Mifflin Company.
Coolican, H. (2004) Research Methods and Statistics in Psychology. London: Hodder &
Stoughton.

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
 Wingstedt et al.: Young adolescents’ usage of narrative functions of media music 213

DeNora, T. (2000) Music in Everyday Life. Cambridge: Cambridge University Press.

Ericsson, C. (2002) Från guidad visning till shopping och förströdd tillägnelse: Moderniserade villkor
för ungdomars musikaliska lärande [From guided exhibition to shopping and preoccupied
assimilation: Modernized conditions for adolescents’ musical learning] (PhD dissertation),
Publications from the Malmö Academy of Music: Studies in Music and Music Education
no. 4. Malmö: Malmö Academy of Music.
Folkestad, G. (1996) Computer Based Creative Music Making: Young People’s Music in the Digital
Age. Gothenburg: Acta Universitatis Gothoburgensis.
Gabrielsson, A. and Lindström, E. (2001) ‘The Influence of Musical Structure on Emotional
Expression’, in P.N. Juslin and J.A. Sloboda (eds) Music and Emotion: Theory and Research,
pp. 223–48. Oxford: Oxford University Press.
Goodwin, A. (1992) Dancing in the Distraction Factory: Music Television and Popular Culture.
Minneapolis, MN: University of Minnesota Press.
Gorbman, C. (1987) Unheard Melodies: Narrative Film Music. Bloomington, IN: Indiana
University Press.
Gullberg, A-K. (1999) Formspråk och spelregler: En studie i rockmusicerande inom och utanför
musikhögskolan [Playing by the rules: A study of rock music performance within and outside
of the music college]. Piteå: Musikhögskolan i Piteå.
Kress, G. (2003) Literacy in the New Media Age. London: Routledge.
Kress, G., Jewitt, C., Ogborn, J. and Tsatsarelis, C. (2001) Multimodal Teaching and Learning:
The Rhetorics of the Science Classroom. London: Continuum.
Nilsson, B. (2002) Jag kan göra hundra låtar: Barns musikskapande med digitala verktyg [‘I can
make a hundred songs’: Children’s creative music-making with digital tools] (PhD
dissertation). Lund: Lund University, Malmö Academy of Music.
O’Neill, S.A. (1997) ‘Gender and Music’, in D.J. Hargreaves and A.C. North (eds) The Social
Psychology of Music. Oxford: Oxford University Press.
Russell, P.A. (1997) ‘Musical Tastes and Society’, in D.J. Hargreaves and A.C. North (eds) The
Social Psychology of Music. Oxford: Oxford University Press.
Sundin, B. (2001) Barns musikaliska utveckling [Children’s musical development]. Stockholm:
Liber.
Tagg, P. and Clarida, B. (2003) Ten Little Title Tunes. New York: The Mass Media Music
Scholars’ Press.
Von Tetzchner, S. (2005) Utvecklingspsykologi: Barn- och Ungdomsåren [Developmental
psychology: Childhood and adolescence]. Lund: Studentlitteratur.
Wingstedt, J. (2004) ‘Narrative Functions of Film Music in a Relational Perspective’, paper
presented at ISME – Sound Worlds to Discover, 14–16 July, Santa Cruz, Tenerife, Spain.
Wingstedt, J., Berg, J., Liljedahl, M. and Lindberg, S. (2005) ‘REMUPP – An Interface for
Evaluation of Relations between Musical Parameters and Perceived Properties’, in
S. Zhou, Z. Ying and L.S. Ping (eds) ACM SIGCHI International Conference on Advances in
Computer Entertainment Technology, ACE 2005, Valencia, Spain, 15–17 June, pp. 346–49.
New York: ACM.
Wingstedt, J., Liljedahl, M., Lindberg, S. and Berg, J. (2005) ‘REMUPP – An Interactive Tool
for Investigating Musical Properties and Relations’, in S. Fels and T. Blaine (eds) The
International Conference on New Interfaces for Musical Expression, NIME-05, pp. 232–35.
Vancouver: University of British Columbia.

JOHNNY WINGSTEDT is an Assistant Professor at the Royal College of Music in Stockholm,

teaching film scoring and music technology at the Department of Music and Media

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
214 Psychology of Music 36(2)

Technology. He has many years’ experience as a composer for film, television and theatre, in
Sweden, Japan, the USA and China – and has a degree in Film Scoring from Berklee College
of Music in Boston, MA. He is currently also a PhD candidate at the Dept. of Music and
Media in Piteå, Sweden, and defended his licentiate thesis, on narrative music, in 2005.
Address: Royal College of Music, Dept. of Music and Media Technology, PO Box 27711,
SE-115 91 Stockholm, Sweden. [email: [email protected]]

took his piano pedagogic certificate and soloist diploma at the School
S T U R E B R Ä N D S T RÖ M
of Music in Gothenburg. For more than 20 years he has combined piano teaching with public
appearances as a pianist. During the 1980s, Sture was engaged in research. In 1995, he
defended his doctoral thesis in pedagogy at the University of Umeå, Sweden. He is currently
Research Professor in Music Education at Luleå University of Technology, Sweden. His areas
of specialization include education sociology and presentations include those to the ISME
Research Commission 1994, 1998 and 2000.
Address: Dept. of Music and Media, Luleå University of Technology, PO Box 744, SE-941 28
Piteå, Sweden. [email: [email protected]]

JA N B E RGreceived his PhD in 2002 with a dissertation on evaluation of spatial quality in

multichannel systems. He is currently an Assistant Professor in audio technology at Luleå
University of Technology, Sweden, where he carries out research work on audio quality
evaluation and supervises a number of PhD students in areas involving different forms of lis-
tening tests. At the Interactive Institute in Sweden, Dr Berg has also studied the relation
between musical parameters and the perceived characteristics of music. In June 2006,
Dr Berg chaired the Audio Engineering Society’s 28th International Conference on Audio
Technology in the Future – Surround and Beyond in Piteå, Sweden.
Address: Dept. of Music and Media, Luleå University of Technology, PO Box 744, SE-941 28
Piteå, Sweden. [email: [email protected]]

© 2008 Society for Education, Music, and Psychology Research. All rights reserved. Not for commercial use or unauthorized
distribution.
 Article 3

Wingstedt, J., Brändström, S. & Berg, J. (2008):

Making Meaning of Media Music: Expressions of Knowledge about Musical
Narrative Functions.

Manuscript submitted for publication.

Making Meaning of Media Music: Expressions of Knowledge about
Musical Narrative Functions

Johnny Wingstedt
Royal College of Music, Stockholm

Sture Brändström and Jan Berg

Luleå University of Technology, Dept. of Music and Media, Piteå, Sweden

Abstract
Narrative media music, used for narrative purposes in multimedia such as film
and computer games, is often our largest source of daily musical experience. To
explore knowledge of narrative functions of media music, a study was designed.
First, 23 young adolescents were asked to adapt musical expression, using a
music software tool, to make it fit different visual scenes. This paper presents
the second part of the study, where each participant was interviewed in a
stimulated recall situation, commenting on their own musical expressions. Five
different verbal statement types could be discerned: the Unclear, Intuitive,
Associative, Analytical and Transformative types. These statements were seen
as ‘evidence of learning’, reflecting various aspects of Swanwick’s (1994)
concepts of intuitive and analytical knowledge. Combining the verbal
statements with how each participant musically demonstrated conformity or
non-conformity to narrative conventions, contributed to a fuller and more
nuanced account of their expressed musical knowledge. It seems likely that
some of this knowledge is achieved by simply using narrative multimedia in
informal situations. It is concluded that we need, both as educators, researchers
and as users of music, to relate to the impact of – and learning opportunities
offered by – the escalating information society.

Keywords: Film music, Game music, Media music, Multimodality, Music education, Musical
knowledge, Narrative music.

With the emergence of new digital media, the degree of exposure, access and control of music
in our daily lives is rapidly changing. We are gradually becoming more actively involved with
music as aesthetic experience, as entertainment or as a means of communication. Today,
music can be bought and owned, downloaded, edited and carried around. It can be copied,
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

organized, re-listened to and examined like never before in history. More than ever we are
using music as a personalized, interactive means for experience and expression, making it an
active and integrated part of our daily lives. The new media also allows for seamless
integration of music with other modes of expression, such as written or spoken language, still
and moving image, animations etc. Narrative media music, music used for narrative purposes
in for example film, games, advertising and other multimodal expressions is made available
not only in cinemas or on television but also by the use of technology such as computers,
Internet, portable media players and mobile phones.

Looking at narrative media music as a mode of communication and representation, it becomes

clear that it is not only used by composers, film makers or game makers as a tool for
communicating a story. Music is also used by the audience, listeners or game players as a tool
for making sense of the experience. And to be able to use this tool, knowledge is required.
Knowledge of musical communicational functions is an essential requirement for being able
to make sense of the plethora of multimodal expressions that we encounter everyday.
Questions arise about how this knowledge is constituted.

Small (1998) introduced the term musicking, emphasizing the active role not only of the
music composer or performer, but also of the listener or other participants of a musical event.
Bratt-Rawden and DeNora (2005), in discussing the learning of the complex skills of how to
‘musick’, point out how it is a process involving enculturation, the accumulation of social
competences. As listeners we learn to make use of musical narrative conventions, we develop
an awareness of available semiotic resources1 (van Leeuwen, 2005) enabling us to interpret
and make sense of the various narrative functions performed by music in interplay with other
narrative elements involved. This type of knowledge we are maybe not typically getting from
formalized learning in school – but more often from simply experiencing the mere presence of
music in our daily lives and through the functional roles that media like film or computer
games play in society. Furthermore, narrative media music tends to be transparent and
experienced on an unconscious and unreflecting level, while visuals and dialogue achieve
salience. Still it seems to actively contribute to how we make meaning from a multimodally
told story (Wingstedt, Brändström & Berg, in press). Ericsson (2002) points out that even in
such casual and unfocused listening situations musical learning seems to take place through
what he calls preoccupied assimilation (p. 229) where the act of learning, so to speak,
becomes unintentional. Ericsson (ibid.) even suggests that it should be possible to view a

2
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

person who listens a lot to music, but who is not necessarily structuring the experience
formally or analytically, as musically knowledgeable (p. 204). Another way to put it is to
suggest that a frequent listener will develop a certain kind of musical literacy by simply
listening to music in informal situations.

This paper will present a study that examines aspects of how music is understood to
contribute meaning in multimodal expressions such as film and computer games. First, a
group of young adolescents were asked to design and adapt musical expression, using a music
software tool, to make it fit different visual scenes. Then, in a second part of the study they
were interviewed about their efforts. The contents and results of these interviews is what will
be presented in this paper.

Ideas and models of how we make meaning not just from language but from many different
modes of representation, such as image, sound, gesture etc, have been suggested in recent
theories of multimodality and social semiotics (e.g. Hodge & Kress, 1988; van Leeuwen,
1999, 2005; Kress & van Leeuwen, 2001, 2006). Different communicational modes are here
seen as having their own distinctive affordances and constraints for representation, which are
materially, socially and culturally determined. Each mode individually bears meaning, but in
the communicational process meaning typically emerges from the interweaving between and
across modes within a multimodal system. Communication, meaning making, learning and
knowledge are viewed as closely linked processes, where learning is seen as socially situated
and as a dynamic process of sign-making – a series of processes of transformation and
internalisation of signs. We learn from all available modes (and their interactions), each mode
offering different affordances and specializations in the process of learning. Studies of
multimodal processes at play in various learning situations have been presented by for
example Jewitt (2006), Jewitt and Kress (2003), Kress, Jewitt, Ogborn and Tsatsarelis (2001)
and Lindstrand (2006).

In this paper, music will be viewed and treated as a mode of communication and
representation – a resource for making narrative meaning in interplay with other modes, and
also for expressing musical knowledge. Similarly, speech will be viewed as a distinctive mode
making available different means for expressing knowledge of music. Looking specifically at
the concept of musical knowledge, Swanwick (1994) introduces the notions of intuitive and

3
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

analytical kinds of knowledge, based on theories of Croce (1992). These concepts will also be
referred to and exemplified in the presented study.

Purpose and Questions

The overall purpose of the study was to explore young adolescents’ use of, and knowledge
about, musical narrative functions and semiotic resources of music as a representational mode.
Three specific questions were formulated to direct the design of the interview study:

How do the participants of the study verbally describe the musical expression and
meaning-making functions of their versions of narrative media music?

What awareness and knowledge about musical narrative functions and conventions can
be discerned from their verbal statements?

How do the verbal and the musical statements of the participants relate to each other?

Methodology
As mentioned, the entire study consists of two main parts. In the first part, the participants
were asked to design and adapt musical expression to fit different visual scenes, using a
software tool. A detailed description of the design and results of this part has been presented
in a previous article (see Wingstedt, Brändström & Berg, 2008). In the second part, which is
the principal topic of this article, individual interviews were made with each of the
participants. Here they listened to the music they had created and made comments about the
musical results. Since the interviews are closely related to and dependent on the design and
results of the first part of the study, a brief description of this part will first be made in the
following sections. This will include the selection of the participants, a short description of the
test environment and a summary of the results of the first part. After that, the procedure and
results of the interview part will be presented.

Design and Environment of the Musical Sessions

With the purpose of recruiting participants for the study, six school classes (grades 6 and 7,
corresponding to ages 12-13) in the city of Piteå in the north of Sweden were contacted. The
students filled out a questionnaire, answering questions about their musical training and media

4
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

habits (amount of casual music listening, watching video and playing computer games). They
were also asked if they wanted to participate in the research project. Twenty-three students
volunteered to participate and were scheduled for individual test sessions. At the sessions,
they were introduced to the non-verbal research software REMUPP (Relations between
Musical Parameters and Perceived Properties; Wingstedt, Liljedahl, Lindberg & Berg, 2005).
The interface was designed to allow users to directly control musical expression, not requiring
any previous musical training. By the use of this software the participants were given the task
of adapting the expression of a musical performance to make the music ‘fit as well as
possible’ different 3D-animated visual scenes shown on a computer screen.

Three different visual scenes (here referred

to as movies) were presented, each one
depicting a physical setting or location.
Each movie was presented three times –
each time combined with a different basic
music example, making for a total of nine
trials comprising a session. The available
movies were:
FIGURE 1: Movie 1 – ’City’
1. Movie 1 (‘City’): City night –
unfriendly neighbourhood (Figure 1).
2. Movie 2 (‘Space’): In space – looking
at eternity (Figure 2).
3. Movie 3 (‘Lake’): Sunny day – picnic
by the lake (Figure 3).

Underneath the movies, seven faders, each

FIGURE 2: Movie 2 – ’Space’
controlling a separate musical parameter,
were made available to the participants for influencing and altering the musical expression.
Thereby the musical meaning could be changed. The controlling faders were presented
without any written labels to make the participants focus on the expressional and meaning-
making functions only by listening. In order from left to right the faders controlled the
following musical parameters:

5
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

1. Instrumentation (instrument set).

Three instrument sets were made
available: (a) ‘Rock’, (b)
‘Electronic’, (c) ‘Symphonic’.
2. Tempo (beats per minute).
3. Harmonic complexity (degree of
consonance-dissonance).
4. Rhythmic complexity (rhythmic
activity).
5. Register (pitch level by octaves).
FIGURE 3: Movie 3 – ‘Lake’, with faders controlling musical
6. Articulation (staccato-legato). parameters.
7. Reverb (reverberation level).

The software recorded every trial, including all the manipulations made by the participants.
The final result of each trial, including the settings of each parameter, was also made available
as numerical values. Thus, differences in musical expression could be described and analyzed
statistically. This data was combined with the results from the questionnaires. Together, this
material made up the quantitative data from which the statistical analysis of the first part of
the study was made.

Results of the Statistical Analysis

The results of the first part (Wingstedt et al., 2008) showed that the participants displayed a
strong degree of in-group conformity and consensus regarding the musical expressions
created for each of the movies. This suggests a shared view of musical narrative functions,
and these results will in the second part of the study be used as a measure for how
‘convention’ is defined by the group. The group’s expression of convention can in turn be
seen as to a certain degree mirroring existing, culturally available, narrative conventions that
we encounter in film, computer games and other multimedia. These results can be seen as
‘evidence of learning’ (Jewitt, 2006: 28), of knowledge about the design of available
conventions – and a willingness to follow these conventions. Examples of approaches where
the musical parameters were used to significantly distinguish the different movies from each
other, was for example the use of the ‘Symphonic’ instrumentation for the Lake movie (with
flute, strings and harp, reflecting a narrative convention for expressing pastoral settings in the
Western romantic music tradition) but a more electronic sound for the darker and more hostile

6
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

City movie and the high-tech and mysterious Space movie. For the City and Space movies, a
lower musical register was used matching the darker colours and suspenseful character of
these settings. The Lake movie was attributed with a higher rhythmical activity expressing the
movements of fluttering butterflies – and less reverb, naturalistically representing the open
outdoor setting. Overall, descriptive and emotive narrative functions were emphasized.

The results were also clearly correlated to factors such as the participants’ musical training,
and habits of music listening, playing computer games and watching movies. Participants
learning to play an instrument generally preferred more complex and expressive musical
structures compared to the ones with no or little musical training. Similar tendencies were
found for the participants doing much recreational music listening, compared to those who did
not listen as much to music. However, the musical results of those who did less recreational
music listening were generally closer to narrative conventions (as defined by the results of the
group). Participants playing comparatively more computer games and watching more movies
were also generally closer to expressing musical narrative conventions compared to those with
less media experience. Considering the potential learning opportunities offered by the
multimodal environment of new media (e.g. Jewitt, 2006), these last results could be
interpreted as indicating that greater experience of narrative media has resulted in more
knowledge about musical narrative functions. However, such claims about causality cannot
unambiguously be made based on the results of this study. Another interpretation could for
example be that people with a better understanding (and acceptance) of narrative functions
and conventions simply like to spend more time watching movies and playing games.

The outcomes of this first part of the study nonetheless raise questions concerning the impact
of the increasing availability of narrative media, how it relates to our habits, attitudes and
opinions as well as to our skills and knowledge. These questions point to the complexity of
musical learning and maybe also of learning in general.

Procedure of the Interview Study

Issues accentuated by the previous results concern matters of knowledge, awareness and
motivations for making choices, and aspects of conformity or non-conformity to conventions.
In order to be able to investigate these matters more closely, it was decided to interview each
participant after the finished musical sessions. The aim was to make a ‘multimodal’ test
design, not only by combining music with images – but also to combine and complement the

7
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

musical statements of the participants (and the results of the questionnaire) with verbal
statements where they could reflect and comment on their own musical expressions. All the
adjustments and manipulations made of the musical parameters were available for later
playback. It was therefore possible to look and listen to each participant’s trial series of nine
scenes while performing the interviews. This way the interviews could be designed as a
stimulated recall situation where the participants commented and reflected on their own
musical efforts.

The interviews were semi-structured and were made immediately after every musical test
session. Each of the 23 interviews took about 30 minutes and was recorded using a digital
audio recorder. For several reasons it was decided not to use video recordings of these
sessions. For the purpose of this project, a full multimodal content analysis including gestures
and facial expressions was not considered necessary. Also, being only two persons in each
interview situation (one test participant and one researcher), the use of a video camera was
estimated as being potentially distractive and might risk making the interviewee too self-
conscious. The audio recordings made then came to include the conversations as well as the
music that was played back as each trial was discussed. The transcription and analysis of the
recordings was thus based not only on the spoken language but also on the music being heard
and its relationship to the corresponding movies. As the movies did not include much
movement it was a simple task for the researcher to make sense of the relationships between
the music and the corresponding visuals, even if the computer screen was not video-filmed.

The interviews followed a similar pattern for each participant. First they answered some
introductory questions concerning the overall impression of the session and the software.
After that the attention turned towards their completed trials. One of the movies was chosen
and they were asked to describe their impression of it, how they thought the music contributed
to their impression and so on. The exact questions were adjusted according to what answers
were given by the participants. After the initial trial, the second musical version of the same
movie was played. Additional questions would be about whether the impression of the scene
now was changed, if there was a difference between the two versions. If so – in what way and
because of what? Then the third version of the same movie was played and similar questions
were asked. The interviewee was then asked to rate which of the three versions that was
preferred, and there were follow-up questions about the reasons for the choices made. A
similar procedure was then followed with the three versions of the next movie – and after that

8
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

the last movie was treated in a similar way. When the nine trials had all been discussed and
compared, the participants were asked to describe how they found each fader affecting the
musical sound. The interview was then rounded up with general questions about narrative
music in film and computer games.

Conditions for the Analysis

The spoken dialogue of the interviews, as well as descriptions of the music played back
during the interviews, were included in the written transcripts. To describe the music, the
actual parameter settings, retrieved from the software, were indicated – and also the subjective
impressions of the transcriber2 were noted. Computer software designed for qualitative text
analysis3 was used as a help to structure, code, categorize and query the data.

As described earlier, the quantitative analysis of the parameter settings indicated a strong in-
group consensus implying a general knowledge of, and conformity to, musical narrative
conventions. However, looking closer at the individual participants also revealed instances of
large individual variations within the group. As mentioned, the results of the statistical
analysis can be seen as a criterion for defining what was taken as the narrative norm for the
participants involved – a ‘measure of convention’ for this group. When closely examining the
parameter settings for each individual participant, and comparing them to the collective norm
for the whole group, two contrasting categories of participants could be established:
Conformers (conforming to the convention) and Non-conformers (deviating from the
convention). Looking at the most extreme representatives of each category, seven participants
could be assigned as being ‘extreme conformers’ while eight were assigned as ‘extreme non-
conformers’. These two categories, in all including 15 participants, were selected as being of
special interest when analysing the interview data. When referring to Conformers or Non-
conformers in the following text, it is these extreme cases that are indicated.

When looking at the selected participants, issues of particular interest would be how they
verbally motivated the musical choices made. What attitudes and levels of awareness could be
found in their descriptions and responses to both the questions posed and to the musical
expression heard? To what extent would their musical statements appear to be the result of
conscious efforts, and what other factors might be seen as influencing the choices made? How
would their verbal statements relate to their musical statements – and how would they
describe the expressional and meaning-making functions of narrative music in general?

9
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

Looking further at the results from the statistical analysis, as we have seen, two factors
appeared to be of special interest, relating to the musical choices made by the participants:
(1) Music experience and interest (musical training and music listening habits) and;
(2) Experience and interest of narrative media (game playing and film watching habits).
Taking these factors into consideration and using the results from the questionnaire, the
distribution and relationship of the involved factors can be shown graphically in a diagram
(Figure 4).

On the x-axis, musical training and listening habits are combined, showing the participants
with higher music experience and interest to the right and lower ratings to the left. The ones to
the right of the centre are all learning to play an instrument, while the ones to the left have no
or little musical training. On the y-axis, factors of game-playing and film-watching are
combined, with higher media experience and interest towards the top. The participants rated
as Conformers are represented by a square and Non-conformers by a circle, the remaining
participants are indicated with an ‘x’. The numbers given are the original id-numbers of the
total of 23 participants.

Looking at the distribution Media

+
of Conformers vs. Non-
conformers in the diagram, 13 10
4 1
x x 16
an interesting pattern 22
x 14 21 18 6 2
emerges. Of the six
Music
participants with lower - x +
19 x 11 x 15 17 x
5 20 3 7 23 12
music ratings, the ones
with higher media ratings
9 x
(3 participants) are all
- 8
assigned to the Conformer
FIGURE 4: Distribution of participants according to their experience of
category – while the music (x-axis) and media (y-axis). Conformers are represented by a
square, Non-conformers by a circle and the remaining participants with
remaining three (with an ‘x’. Id-numbers of the 23 participants are indicated.

lower media ratings) are assigned to the Non-conformer category. A possible interpretation of
this pattern could be that the ones with higher media experience are also more familiar with
narrative conventions – and are therefore better at expressing them. Such an interpretation
would seem to be in accord with the statistical results presented earlier. From this would
follow that the Non-conforming ratings of the three participants listed as ‘Low Media’ (as

10
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

well as ‘Low Music’) is a sign of their relative lack of media experience. Their Non-
conforming rating would thereby reflect their lesser knowledge of narrative conventions –
rather than, for example, being a result of their conscious creative choice. To make such an
interpretation at this stage would however be somewhat speculative and probably unduly
simplistic. Much of the earlier statistical results already suggest rather complex factors behind
the results gained so far. Also, the sample of ‘extreme’ participants used in Figure 4 would be
too small to be used as a basis for any such conclusive interpretations. Reflecting on the issue
however introduces various questions, such as about the interrelations of experience,
awareness, knowledge and conscious choice. Among the participants in the diagram with
higher music ratings there is no corresponding distinction between Conformers and Non-
conformers. No simple pattern is easily detected in the diagram.

Metafunctions of Communication
As a starting point for analysing the interviews, Halliday’s (1978) metafunctions of language
was used – these have later been extended into more general metafunctions of communication
by Kress and van Leeuwen (2001; 2006) and others. Simply put, the concept of the
communicational metafunctions are based on the notion that: (1) Each message is about
something; (2) It addresses someone; (3) It does so through meaning-making resources,
modes, each with its own culturally and materially determined constraints and affordances for
structural organization. Using Halliday’s terminology, these three basic functions are named
the ideational, the interpersonal and the textual metafunctions respectively.

The ideational metafunction is the content function of communication. Kress et al. (2001)
describe it as representing what goes on in the world; material, verbal, mental and observed
relational processes – ‘who does what, with or to whom and where’ (p. 13). The interpersonal
metafunction is the participatory function of communication, communication as doing
something. It is the component through which the communicator expresses her own attitudes
and judgments – and seeks to influence the attitudes and behaviour of others. It establishes,
maintains and specifies relationships between members of societies or groups through
expression of social relations, interrelations of power and knowledge. The textual
metafunction is the component which provides the texture (in relation to the environment), the
organizing of a text (in a broader sense) as a coherent message through textural resources of a
mode. The textual component has an enabling function with respect to the other two. It is only

11
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

in combination with textual meanings that ideational and interpersonal meanings are
actualized (Halliday, 1978: 113).

The metafunctions will here primarily be used to examine the means and strategies of the
participants in their descriptions of the expressed musical narrative functions. That is, how
they verbally relate and refer to the audio-visual scenes resulting from the trial sessions.

Results of the Interview Material

During the course of the interview sessions, it was apparent how the individual participants all
established distinct strategies, styles and profiles in how they discussed musical and narrative
features. They usually maintained their strategies quite consistently for the duration of each
interview. Due to the complexity of the material, some aspects of the results will be discussed
and reflected on directly in the context of their presentation.

In analyzing the comments made by the participants, five basic types of statements could be
discerned. As will be demonstrated, these can be associated to the different metafunctions of
communication and will be described as the Unclear, Intuitive, Associative, Analytical and
Transformative types. Put in this order, the types can be seen as representing a successively
increasing level of expressed conscious awareness about musical narrative functions. As we
will see, however, the relationship of these types will be treated as being more complex than
simply being labels implying ‘from less to more’. Usually, the verbal profile of a participant
could be attributed to a combination of two of these types. The analysis showed that when
several types dominated the utterances of one participant, they would typically be of adjacent
types (according to the order indicated above). That is, with very few exceptions, the
participants could be described as for example: intuitive-associative, associative-analytical,
analytical-associative etc. The type put first in a pair indicates it as the dominant of the two.

A common trait for all the participants was that they generally expressed a strong sense of
whether the music for a certain movie ‘worked’ or not. In this, they all showed a high level of
engagement and involvement. It was also apparent how they generally tended to see
themselves as agentive – as the creators of the music, even if the music making process and
environment of the test sessions might not correspond to what is traditionally seen as
‘composing music’ . Lars (participant no. 4, with no previous musical training), when

12
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

commenting on the entire session, proudly exclaimed: ‘Hmm... I didn’t know I was this good
at making music!’ Similar views were expressed by many of the participants. A typical
response, when judging the results of a certain trial scene, was also to indicate one’s own
creative agency as part of the result, such as: ‘Alright, I’m pretty satisfied with this one!’
(Karl, no. 1, listening to his second version of the Space movie), or Mia, (no. 23), responding
to her first version of the Lake movie:

Mia: No, this music was completely wrong! (laughs)

Q: Why is that?
Mia: I had intended it to be much nicer!

In these last two statements the interpersonal metafunction of the speaker in relation to the
musical expression is doubly emphasized. Firstly, the participant’s role as a creator of the
music is clearly established. Also, in making a distinct evaluative statement about the
aesthetical and narrative functions, the responding participant’s position in relation to the
experienced musical expression is clearly brought out and articulated. This kind of comments,
offering decisive evaluations about the appropriateness of the music – such as ‘this is better’,
‘I like it’ or ‘it doesn’t fit’ – offer no associative description or information about the
ideational content of the scene. They simply establish the interpersonal relationship of the
speaker towards what is represented in music and image. The comments can be seen as
examples of statements reflecting an intuitive kind of understanding and awareness of the
music. Comments like these, where there are no further explanations or descriptions of the
music, were designated as belonging to the Intuitive type. Intuitive awareness of this kind is
akin to what Swanwick (1994) describes as intuitive knowledge. According to Swanwick
(ibid.), intuitive knowledge is fundamental to all knowledge. Before developing conceptual or
analytical forms of knowledge, a basis of intuitive knowledge first has to be established.

In the interviews, statements attributed to the Unclear type were very few, but when used
would reflect the speaker’s hesitation or uncertainty about the impressions of the music
played. Here too, the interpersonal meaning is salient, indicating an ambivalent or unclear
relationship between the speaker and the musical (or audio-visual) expression. Typically, the
reasons for the uncertainty could not be described by the speaker, except maybe for it being
‘difficult’ to comment.

13
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

Some intuitive comments would, when the speaker was asked to elaborate, result in circular
reasoning such as with Henrik (no. 14) here listening to his preferred version of the Lake
movie:

Henrik: Mmm... that’s better.

Q: Why is that?
Henrik: It fits.
Q: What makes it fit?
Henrik: (laughs) It’s the music.
Q: In what way?
Henrik: Well... it’s just right...

This illustrates the difficulties involved in verbally trying to express musical qualities, which
is not unusual even among professional musicians. It was apparent that some of the
participants found it harder to verbally express the outcome of the music than they thought it
was to express the actual musical sound using the software. Henrik belongs to the Conforming
category (as described earlier), indicating a certain amount of practical knowledge and control
of musical narrative functions. Here he tries to explain his opinions about his first, not
preferred, version of the Lake movie:

Q: What do you think?

Henrik: Hmm... it works so-so...
Q: What’s wrong with it?
Henrik: It’s the music.
Q: In what way?
Henrik: It’s not quite...
Q: How do you want it to be?
Henrik: Better...

Generally, Henrik’s musical examples were very clearly expressed with distinct articulation
and well-balanced sound. He closely followed musical narrative conventions in most of his
trial examples. It is however obvious that he found it difficult to verbally comment on the
music. He consistently demonstrated a clear intuitive opinion about whether the music was
‘right’ or not – but he had a hard time going beyond that. There was a noticeable difference in
how he expressed musical knowledge ‘musically’ compared to how it came out verbally.

Looking next at the Associative type includes comments where ideational meaning of the
scenes is foregrounded. In such statements, associations to observed emotional content or
descriptions of narrative events are offered. In the interviews, associative comments would

14
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

often be short, such as: ‘This is dark and creepy’ (Nina, no. 6, about her preferred, fast and
dissonant electronic version of the City movie) ; ‘It sounds like summer’ (Karl, about a slow
and ‘airy’ version of the Lake movie, played by flute and strings); or ‘Something weird is
about to happen’ (Hans, no. 12, about a version of the Lake movie with a staccato melody in a
low register with plenty of reverb). Many of these comments would be preceded and
combined with intuitively evaluative statements as described earlier. The analysis of the
interviews showed that the associative comments turned out to constitute the quantitatively
largest type. There were more statements describing the musically represented ideational
‘content’ of the scenes than of any other type. This speaks against the traditionally held notion
saying that it is not possible to express ideational meaning through music (e.g. Hanslick,
1955). To the participants of the study it came natural to speak of music as communicating
situations, events and factual information. It was generally much harder for them to describe
the musical parameters involved, or other ‘intra-musical’ aspects. The emphasis on the
‘content’ of the music can to a large degree be attributed to the clear narrative and multimodal
context of the trial situation. Just as in film and computer games, this kind of context greatly
contributes to also making the musical meanings clear and unambiguous, the meaning arises
from the multimodal interplay of music combined with visuals and other narrative modes
involved (Wingstedt et al., in press).

The ideational descriptions could be more or less vivid, but clearly indicated a narrative
interpretation of the music heard in connection to the various movies. However, there would
not necessarily be any further suggestions available about the musical design behind the
associations made. In some cases the narrative descriptions were more detailed or even
extended into short dramatic episodes. An example is Jenny (no. 2) comparing two versions
of the Lake movie, the second one accompanied by dissonant music with a slow but steady
rhythmical pulse:

Jenny: The first one was, like, calm and an idyllic spot... but the other one was – when I hear
it I think more like – well, it’s kind of calm, but something scary is about to happen, a
monster comes up from the lake or so [...] at first there’s peace and quiet or whatever – and
then something big and disgusting will appear – like, from the lake or from the forest or
something like that...

Interestingly, Jenny chose this second version as her preferred version of the Lake movie. She
motivated her choice, opting out of the other more conventional versions, with: ‘the other
[versions] feel a little too un-exciting’. By using this combination of visuals and music, she

15
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

was able to creatively design her own narrative, not suggested by the visuals alone. It was
apparent that she was quite aware of the ‘language of convention’ – maybe to the point that
she found it a little too predictable. She preferred to extend the boundaries. Swanwick (1994)
includes associative characteristics, such as the ones described, as attributes of intuitive
knowledge. He refers to it as ‘the exercise of imagination, the creative forming of images’
(ibid., p. 29). However, he also argues that a certain kind of analytical work is involved to
make this intuitive meaning-making process possible. He calls this primary or intrinsic
analysis, ‘wordlessly implicit in all musical experience’ (p. 43).

Statements attributed to the Analytical and Transformative types will include comments
indicating how textual features of the music contribute meaning to the scenes.4 Analytical
statements include observations of musical structures or performance factors contributing
narrative meaning: ‘It fits well because there’s lots of echo and the notes stick together... and
it’s fast tempo’ (Joel, no. 18, describing a version of the Space movie). Viktor, (no. 17), starts
out with an associative comment about his preferred Space version and then continues with
analytical explanations:

Viktor: Yes, this is ‘space chase’ (laughs).

Q: What gives you that impression?
Viktor: It is faster... If you’d lower the speed it becomes... not as good. It has to be the right
speed for it to be good.
Q: I see.
Viktor: It mustn’t be too sluggish...

Viewing music as a mode of representation, this kind of comment emphasizes the intramodal
aspects of music – i.e. the inner structures of the musical sound.

Statements of the Transformative type use a different approach and typically suggest new
structural or narrative possibilities for the music. This can be done for example by implying
alternative uses, placements or combinations, such as: ‘This one would have worked well with
that Space too’ (Lars, describing an electronic version of the City movie), or ‘This one would
work better in a computer game than in a movie...in a movie it needs to be longer’ (Viktor,
suggesting alternative uses for his preferred City version and at the same time making an
analytical comment referring to the short repeating riff used). Sometimes more general
narrative or form-related functions might be specified: ‘This is almost like an introduction...
an intro to a film’ (Joel). A common factor of the Transformative type is how the comments

16
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

reposition the music to new situations or functions not immediately apparent in the listening
situation. This can be seen as emphasizing the textually intermodal aspects of the music – i.e.
the relationship of the music to other modes of expression.

Swanwick (1994) describes analytical knowledge as a kind of knowledge that chronologically

can follow after intuitive knowledge. According to him, musical analytical knowledge is
intellectually based knowledge concerned with matters such as relationships, conceptions,
tradition and underlying form. The type of analysis related to this kind of knowledge he calls
secondary or extrinsic analysis: ‘reflective discourse about particular music’ (ibid., p. 43).
Secondary analysis involves extra-musical ways of pointing to the intuitive knowledge that
results from primary analysis processes. Intuitive and analytical knowledge are therefore not
to be seen as two distinct types of knowledge (or ‘lesser’ or ‘better’ kinds of knowledge).
Rather, they are to be seen as interdependent and interwoven. ‘The relationship is not between
contrasting functions but between previous and subsequent phases in coming to knowledge’
(ibid., p. 29). However, we are always to some extent transformed by new knowledge (Kress
et al., 2001). Thus, as we go through processes of secondary analysis and gain new analytical
knowledge, our intuitive knowledge is simultaneously changed and re-defined. Swanwick
(1994) sees building knowledge as a continuous spiral with intuitive response and analytic
processes as mutually reinforcing energies.

The participants’ comments can be seen as evidence of learning, reflecting their knowledge of
narrative musical functions. Looking at this from a communicational perspective, verbally
expressing interpersonal, ideational or textual aspects of the music can, using Swanwick’s
(1994) concepts, be seen as expressions of either intuitive or analytical kinds of knowledge.
Statements of the Intuitive, and to some extent the Unclear, types can thus be seen as
reflecting intuitive knowledge (or lack of it) while statements of the Analytical and
Transformative types are primarily reflecting analytical knowledge. Associative comments
may be seen as essentially expressing intuitive knowledge, in the way that Swanwick
describes, but are also in a way bridging and connecting the intuitive and analytical domains.

Basically then, as most of the participants were fairly consistent in their types of comments,
we can look at them as showing either primarily intuitive or analytical knowledge profiles.
For example, participants with an ‘analytical-associative’ profile can be seen as expressing
mainly an analytical kind of knowledge while ‘intuitive-associative’ profiles rather seem to

17
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

express predominantly intuitive knowledge. Comparing this to their backgrounds concerning

musical training shows that there’s a tendency towards making analytical statements for the
participants who have learned to play an
instrument. Conversely, participants with
Table 1: Distribution of comment type profiles
no or little musical training show a related to musical training (N = 23)
Comment type profile
tendency towards using intuitive comments
Musical Intuitive Analytical
(see Table 1).5 Again, it should be pointed Training
Yes 1 12
out that this result illustrates a correlational No 6 4
relationship rather than a causal one.

Combining Results
The above results, however, only reflect how the participants’ knowledge is expressed
verbally. Kress et al. (2001) point out how different representational modes will represent
knowledge differently. As the examples with Jenny and Henrik (above) illustrate, the modes
of music and speech can reflect different aspects and strategies of knowledge and awareness.
Combining different modal expressions will give an opportunity to complement and nuance
the representations of musical knowledge. Jenny, in her verbal comments, often links
associative descriptions with analytical explanations, such as: ‘these higher notes make you
feel that something is about to happen’ (about her preferred City version). She is in this way
expressing a primarily analytical kind of knowledge. In her musical expressions, using the
software, she is often deviating from the conventions established by the group of participants.
The example presented earlier illustrates how she for example prefers a more dramatic version
of the Lake scene, rather than the soft and idyllic one conventionally used. Looking at both
her verbal and musical expressions then defines her statements as being both analytical and
unconventional. This implies that she is deviating from the norm by choice, that she is aware
of the meaning-making potential of specific features of the musical structure – and that she
uses this knowledge to realize her expressional ideas.

The verbal profile of Viktor is quite similar to Jenny’s. He usually combines associative
descriptions with analytical explanations of the musical structure. Musically however, he
sticks very close to the convention. He verbally demonstrates a similar kind of awareness as
Jenny, but he makes very different musical choices. His profile can be described as analytical
and conventional. But also in his case it seems to be a matter of making ‘informed’ choices.

18
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

Henrik is, just as Viktor, musically very close to the convention. The music made by these
two boys is quite alike and performs similar narrative functions. Both of them this way
demonstrate knowledge of musical narrative functions by how these are adhered to. But
verbally, Henrik’s comments are consistently intuitive-associative. He has a clear opinion of
whether the music works or not (see examples of quotes earlier), and he sometimes makes
short associative comments about the emotional content of the scenes, such as: ‘This one is a
little creepy’ or ‘hmm... exciting’. But he apparently has no strategies for verbally expressing
underlying structures or relations of the music. According to the questionnaire, Henrik spends
much time playing computer games, and watches movies several hours every day – but he has
no formal musical training. It seems likely that his demonstrated intuitive knowledge of
musical narrative functions to a certain degree is connected to his high level of media
exposure in informal situations. Considering the seemingly unreflected nature of his verbal
statements it can however be asked to what extent the musical result is an expression of
conscious choice – or whether it might be a matter of being ‘slave of convention’.

Jenny, Viktor and Henrik can be seen as representing three different positions: Analytical-
Unconventional, Analytical-Conventional and Intuitive-Conventional, respectively. A fourth
position is also available: Intuitive-Unconventional. In this study, two participants fit into that
position – Nadja (no. 11) and Niklas (no. 15). Coincidentally they both belong to the lower
left quadrant of Figure 4, having no musical training and comparatively low media experience.
In their comments, both of them are predominantly intuitive-associative. At times Niklas
offers relatively unusual readings of the scenes, which also seem to account for his
unconventional music. For example, he wants the music for the Lake scene to be ‘mystical’
because the scene is so ‘empty’. Nadja is at times somewhat unclear about what she wants.
She also seems a little unsure about the parameter controls. Maybe her unconventional music
is a result of her being indecisive – or simply confused by how to control the musical
parameters.

Looking this way at both the verbal and musical expressions of the participants, illustrates
how different aspects of knowledge and awareness will be made available and salient
depending on the mode of expression used. When combined, the expressions also co-
contextualize each other, together providing a more articulate and nuanced manifestation of
the participants’ expressed knowledge. This is analogous to how musical meaning becomes

19
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

both clearer and at the same time more elaborate when used in a multimodal context,
combined with e.g. visuals, dialogue and sound effects (Wingstedt et al., in press).

Conclusion
The study has been designed in an effort to address and reflect some of the complexity of
musical knowledge. There is however an inherent contradiction and paradox in formally
attempting to study processes of learning that (possibly) occurs in informal situations. The
actual learning process is difficult to access. What is available for study is rather what has
been referred to as ‘evidence of learning’ (Jewitt, 2006). Expressions of knowledge are
however always partial. Also, as the study illustrates, different modes of expression will
represent different aspects of knowledge. As in all interpretative work, the results will always
be partial. It should also be mentioned that a certain caution should be taken when it comes to
generalizing the results, due to the non-randomized allocation of the volunteering participants.
It is for example possible that those who chose to participate in the study have a
comparatively high interest in narrative media and therefore also have a relatively high degree
of familiarity with available semiotic resources and narrative conventions. This study does not
however venture to be exhaustive or widely generalizing, but rather aim to study features of
musical learning related to individual learners – and to take that as a starting point for raising
questions concerning learning opportunities in informal situations.

The non-verbal REMUPP tool, allowing the participants to express themselves through
‘musical action’, made visible representations of knowledge that would otherwise not be
available for study. It also provided a game-like situation that served to heighten the
‘experimental realism’. Being able to use the same tool as a focal point during the stimulated
recall interviews, rather than watching video recordings of the participants, further seemed to
lessen their concern about being tested. This contributed to establishing a relaxed atmosphere
and maintaining a focus on the music rather than on the participants themselves.

It seems likely that listening to music in informal situations involves opportunities for
learning, even if the listening process is unfocused and unstructured and taking place on a
subconscious and unreflected level. Given the design and learning opportunities offered by
narrative media, its escalating abundance and the amount of time spent using it on a daily
basis in contemporary society, it can be assumed that some learning about its functions is

20
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

achieved by this use. Through watching television and movies and playing computer games,
knowledge can be gained about available meaning potentials of music. We learn certain ways
to understand and use music and to make meaning from the ways music is structured and
combined with other narrative modes. For the participants of the study, using the REMUPP
software to express narrative musical meaning was a new experience. However, with a
minimum of time and effort distinct results could be achieved. When asked about their
impression of the task, the participants generally answered that they did not find it difficult or
strange. The hardest thing was to get the music exactly the way they wanted to, since they
usually had quite a clear idea of how they wanted it to be. It appeared to come naturally for
them to treat music as a narrative mode, expressing ideational meanings just as with language
or image. In this sense, they could be described as being musically literate.

The expanding uses of music in new digital media entails expanded opportunities for
musicking. Folkestad (2006) discusses Ziehe’s (1986) concepts of common and uncommon
learning practices. Folkestad remarks that with the massive amount and range of music being
available via new media today, the musical learning taking place informally outside of school
is now experienced as the ‘common’ for today’s children while learning in school appears as
the ‘uncommon’ learning practices. If this is the case, then questions arise about how formal
education does, can and should relate to the evolving changes. The results of the study suggest
that the participants with formal musical training express their knowledge more analytically
than the ones with no formal training. Is this then an indication that a suitable role of music
education is to be the provider of tools for a more analytically oriented knowledge? From the
point of view of the discussion made so far, this would be a simplified way of looking at it.
An apparent question is whether the learning of analytical knowledge is a worthwhile goal in
itself. Swanwick (1994) warns about the danger of coming to imagine that analytical
knowledge is what music is all about: ‘Surely Beethoven did not write the first movement of
the Eroica Symphony to illustrate the use of “sonata form”’ (p. 33). As argued earlier, it can
be questioned to what extent analytical knowledge without intuitive knowledge is to be seen
as musically meaningful. Conversely, it can also be asked if intuitive knowledge without
analytical knowledge is enough. Maybe the role of music education is rather to provide
opportunity and space for the different kinds of knowledge to develop and mature in dynamic
and mutual interplay. And not only supply new knowledge but also to attend to the knowledge
that students bring from the various informal learning situations they encounter every day.
Such an environment would provide opportunities for intuitive and analytical knowledge to

21
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

not only develop and grow – but to combine and interact, forming a basis for constructive and
creative strategies for music practices involving performance as well as listening.

We need, both as educators, researchers and as users of music, to relate to the impact of the
escalating information society. In order to find strategies for this, we need more knowledge.
This involves an expanded view on what it means to ‘learn music’, to be musically literate.
We need to understand not only musical structure, history or catalogues of works. It is also
important to increasingly explore and understand various musical contextual functions and
purposes. In the study we could see how certain participants chose to follow conventions
while others chose to break them. For others it rather seemed to be a matter of being ‘slaves to
convention’ – or sometimes not being aware of the available choices. The prospect for music
education and research is then perhaps not primarily to commend or denounce particular
available cultural conventions. Rather it should maybe to a higher extent be about providing
knowledge, tools and thereby agency and freedom to make choices – which is the heart of all
issues related to expression and creativity.

Acknowledgements
The authors wish to thank Mats Liljedahl for programming the REMUPP interface, Stefan
Lindberg for composing the basic music examples and Jacob Svensson for creating the 3D
animations. Many thanks also to the participants of the study. For the sake of maintaining
anonymity, the names of the participants have been changed.

References
Bratt-Rawden, K. & DeNora, T. (2005). ’Music and informal learning in everyday life’, in
Music Education Research, Vol. 7, No. 3, pp. 289-304.

Croce, B. (1992). The Aesthetic as the Science of Expression and of the Linguistic in General
(C. Lyas, Trans.). Cambridge: Cambridge University Press. (Original work published
1902).

Ericsson, C. (2002) Från guidad visning till shopping och förströdd tillägnelse:
Moderniserade villkor för ungdomars musikaliska lärande [From guided exhibition to
shopping and preoccupied assimilation: Modernised conditions for adolescents’ musical
learning] (Doctoral dissertation). Malmö, Sweden: Malmö Academy of Music: Studies in
Music and Music Education no 4.

22
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

Folkestad, G. (2006). ‘Formal and informal learning situations or practices vs formal and
informal ways of learning’. British Journal of Music Education, 23:2, pp. 135-145.

Halliday, M.A.K. (1978). Language as Social Semiotic. London: Edward Arnold.

Hanslick, Eduard. Om det sköna i musiken [On the Musically Beautiful]. Uppsala: Schismens
skriftserie 1, 1955. (Original work, Vom Musikalisch-Schönen, 1854).

Hodge, R. & Kress, G. (1988). Social Semiotics. Cambridge, UK: Polity Press.

Jewitt, C. (2006). Technology, Literacy and Learning: A Multimodal Approach. Oxon:

Routledge.

Jewitt, C. & Kress, G. (Eds.). (2003). Multimodal Literacy. New York: Peter Lang Inc.

Kress, G., Jewitt, C., Ogborn, J. & Tsatsarelis, C. (2001). Multimodal Teaching and Learning:
The Rhetorics of the Science Classroom. London: Continuum.

Kress, G. & Van Leeuwen, T. (2001). Multimodal Discourse: The Modes and Media of
Contemporary Communication. London: Arnold.

Kress, G. & Van Leeuwen, T. (2006). Reading Images: The Grammar of Visual Design, (2nd
edition). London: Routledge.

Lindstrand, F. (2006). Att göra skillnad: Representation, identitet och lärande i ungdomars
arbete och berättande med film [Making Difference: Representation, Identity and Learning
in Teenagers’ Work and Communication with Film] (Doctoral dissertation). Stockholm:
HLS Förlag.

Small, C. (1998). Musicking: The Meanings of Performing and Listening. Middletown:

Wesleyan University Press.

Swanwick, K. (1994). Musical Knowledge: Intuition, Analysis and Music Education. London:
Routledge.

Van Leeuwen, T. (1999). Speech, Sound and Music. London: Macmillan.

Van Leeuwen, T. (2005). Introducing Social Semiotics. London: Routledge.

Wingstedt, J., Brändström, S. & Berg, J. (2008). ‘Young Adolescents’ Usage of Narrative
Functions of Media Music by Manipulation of Musical Expression’. Psychology of Music,
Vol. 36, No. 2, pp. 193-214.

Wingstedt, J., Brändström, S. & Berg, J. (in press). ‘Narrative Music, Visuals and Meaning in
Film’. Visual Communication.

Wingstedt, J., Liljedahl, M., Lindberg, S. & Berg, J. (2005). ‘REMUPP – An Interactive Tool
for Investigating Musical Properties and Relations’. In S. Fels & T. Blaine (Eds.), The
International Conference on New Interfaces for Musical Expression, NIME-05 (pp. 232-
235). Vancouver: University of British Columbia.

23
 Wingstedt et al.: Making Meaning of Media Music: Expressions of Knowledge about Musical Narrative Functions

Ziehe, T. (1986). Ny Ungdom: Om Ovanliga Läroprocesser [New Youth: On Uncommon

Learning Processes]. Stockholm: Norstedts.

Notes
1
According to van Leeuwen (2005), semiotic resources are defined as ‘the actions, materials and
artefacts we use for communicative purposes’ (p. 285). This term is used to emphasize that meaning-
making modes of expression (such as music) are not to be seen as being based on fixed pre-existing
sets of rules to be learned (codes) – but as making available resources for meaning that has a meaning
potential (based on past uses) and a set of affordances (based on possible uses) which will be
actualized in concrete social contexts. The perceived meaning of a certain musical performance will be
determined by the complex interactions of materially and socially determined constraints and
affordances – at work in structural, multimodal, narrative, personal, situational, social, historical and
cultural contexts.
2
The transcriber/researcher has a degree in composition.
3
NVivo 7, trademark of QSR International.
4
It should be noted that on the level of language, also in the Analytical and Transformative types of
statements it is the use of ideational meaning that is examined, just as in the Associative type. In this
case however, it is textual aspects of the music that are described ideationally through language.
5
The statistical significance of the distribution of comment type related to musical training was
examined using Fisher’s Exact Test, due to the small sample size (N = 23). The distribution as shown
in Table 1 was found to be significant (p < .05).

24