“Live Visuals is a timely and compelling account of the relationship between

sound and image. It charts a historical course that is long overdue. The book
brings together an impressive array of voices that in combination mix historical
context, theoretical analysis, and ref lections on contemporary practice, to great
effect. It is an invaluable resource for audio-visual students and scholars and
makes a very significant contribution to intellectual debate in this field.”
Professor Stephen Kennedy, Professor of Critical Theory
and Practice, Greenwich University

“Live Visuals presents a timely historical and conceptual overview of the art and
design of live media. Featuring the work of the early pioneers to some of today’s
leading designers of spatial media, Live Visuals offers a framework for creative
practitioners and students of the art of immersive visual experiences.”
Damien Smith, Creative Partner, ISO, http://isodesign.co.uk/
LIVE VISUALS

This volume surveys the key histories, theories and practice of artists, musicians, filmmakers,
designers, architects and technologists that have worked and continue to work with
visual material in real time.
Covering a wide historical period from Pythagoras’s mathematics of music and colour
in ancient Greece, to Castel’s ocular harpsichord in the 18th century, to the visual music
of the mid-20th century, to the liquid light shows of the 1960s and finally to the virtual
reality and projection mapping of the present moment, Live Visuals is both an overarching
history of real-time visuals and audio-visual art and a crucial source for understanding
the various theories about audio-visual synchronization. With the inclusion of an
overview of various forms of contemporary practice in Live Visuals culture – from VJing
to immersive environments, architecture to design – Live Visuals also presents the key
ideas of practitioners who work with the visual in a live context.
This book will appeal to a wide range of scholars, students, artists, designers and
enthusiasts. It will particularly interest VJs, DJs, electronic musicians, filmmakers,
interaction designers and technologists.

Steve Gibson is an interactive media artist and audio-visual performer. He has presented
at many world-leading venues, including Ars Electronica, Banff Centre for the Arts,
the European Media Arts Festival and Cabaret Voltaire. He is an Associate Professor in
Innovative Digital Media at Northumbria University, Newcastle, UK. www.telebody.ws

Stefan Arisona is a computer scientist and artist with interests in computer graphics,
extended reality, urban planning and digital art. He is a member of the Scheinwerfer
VJ collective and leads XR software development at the Esri R&D Center Zurich,
Switzerland. https://robotized.arisona.ch

Donna Leishman is a media artist, designer and researcher. Recent works include an
AR project To Have & To Hold and Front, a modern cautionary tale about social media.
She is an Associate Professor in Communication Design at Northumbria University.
http://6amhoover.com

Atau Tanaka conducts research in music human–computer interaction (HCI), focusing

on embodied musical interaction. By using muscle sensing in performance, the human
body becomes a musical instrument. He carries out his work at Goldsmiths, the Bristol
Interaction Group and the Maison des Sciences de l’Homme Paris Nord. www.ataut.net
Routledge Advances in Theatre & Performance Studies

This series is our home for cutting-edge, upper-level scholarly studies and edited
collections. Considering theatre and performance alongside topics such as reli-
gion, politics, gender, race, ecology, and the avant-garde, titles are character-
ized by dynamic interventions into established subjects and innovative studies
on emerging topics.

Deburau
Pierrot, Mime, and Culture
Edward Nye
Performance at the Urban Periphery
Insights from South India
Cathy Turner, Sharada Srinivasan, Jerri Daboo, Anindya Sinha
Australian Metatheatre on Page and Stage
An Exploration of Metatheatrical Techniques
Rebecca Clode
Functions of Medieval English Stage Directions
Analysis and Catalogue
Philip Butterworth
Live Visuals
History, Theory, Practice
Edited by Steve Gibson, Stefan Arisona, Donna Leishman and Atau Tanaka
The Celestial Dancers
Manipuri Dance on Australian Stage
Amit Sarwal

For more information about this series, please visit: www.routledge.com/

Routledge-Advances-in-Theatre – Performance-Studies/book-series/RATPS
LIVE VISUALS
History, Theory, Practice

Edited by Steve Gibson, Stefan Arisona,

Donna Leishman and Atau Tanaka
Cover image: Richie Hawtin, aka Plastikman live at Time Warp in
Mannheim, Germany 2010. Photo by Stefan Solf, © Richie Hawtin and
Stefan Solf. Used by permission.

First published 2023

by Routledge
4 Park Square, Milton Park, Abingdon, Oxon OX14 4RN
and by Routledge
605 Third Avenue, New York, NY 10158
Routledge is an imprint of the Taylor & Francis Group, an informa business
© 2023 selection and editorial matter, Steve Gibson, Stefan Arisona,
Donna Leishman, Atau Tanaka; individual chapters, the contributors
The right of Steve Gibson, Stefan Arisona, Donna Leishman, Atau
Tanaka to be identified as the authors of the editorial material, and of
the authors for their individual chapters, has been asserted in accordance
with sections 77 and 78 of the Copyright, Designs and Patents Act 1988.
All rights reserved. No part of this book may be reprinted or reproduced
or utilised in any form or by any electronic, mechanical, or other
means, now known or hereafter invented, including photocopying and
recording, or in any information storage or retrieval system, without
permission in writing from the publishers.
Trademark notice: Product or corporate names may be trademarks
or registered trademarks, and are used only for identification and
explanation without intent to infringe.
British Library Cataloguing-in-Publication Data
A catalogue record for this book is available from the British Library
Library of Congress Cataloging-in-Publication Data
A catalog record for this book has been requested

ISBN: 978-1-032-25261-2 (hbk)

ISBN: 978-1-032-25268-1 (pbk)
ISBN: 978-1-003-28239-6 (ebk)
DOI: 10.4324/9781003282396

Typeset in Bembo
by Apex CoVantage, LLC
CONTENTS

List of Contributors x
Acknowledgements xii

Introduction: The Long History of Moving Images

Becoming Alive 1
Steve Gibson

PART I
The History of Live Visuals 7

1 Inventing Instruments: Colour-Tone Correspondence

to Colour-Music Performance (Pre-1900) 9
Maura McDonnell

2 Moving Towards the Performed Image (Colour Organs,

Synesthesia and Visual Music): Early Modernism (1900–1955) 41
Steve Gibson

3 Liquid Visuals: Late Modernism and Analogue Live Visuals

(1950–1985) 62
Steve Gibson

4 Scratch Video and Rave: The Rise of the Live Visuals

Performer (1985–2000) 89
Léon McCarthy and Steve Gibson
viii Contents

5 The Post-Conceptual Digital Era (2000–Present) 109

Paul Goodfellow and Steve Gibson

PART II
The Theory of Live Visuals 133

6 Cross-Modal Theories of Sound and Image 135

Joseph Hyde

7 Live Visuals in Theory and Art 164

Paul Goodfellow

8 Live Visuals: Technology and Aesthetics 194

Léon McCarthy

9 AVUIs: Audio-Visual User Interfaces: Working With Users

to Create Performance Technologies 208
Nuno N. Correia and Atau Tanaka

10 A Parametric Model for Audio-Visual Instrument Design,

Composition and Performance 240
Adriana Sá and Atau Tanaka

11 Presence and Live Visual Performance 267

Donna Leishman

PART III
The Practice of Live Visuals 287

12 VJing, Live Audio-Visuals and Live Cinema 289

Steve Gibson and Stefan Arisona

13 Immersive Environments and Live Visuals 308

Steve Gibson

14 Architectural Projections: Changing the Perception

of Architecture With Light 335
Simon Schubiger, Stefan Arisona, Lukas Treyer
and Gerhard Schmitt

15 Design and Live Visuals 352

Donna Leishman
 Contents ix

PART IV
Interviews With Key Practitioners 375

16 Interview 1: Tony Hill, Expanded Cinema Pioneer 377

Steve Gibson

17 Interview 2: Christopher Thomas Allen, Founder

and Director, The Light Surgeons 392
Steve Gibson

18 Interview 3: Greg Hermanovic, CEO, Derivative 407

Steve Gibson

19 Interview 4: Markus Heckmann, Technical Director,

Derivative; Programmer for Carsten Nicolai and Others 422
Steve Gibson

20 Interview 5: Peter Mettler, Digital and Live Cinema Artist 437

Steve Gibson

Afterword 450
Steve Gibson

Index 453
CONTRIBUTORS

Christopher Thomas Allen is the founder and director of The Light Surgeons,
London. The Light Surgeons are renowned for their experimental films, instal-
lations and live cinema performances. They work across the disciplines of audio-
visual production on projects that blur the boundaries between research, film,
music, art and live performance.

Nuno N. Correia is an Associate Professor in Digital Transformation at Tallinn

University, Estonia. His work has been presented in such venues as ACM Mul-
timedia – Interactive Arts, FILE, Optronica/British Film Institute, PixelAche/
Kiasma and SXSW.

Paul Goodfellow is an artist with an interest in the application of systems and

processes in art. He has presented his work at venues such as ISEA Istanbul,
Transart Film Festival and Museum of Contemporary Art Taipei. He is a Senior
Lecturer in Digital Art Theory and Practice at Abertay University.

Markus Heckmann is a technical director with Derivative, the makers of Touch-

Designer, in Toronto, Canada. Apart from producing show visuals and installa-
tions, he heavily enjoys providing visual stimulus to the techno crowd under his
moniker Wüstenarchitekten.

Greg Hermanovic is co-founder of Derivative and Side Effects Software. Greg

has received two Scientific and Engineering Academy Awards for the advance-
ment of procedural visual effects tools in the film industry. TouchDesigner has
been used in projects for Walt Disney, Google, MIT Media Lab, Michael Snow
and Rush.
 Contributors xi

Tony Hill is a pioneer of expanded cinema and has presented his work at many
art galleries and in film festivals worldwide. He has been working as an artist
filmmaker since 1973, usually taking on all aspects of production, including
developing and building his own equipment.

Joseph Hyde has directed the Seeing Sound Symposium since 2009. His sound
and audio-visual works have been performed worldwide. He often works with
collaborators – scientists, engineers, artists and dancers/choreographers.

Léon McCarthy is a software developer, multimedia designer and audio-visual

performer in Dublin, Ireland. He is currently lead developer at FenestraPro, was
formerly a Lecturer in Digital Video at University of Limerick and co-founded
the Irish video production company MercuryBoy.

Maura McDonnell is a multimedia artist and Assistant Professor in Music and

Media Technologies at Trinity College, Dublin. Her academic research focuses
on the field of visual music. Her arts practice explores the visual art space of
abstract video using musical expression and has been presented worldwide.

Peter Mettler is a Swiss-Canadian film director, cinematographer and live visu-

alist. He is best known for his distinctive approach to documentary, as shown in
such films as The End of Time. He has also worked extensively as a live visualist
collaborating with artists such as Fred Frith, Jim O’Rourke and Biosphere.

Adriana Sá is transdisciplinary artist, performer and musician/composer. Design-

ing and building the instrumentation is part of her creative process. Her research
bridges creative practice and perception science. She currently lectures at Lusó-
fona University, Lisbon, Portugal.

Gerhard Schmitt is Professor Emeritus of Information Architecture at ETH

Zurich, former senior vice-president of ETH Zurich and former director of the
Singapore-ETH Centre for Global Environmental Sustainability (SEC).

Simon Schubiger works as a senior principal software engineer on 3D technol-

ogy at Esri, Zurich. He is a co-developer of the procedural 3D modelling soft-
ware CityEngine, Soundium2 multimedia platform and the NOVA software. He
is also a member of the VJ collective Scheinwerfer.

Lukas Treyer holds a master’s degree in architecture from ETH Zurich, Switzer-
land. His main interests are in improving the usability of tools in urban planning
and architectural design.
ACKNOWLEDGEMENTS

Special thanks to:

The Arts and Humanities Research Council (AHRC) for funding the original
Real-Time Visuals Network: https://gtr.ukri.org/projects?ref=AH%2FJ0110
37%2F1
Lily Drabble, image editor and copyright manager
Northumbria Opportunities for funding that role
The School of Design at Northumbria University for awarding a sabbatical to
Steve Gibson for this book
Tom Kuo for arranging the interviews in Toronto
Our editors at Routledge:
Laura Hussey, editor, Theatre and Performance
Swati Hindwan, senior editorial assistant, Theatre and Performance

For allowing us to use their images as figures, the authors would like to thank
these organisations and/or artists:

Julian Abrams
Cory Arcangel
ARTSITE IN for Ronald Nameth
George Barber
Christopher Bauder
Will Bauer
Valerio Berdini for Amon Tobin
Jean-Paul Berthoin for Coldcut
Matt Black for Coldcut
BlackMagic for the Fairlight CVI
Nate Boyce for Oneohtrix
 Acknowledgements xiii

Luc Courchesne
Char Davies
Draxtor Despres and Jo Yardley
Martina Eberle
First Sounds Project
Garagecube for Modul8
Matthias Gubler
The Collection of Historical Scientific Instruments, Harvard University
Caroline Hayeur
Markus Heckman
Isabela Herig for KAWS studio
Greg Hermanovic
Tony Hill
Martin Howse
Ryioji Ikeda
The Joshua Light Show
Joerg Jewanski
The Klip Collective
Tom Kuo
Ruedi Kuchen for Scheinwerfer
Kunsthalle Bremen for the Paik-Abe Synthesizer
Alan Kwan
Rik Lander and Peter Boyd Maclean for the Duvet Brothers
The Light Surgeons
Rafael Lozano-Hemmer
Thor Magnusson
Bryony McIntyre
Alex McLean/Shelly Knotts
Peter Mettler
Rob Mullender
National Film Board of Canada for Norman McLaren
Paul Nicholson for Aphex Twin
Colin Nightingale & Stephen Dobbie (A Right/Left Project) and James
Lavelle (UNKLE)
Jérôme Noetinger
The Norton Simon Museum for Paul Klee
alva noto/Carsten Nicolai
NoTV
Novak Collective
Robin Parmer
Robert Pepperell for Hex
Ivan Picelj
Gabriela Prochazka for Konx-Om-Pax
Jason Rhoades
xiv Acknowledgements

Don Ritter
Babycakes Romero for Massive Attack
Jeffrey Shaw
Andrey Smirnov
Damien Smith for ISO
Stefan Solf for Plastikman/Richie Hawtin
STEIM
Superf lux
Ivan Sutherland
Universal Everything
Susana Valadas
Stuart Warren-Hill for Hexstatic
James and John Whitney
Richard Winchell
INTRODUCTION
The Long History of Moving Images
Becoming Alive

Steve Gibson

Introduction to Live Visuals: History, Theory, Practice

The book is a collective endeavour to document the primary material related to
the concept of the performed, live image as well as the cultural history, theory
and practice of Live Visuals and audio-visuals. There is no one single source for
the history of Live Visuals, and at present researchers and practitioners have to
consult a patchwork of written and online sources (i.e. from film and art history
and theory, music theory, electronic music, performance studies, etc.). We tend
to assume Live Visuals are a thing of the present as they are now omnipresent
in clubs, popular and electronic music concerts, public venues, parks and on
buildings; however, as this book makes clear, the idea of the performative image
has a deep history, often entwined with ideas focused on the material similari-
ties between sound and image or light, as discussed immediately following this
introduction by Maura McDonnell in Chapter 1.
The primary goal of this book is to bring together key precepts in order to
establish a clearer understanding of the historical context and theoretical frame-
works for Live Visuals. This endeavour also outlines the most significant histori-
cal inventions, artists and movements that have contributed to the development
of Live Visuals. Following from early prototypes such as Louis-Bertrand Castel’s
ocular harpsichord from the 18th century; to 19th-century inventions such as
the light/colour organ; through visual-music film in the 1920s, 1930s and 1940s;
to liquid light shows in the 1960s and 1970s; to video synthesizers and scratch
video in the 1970s, 1980s and 1990s; and finally to the present-day experiments
in tactile audio-visuals and virtual reality, this book collects this diverse body
of technological experimentation into one published volume. The artistic forms
that these technologies have engendered – colour music, visual music film, liq-
uid light shows, live and expanded cinema, audio-visual performance, VJing,
DOI: 10.4324/9781003282396-1
2 Steve Gibson

immersive art, projection mapping – are also covered in detail, along with the
key artists associated with these movements.
It should be stated at the outset though: this book is not a detailed history
of the individual movements (such as visual music, expanded cinema or VJing)
addressed within its borders. There are excellent books that cover those histories
in great detail, such as Kerry Brougher, Jeremy Strick, Ari Wiseman and Judith
Zilczer’s Visual Music: Synaesthesia in Art and Music Since 1900; Adriano Abbado’s
Visual Music Masters; Gene Youngblood’s Expanded Cinema; and D-FUSE’s VJ:
Audio-Visual Art + VJ Culture; amongst many others. While understandably not
totally comprehensive, Live Visuals: History, Theory, Practice presents a broad and
diverse overview of the key figures, technologies and artistic developments in
the long history of images becoming alive.
In addition to the lack of a coherent documented history in one locale, there
has been a dearth of comprehensive theoretical analysis of Live Visuals in one
easily accessible source. This is due to a number of factors, but most obviously
the diverse forms of Live Visuals production have made the task of document-
ing consistent theoretical discussions of Live Visuals a complex one. These are
often pursued in (somewhat) fragmented disciplines such as music technology,
digital media, film and media theory, computing and design and in conferences/
festivals such as New Interfaces for Musical Expression (NIME), the Interna-
tional Symposium on Electronic Arts (ISEA), Ars Electronica and many others.
Theories surrounding structure, narrative, aesthetics and form in Live Visuals
are key issues that are addressed in this book, particularly by Joseph Hyde in
Chapter 6 and Léon McCarthy in Chapter 8. The book also brings together
chapters from key practitioners working in diverse areas of Live Visuals produc-
tion, from audio-visuals, to VJing, to live cinema, to architecture, to design.
We also include interviews with key Live Visuals practitioners and researchers,
including figures as diverse as expanded cinema pioneer Tony Hill, Light Sur-
geons Director Chris Allen and the founder of Derivative, Inc. (makers of the VJ
software TouchDesigner) Greg Hermanovic.

The Long History of Moving Images Becoming Alive

This book provides an expansive and interconnected situational framework for
Live Visuals culture by retracing the history of images that created worlds of illu-
sions via pictorial, audio-visual, performative and architectural elements. This
long history is one that involves multiple access points through disciplines as
diverse as philosophy, mathematics, acoustics, visual arts, instrument making,
film theory and electronic music. Each one of these fields has contributed in
some way to the evolution of the performed image. As Part I of this book makes
abundantly clear, Live Visuals are used in an astonishing array of past and pres-
ent contexts and have deep roots in past activities that are centred on the gen-
eral interaction between sound and image, as well as the immersive use of the
pictorial.
 Introduction 3

The role of Live Visuals is not merely that of entertainment but also of
deep social participation which, based upon an immersive physical experience,
emphasises a sense of cultural belonging, common identity, technological won-
der and sensory immersion. The earlier history of audio-visual culture, at least in
Western society, has commonly been centred around what is commonly referred
to as ‘art’ or ‘high’ culture (i.e. fine arts, ‘classical’ music), as well as scientific
and quasi-scientific notions that informed the development of an immersive and
interactive approach to the image (see Chapter 1 for a detailed discussion of
this). As the 20th century dawned and unfolded, various forms and movements
as diverse as the visual music film, Fluxus and pop art broke down the barriers
between art culture and popular culture, so much so that today the bound-
ary between the two now is uncertain and f luid. Live audio-visual culture has
undoubtedly played a continuing role in this uninhibited mingling, and this is
documented extensively in this volume.
As discussed by the author in Chapter 13, the long history of the immersive
image can be traced back (at the least) to theories such as Leon Battista Alberti’s
theory of vanishing point perspective in the 15th century, as well as the resulting
inf luence of this theory on the development of immersive spaces by painters such
as Masaccio, who employed Alberti’s vanishing point theory to create detailed
and immersive frescos in churches and chapels. Similarly, as discussed by Simon
Schubiger, Stefan Arisona, Lukas Treyer and Gerhard Schmitt in Chapter 14,
devices such as the camera obscura in the 15th century and the magic lantern in the
16th century point to a desire to create magical illusions that immerse viewers
in a real-time experience of fantastical visual environments. These experiments
were continued in the 18th and 19th centuries, including the various colour and
light organs discussed in Chapters 1 and 2 by McDonnell and Gibson, as well the
other multivarious devices discussed in Chapters 3, 4 and 5 by Gibson, McCar-
thy and Paul Goodfellow. Specialist audio-visual performance technologies were
employed throughout the 20th century (and into the 21st), many of which con-
sisted of repurposed technologies that were used in one context but transported
to another (i.e. overhead projectors that were originally used in the classroom
were repurposed by live visual artists in the 1960s to mix coloured oils live at
music concerts – see Chapter 3 for more on this).
The long history of images becoming alive is far deeper, more varied and far
lengthier than even this author realised when beginning to research for this under-
taking. Certainly, there can be no doubt that there is a rich history of the per-
formed image, and this book makes clear that there is much more to Live Visuals
than just club VJing and projection mapping, even though those are arguably the
two most visible examples of the performed image in general public consciousness.

The Experience and Shape of Live Visuals

The experience of Live Visuals, as well as the human participation in the live
visual event itself, requires an analysis that moves beyond the formulaic notion of
4 Steve Gibson

mere entertainment and reintroduces an understanding of the social experience

of visual, audio-visual and immersive media as a space in which the visuals are
not just live because they move, seem real and happen experientially. As Donna
Leishman makes clear in Chapter 11, the viewer’s experience of Live Visuals has
deep implications for understanding the nature of the ‘presence’ of the audience
in both audio-visual performance and more singular experiences such as virtual
reality. The audience’s presence can be one of being actively hedonistic, feeling
present in the spectacle itself and/or deeply immersed in the audio-visual envi-
ronment (amongst other involvements). Live Visuals and participants’ engage-
ment should be conceived as a moment in which – if we can reinterpret the image
and its experience beyond Oliver Grau’s definition of “enclosure of the observer
within the image space”1 – all the dissected components unify in order to create
a symbiotic encounter in which both the visuals and the participants are alive.
In other parts of this book (such as the discussion of synesthesia in Chapter 2
or in the interview with filmmaker and live visualist Peter Mettler in Chap-
ter 20), the occasionally ecstatic audience reception of multisensory live events
also leads to quasi-spiritual discussions of how artists have employed techniques
such as the simulation of synesthesia (synesthesia can be defined as “a neurologi-
cal phenomenon that occurs when a stimulus in one sense modality immediately
evokes a sensation in another sense modality”2) to create a heightened sense
of sensory awareness by reinforcing connection between mediums, or even to
transport audiences to quasi-spiritual planes via an excess of conjoined media
elements.
Direct theories of audio-visual mapping are also discussed by Nuno N. Cor-
reia and Atau Tanaka in Chapter 9 and Adriana Sá and Atau Tanaka in Chap-
ter 10. Both of these chapters outline unified (though not singular) theories of
audio-visual performance. Correia and Tanaka discuss different contemporary
interfaces for live multimodal audio-visual performance, including live coding,
tangible interfaces and their own audio-visual user interfaces (AVUIs). Sá and
Tanaka present a parametric model for audio-visuals. They describe a model for
using different parameters (including audio-visual ‘fit,’ and the performers posi-
tion and the ‘arena’) in order to establish a more complete model of audio-visual
interactions. Taken together, all of these theory chapters present a multitude
of possible approaches to both the discussion of and the structural, narrative or
formal uses of visuals in a live situation. These theories of the performed visual
reconnect notions of the social and the conceptual with the structural and the
formal, emphasising how Live Visuals practice makes use of a promiscuous blend
of ideas previously seen as in opposition to each other.

The Uses of Live Visuals

Audio-visual performance, live cinema, immersive and interactive art, architec-
ture and design are all contexts in which Live Visuals are commonly used. The
diversity of tasks that Live Visuals are used for, and situations in which they are
 Introduction 5

consumed, is as varied as the long history of images becoming alive. Covering

all of these areas (and more), this book makes the case that Live Visuals are much
more ubiquitous than is commonly acknowledged. The aims of the different
projects that employ Live Visuals might be extremely practical, as outlined in
relation to architectural planning in Chapter 14; wildly experimental, as out-
lined in relation to immersive virtual environments in Chapter 13; or more
commercial, as outlined in relation to design and electronic music in Chapter 15.
The key connector to these experiences is the ‘liveness’ of the visual imagery. We
do undoubtedly exist (at least at present) in a world primarily dominated by pre-
prepared, non-dynamic, ‘un-live’ visual content, from TV shows on Netf lix, to
films at the cinema, to video advertisements on billboards, to YouTube videos;
however, many of our visual experiences are becoming more demonstrably and
increasingly live, immersive and interactive. From virtual reality (VR) experi-
ences on headsets, to live theatre performances, to massive online role-playing
games, artists and designers are now more aware of the possibilities and richness
of visual liveness. We will no doubt see further uses of Live Visuals in a multi-
tude of contexts, and post-COVID these will be experienced almost certainly
in different contexts than the music stage or the dance club. Undeniably, the fact
of visual liveness is no longer in doubt, as the technologies and techniques that
have enabled the performance of the image in a manner similar to the live musi-
cian are now established, functional and rich with artistic and cultural potential.

Notes
1 Oliver Grau, Virtual Art: From Illusion to Immersion (Cambridge, MA: MIT Press, 2003),
238.
2 Cretien van Campen, The Hidden Sense: Synesthesia in Art and Science (Cambridge, MA:
MIT Press, 2007), 1.
PART I

The History of Live Visuals

1
INVENTING INSTRUMENTS
Colour-Tone Correspondence to Colour-Music
Performance (Pre-1900)

Maura McDonnell

Introduction
This chapter focuses on examining some of the pertinent activities of this pre-
1900 historical period. In a fascinating and intricate narrative thread that runs
from Greek antiquity to the present day, the 18th- and 19th-century effort
to correlate and combine sound and colour led to the significant invention of
colour-music performance instruments. As an early art-science development
based in the theories of optics and, to a lesser extent, acoustics, it led to the
emergence of new hybrid art forms. Many activities sought to theoretically and
physically correlate properties between the hearing of sound with the seeing of
colours, and several historical authors correlated colour hues with musical tones.
An overarching term, the colour-tone analogy1 describes the various versions of
such correlations. Subsequently, there were several declarations of a new colour
music art made by inventors that were then taken up by artists and musicians in
the 20th century. Today’s technologies facilitate the integration of the optical
and the acoustic, facilitating the most diverse possibilities and permutations for
combining images and sounds, where the optical and the acoustic have become
a type of malleable material from both a conceptual and technical point of view.
Digital technology, as Dieter and Naumann note, “has rendered the optical and
acoustic de facto calculable, transformable, and manipulable at will.”2 The his-
torical efforts accounted for here, however, demonstrate that our contemporary
audio-visual culture and audio-visual performative practices of the 20th and 21st
centuries, in all the variable methods, thought and technology, are rooted in a
past that had similar interests.
In this chapter, the significance of the colour-tone analogy will be traced
from its theoretical and mathematical underpinnings to the emergence of vari-
ous authors’ technical proofs and unique instrument inventions that attempted
DOI: 10.4324/9781003282396-3
10 Maura McDonnell

to provide a practical demonstration of the authors’ proposed theoretical connec-

tions. This led some inventors to abandon a strict concern to provide a mechani-
cal demonstration of a scientific proof towards the invention of performance
instruments, and so, to move toward exploring the aesthetic and imaginative
possibilities of a mobile colour either like music or for music performance.

The Analogy Between Musical Tone and Colour Based on

Mathematical Laws
From the ancient Greeks to Sir Isaac Newton (1642–1727), analogy was a means
of generating new knowledge that contained the certainty of mathematical rea-
soning. The ancient Greeks believed that it was the mind that could make intel-
ligible “the pattern of nature” and the eternal laws of the universe.3 They thus
believed that this pattern of nature could be best understood through geometry,
where the connections and correspondences of the universal harmonies of nature
“in the various realms of perception”4 could be revealed through mathematical
thinking, and therefore the “objects of our sense-perceptions appeared to be
capable of being expressed by geometric shapes.”5 They reasoned that simple
geometric forms, ratios and numbers could describe the order of natural phe-
nomenon. Amongst the many types of analogies, Ralph McInerny explains that
when analogy is based on a proportion or proportionality, the proportions sig-
nify a determinate relation of one quantity to another, and the proportionality
is based on the similarity of the two proportions6; for example, the propor-
tion of 4 to 2 is double. Analogies based on mathematical proportionality, then,
can be deployed to compare those quantities from one domain to quantities
from another domain through a common mathematical relation. This, in turn,
enabled analogies to be made between hearing and seeing across “phenomenal,
metaphysical, mathematical, physical, and physiological levels.” 7 Colour and
light were analogically compared to music tones and sound through such math-
ematical and geometrical relations.
The Greek philosopher Pythagoras (570–495 BC), and his immediate fol-
lowers the Pythagoreans, argued that proportion and number underpinned the
acoustical laws of harmony and consonance in music. Proportion could be used
to characterise various magnitudes operating in the physical devices that sounded
music intervals, such as weight (as in hammer weights) or length (as in pipe
lengths or string lengths).
To see the mathematical truth of this music harmony, the physical magni-
tudes could subsequently be disregarded,8 and the focus could then be on the
actual numerical proportions of the music intervals. The Pythagoreans therefore
discovered that simple geometric mathematical laws characterised the pleasing
consonances in the musical scale and discovered that “the relative lengths in
every harmonious combination of plucked strings can be expressed as ratios of
whole numbers.”9 For example, what was considered to be the pleasing conso-
nant intervals of the octave, the fifth and the fourth, both ref lected and exhibited
 Inventing Instruments 11

FIGURE 1.1 Illustration in Theorica musicae by Franchino Gaffurio (1492) depicting

Pythagoras’s experiments on various instruments such as bells, strings
and f lutes, calculating the geometric proportions of music intervals.
Source: Public Domain. https://commons.wikimedia.org/wiki/File:Gaffurio_Pythagoras.png

simple numerical laws and could be numerically represented by the geometric

ratios of 16:8 or 2:1, 6:4 or 3:2, 12:9 or 4:3. Italian music theorist and composer
Franchino Gaffurio (1451–1522) provides a woodcut illustration in Theorica musi-
cae (1492) of the various proportions to various magnitudes examined by the
Pythagoreans in relation to various musical instruments (see Figure 1.1). The
Pythagoreans calculated all the numerical ratios of the music intervals. In rela-
tion to colour, the Pythagoreans delineated four colour species from which all
other colours were made, and these were white, black, red and yellow.10 The pro-
portions of 2:1, 3:2, 4:3 and 9:8 that the Pythagoreans found in music, however,
were associated with colour hues, arranged from white to black, in subsequent
theories of colour in the Greek period.11
Aristotle (384–322 BC) also compared colour to music. In his De sensu et sen-
sibli Aristotle set forth the idea that “the colour groupings most pleasing to the
eye rest on the same basis of simple number relationships as do the consonances
12 Maura McDonnell

of music.”12 The quantity that Aristotle applied the number relation to was that
of brightness, or as was said then, translucence. Aristotle’s system of tonal pro-
gression of the colour species is based on the principle of how much brightness
emanates from an object to give us the particular shade or hue of a colour, and
he states: “It is therefore the Translucent, according to the degree to which it
subsists in bodies (and it does so in all more or less), that causes them to partake
of colour.”13 Aristotle identified seven fundamental colour species, with white
and black at the outer bounds and of which the five intermediate colours: yellow,
scarlet, purple, green and deep blue, are arranged in a geometrical proportion-
ate relation between the parts black and white in the hue, with red having more
black in it and yellow having more white. In other words, his arrangement of
the main colours is based on a distinction of shade,14 with the two quantities of
light (white) and dark (black) in the colour expressed as a geometric ratio that
represents their proportions.
The principle of consonance operated in this balance between dark and light.
The most pleasing consonant colours have a simple whole number ratio. The
proportions of white or black in the colour are compared to the musical intervals
of a fourth (4:3) and a fifth (3:2). In a passage of text in De anima, 439b-440a,
Aristotle explains that the most agreeable colours are in analogy with the ratios
of concords in music:

It is thus possible to believe that there are more colours than just white and
black, and that their number is due to the proportion of their components;
for these may be grouped in the ratio of three to two, or three to four, or
in other numerical ratios, or they may be in no expressible ratio, but in an
incommensurable relation of excess and defect, so that these colours are
determined like musical intervals. For on this view the colours that depend
on simple ratios, like the concords in music, are regarded as the most attractive,
e.g., purple and red and a few others like them – few for the same reason
that the concords are few – while the other colours are those that have no
numerical ratios.15

From the 13th century onwards, the idea of the same harmonious proportions
occurring in the sounds that please the ear and the colours that please the eye
continued into theoretical texts,16 and several philosophers based their arrange-
ments of colour on similar divisions to Aristotle’s seven colours, with black and
white at the two extremes from which, and in between, all the rest of the colours
are compared. Rolf G. Kuehni notes that the selection of seven main colours
“had an additional purpose: to show congruence with the then well-established
musical scale of seven tones,”17 and further explains that this relates to a belief in
“the possibility of a law of colour harmony comparable to that of musical conso-
nance.”18 The Renaissance thinkers investigated further the mathematical basis
of nature and believed that the mathematical relationships in a phenomenon are
 Inventing Instruments 13

pre-existing, and even though our senses may only come to know events one by
one, these mathematical relationships are there, awaiting to be discovered. Mor-
ris Kline explains that for Renaissance thinkers, the “universe was likened to a
mathematically and harmoniously designed machine in which the mechanical
action of forces of the motion of objects in space and time obey mathematical
laws and the realities of shapes in space or motion in space and time could be
expressed mathematically.”19
They thus turned their attention to examine not just static objects and shapes
but also the motion of objects in space and time. Shapes in space and motion in
space were considered to be an integral part of the concept of extension and, as
expounded by René Descartes (1596–1650), an important property of all moving
realities.20 Descartes’s theory of colour relates to the conception of the mechani-
cal properties of the constituent particles of a concept of the aether, a pervasive
medium, that is continually in motion. The various perceived colour hues are
given their distinct colour species by a state of rotation that corresponds to angu-
lar velocities of the aether,21 so that this results, for example, in “the particles
which rotate most rapidly giving [rise to] the sensation of red, the slower ones of
yellow and the slowest of green and blue.”22

Music Harmony to Colour Scales

Of interest here is the revival of a discussion of colour in terms of musical har-
mony. For music theorist Gioseffo Zarlino (1517–1590), colour was a natural
quality, along with size, shape, distance, state and motion.23 Zarlino appeals to
the reactions of the senses in what appeals to hearing and seeing, stating in Istitu-
tioni harmoniche (1558) that the “reaction of the ear to the combinations of sounds
is analogous to the reaction of the eye to the combination of colours. Such com-
binations have a kind of harmony, in that they are composed of diverse colours.”24
He compares colours to music composition, noting that in music, it is the com-
position of tones which is pleasing for hearing, and so too for colours, it is in the
composition of the diversity of colours that is most pleasing for sight.25 Filippo
Mocenigo (birth/death dates unknown) in 1581 devised a linear system of three
colours intermediary between black and white, and these are golden, hyacinth
and red. Like Aristotle, he sought to relate the location of the colours to music
in a harmonic relation of intervals, but in a rather unusual way: for example, the
colour hyacinth in the middle is described as a diapente interval with respect to
black, a diatessaron interval with respect to white, a semitonum with respect to red
and a ditonum with respect to yellow. The relationship between black and white
is the diapason [octave].26 Marin Cureau de la Chambre (1594–1669) focused on
establishing a theory of colour harmony based on music harmony and used a
two-octave base from which to assign seven colours in a relation of successive
fourths, fifths and octaves. Dissonance in colour was compared to dissonance in
music. For example, colour combinations were described as clashing if they were
14 Maura McDonnell

in dissonance. Red-blue and blue-purple combinations were dissonant, yellow

clashes with either red or purple and green being in the middle between black
and white harmonised with all colours.27 Athanasius Kircher (1602–1680) sup-
ported the hypothesis that sound and light were part of the same type of physical
processes: “they both are ref lected by plane surfaces; both can penetrate into
denser media and are refracted in the process; and both can be concentrated
in a focus by a hollow mirror,”28 and he allocated colours to tone intervals.29
Maarten Franssen further notes that the significance of Kircher’s colour-tone
analogy is how it highlights an imaginative association of sounds in music to
colours. Kircher imagines a diversity of colours in relation to sounds. Franssen
quotes two short passages from Kircher, first, one imagining what would one see
in the air when music sounds: “If, when a musical instruments sounds, someone
would perceive the finest movements of the air, he certainly would see nothing
but a painting with an extraordinary variety of colours”30; and, second, one that
stresses the harmony of colours and sounds as what pleases: “The colours also
have their harmony, which pleases no less than music, and this analogous har-
mony even has a very strong power to excite the affects of the mind.”31

Newton’s Musical Division of the Spectrum

Sir Isaac Newton started experimenting with the phenomenon of colours in
the late 1660s. It is Newton’s experiment that incorporated a detailed and dia-
grammatic colour-tone analogy that inspired subsequent colour music instru-
ment inventors to make such connections visible. Newton was interested in the
physical properties of light for both theoretical and practical purposes in order
to solve problems – for example, in order to solve problems in the design of
lenses for optical instruments such as telescopes. At the same time, Newton also
“sought to analyse colours mathematically.”32 His experiments consisted of vari-
ous tests with the refrangibility of light (that is, the disposition of rays of light to
be refracted) and the ref lection of white light with prisms projected onto paper
or strings in a darkened room. Newton presented his initial experiments in lec-
tures on optics at the Royal Society Lectures in London, where he contributed
“further examples of the process of analysis by which coloured phenomena are
produced.”33 The experiments of particular note here are those where Newton
presents a comparison of colour with the notes of the musical scale.
In a paper, titled “New Theory about Light and Colours” read to the Royal
Society in 1672,34 Newton demonstrated that white light consists of a spectrum
of a heterogeneous mixture of individual light rays. Each distinct light ray is per-
ceived as a distinct colour band (for example, a distinct colour species such as red,
with all its varying shades, is one distinct colour species). Newton was able to
estimate the characteristic refractions of these individual rays. Each colour band
therefore belonged to its own independent ray of light or its own wavelength.
How he came to find that colour bands had individual and different wave-
lengths is through observation, hypothesis and measurement. Newton observed
 Inventing Instruments 15

that a circular beam of sunlight or white light, when it passes through a small
hole and is collected by a lens and focused on a prism, produces an elongated,
round-edged, oblong shape and not a perfect circle of light, hypothesising that
the projected image ought to be round if the rays were equally refracted 35 (see
Figure 1.2). The shape consists of several rays of light that arrive at different
places in the shape, according to their length, and this is why the shape has an
elongated appearance.
Newton states: “Light which is different in colour, differ also in degree of
refrangibility and The Light of the Sun consists of rays differently refrangible.”36
Newton demonstrated that the physical element in colour is a set of wavelengths
that the eye receives and understands as colour.37 Up to the time of Newton’s
experiment, it was generally thought that colours were mixtures of light and
darkness and that prisms imparted colours to light. Newton’s experiment proved
this incorrect, and he demonstrated that it was the interaction of the object with
light that creates the colour of the object and not the object itself.38
From this initial experiment, Newton identified that white light consisted
of five distinct wavelengths. Each of these wavelengths corresponded to a per-
ception of a distinct colour species, and therefore, five colours were identified,
and these were red, yellow, green, blue and purple. In subsequent experiments
he identified seven wavelengths corresponding to the perception of seven dis-
tinct colours. These colours were deemed to be the principal colours of the
spectrum.
Newton’s interpretations of the results of his experiments were disputed by
his contemporaries, especially by Robert Hooke,39 which led to Newton provid-
ing even more improved and detailed descriptions of his experiments and results
in order that the findings would not be misunderstood. Interestingly, Hooke
used an acoustic analogy to dispute Newton’s theory that colours were rays of
light within white light and comparing it to whether the vibrations of a string in
a monochord have motions dormant within it, and so, by analogy arguing how
light could have all the colours in it.40 Newton addressed Hooke’s criticism in his
second optical paper of 167541 by including a detailed music-to-colour analogy
to demonstrate and reason that “the seven bands of colour in the spectrum have
widths in the same harmonic ratios as the string lengths on the monochord that
produced the musical scale,”42 dividing the spectrum in the manner of a musical

FIGURE 1.2 Illustration of the spectral oblong shape PT to the left of this diagram,
from The First Book of Optics, Part 1 (1718). Public Domain, edited by
Maura McDonnell.
16 Maura McDonnell

chord. Newton also adds two more colours, orange and indigo, to the spec-
trum of colours to make up seven principal colours. Commentators have noted
that Newton was susceptible to a colour-tone analogy because he had a philo-
sophical leaning towards a mathematical universal law of nature and appeared
to be answering Hooke’s criticism by demonstrating a colour-tone analogy in
his experimental results. Newton had learned about music harmony mainly for
mathematical reasons and “throughout his life retained a belief in the musica mun-
dana, or universal harmony of the world.”43 Thus, as Olivier Darrigol notes, “he
assumed that colours and tones obeyed the same rules of harmony.”44
One of Newton’s early diagrams to illustrate the correspondence of the bands
of colours with the seven notes of a Dorian mode scale (see Figure 1.3) takes the
lines separating two different colours and places them at the frets of a monochord
to yield the notes of a music scale.45 Penelope Gouk points out that what mat-
tered for Newton about using this Dorian music scale was its symmetry, as the
Dorian mode scale is a symmetrical one, as the pattern of tones and semitones are
the same both ascending and descending.46
Darrigol explains how the diagram yields the colour-to-tone analogy:

The points x, E, G, H, I, K, M mark the ends of a vibrating string begin-

ning at z. The corresponding sounds (marked in a contemporary notation
that has nothing to do with the present naming of notes in Latin languages;
DEFGABC in the modern English scale) yield the limits between two suc-
cessive colours of the spectrum ABCD. For instance, y is an octave higher
than x and I is a fifth higher than x because yz/xz = 1/2 and Iz/xz = 2/3.47

Newton in his publication Opticks (1704) presents a more “definite scientific

statement of the demonstrable physical relationship between colour and musical

FIGURE 1.3 Isaac Newton’s spectrum with bands of colour compared to the notes of
a musical scale (colours have been added by the author for illustration
purposes). The line from P to Z represents the length of the string, the
line from y to z represents the octave and the various points of colour
bands denoted from E to M are aligned to the musical scale notes from
Sol to Sol.
Source: Creative Commons. https://gallica.bnf.fr/ark:/12148/bpt6k3362k/f107.item
 Inventing Instruments 17

sound.”48 He elaborates further on the analogy between colour and sound, as

can be seen in another version of a colour-tone analogy diagram to “define the
refrangibility of the several sorts of homogeneal Light answering to the several
Colours”49; the mathematical proportions and calculations used from a musical
scale are aligned to the confines of the colours and expressed in mathematical
ratios.
Newton explains how he and his assistant measured the dimensions of the
distinct colours by drawing lines across the spectrum, noting “the confines of
the Colours that is of the red MαβF of the orange αγδβ, of the yellow γεζδ, of the
green εηθζ, of the blue ηιχθ, of the indico ιλμχ, and of the violet λGAμ.”50 He
does not say that this is a very exact experiment and admits that the division in
the manner of a musical chord as applied to the distinct colours is an approximate
division:

I found that the Observations agreed well enough with one another, and
that the rectilinear sides MG and FA were by the said cross lines divided
after the manner of a musical Chord. Let GM be produced to X, that MX may
be equal to GM, and conceive GX, λX, ιX, ηX, εX, γX, αX, MX, to be
in proportion to one another, as the numbers, 1, 8/9, 5/6, 3/4, 2/3, 3/5,
9/16, 1/2, and so to represent the Chords of the Key, and of a Tone, a third
Minor, a fourth, a fifth, a sixth Major, a seventh, and an eighth above that
Key: And the intervals Mα, αγ, γε, εη, ηι, ιλ, and λG, will be the spaces
which the several Colours (red, orange, yellow, green, blue, indigo, violet)
take up.51

An even more detailed diagram was provided of the colour-tone analogy in New-
ton’s circle of colours (see Figure 1.4), also published in Opticks. The colour circle
represents many of the findings and measurements of the characteristics of colour
from many of his optical experiments. The musical scale goes from the note D
to D around the circumference. The various unequal proportions of colour are
represented in different-sized arc subdivisions that ref lect the geometric propor-
tions of the intervals of the music scale and the proportionate amount of space
each colour species occupies in the spectrum, as represented by small discs placed
under each colour. A type of closure is suggested in the circle of colours, where
the end and beginning colours of the spectrum are similar in colour perception –
that is red and violet – and hence emulates the octave of the musical scale. The
aim of the circle of colours is to also demonstrate rules for determining a colour
produced by “the additive mixing of coloured lights,”52 and it provides a means
to “plot mathematically the location of any mixture”53 in relation to white and
their mixture. This circle also models information about the various gradations
of intensity of a colour from white light to the fullest intensity and provides
a quantitative measure of the colours’ distance from whiteness.54 Newton had
found that not only can the spectrum be split to produce all the coloured lights,
so too can the coloured lights be reconstituted to make white light.
18 Maura McDonnell

FIGURE 1.4 Newton’s circle of colours published in Opticks (1704) book 1, part 2,
Figure 11, with colour added in by the author and the ratios of the music
scale added in, taken from David Brigg’s article on the dimensions of
colour.
Source: This diagram is adapted by Maura McDonnell to incorporate colour and ratios. This image
has been adapted from this one in the public domain: https://commons.wikimedia.org/wiki/
File:Newton%27s_colour_circle.png

White light is represented by the letter ‘O’ in the centre of the circle. From his
“Colours of Thin Plates” experiments, also presented in Opticks, Newton had
observed that opposite colours can be produced by transmitted and ref lected
light,55 and this is also depicted in the circle, which we now know as comple-
mentary colours. He had also found that mixing the rays of light or superimpos-
ing two coloured lights can produce a colour perceptively similar to another
colour56 and that if two of the principal coloured lights are mixed, they yield a
type of off-white, or neutral-type colour.57
In the diagram, a colour ‘Y’ is the result of a mixture of colour that is worked
out mathematically and proportionally to a centre of gravity ‘z,’ along the radius
‘OY’ representing the relation of the colour mixtures to each other and to white
light; thus, the “ratio Oz/OY gives the ‘fullness or intenseness’ of the colour.”58
As is noted by Shamey and Kuehni, the colour circle was, “if lacking the non-
spectral colours, an early representation of what became centuries later, in a
modified form, the chromaticity diagram.”59

Louis-Bertrand Castel’s Ocular Harpsichord and Colour-Music

By the 1720s it was generally accepted that white light can be separated into its
simple principal colours.60 Yet despite this, Newton’s colour-tone analogy was
not as successful in gaining acceptance amongst natural scientists of the day,
 Inventing Instruments 19

or even amongst supporters of an analogy between colour and musical tones.61

Many disputed the colour-to-tone choices that Newton had made in particu-
lar but agreed with the principle of a colour-tone analogy. Those interested in
furthering this work on the colour-tone analogy therefore sought to devise and
arrange their own systems of colour-to-tone correspondences. Louis-Bertrand
Castel (1688–1757) was one such author.
Louis-Bertrand Castel was a Jesuit priest, physicist and mathematician. He
was science and mathematics editor of the Mémoires de Trévoux (formally called
Mémoires pour l’Histoire des Sciences & des beaux-Arts) from 1720 to 1746 and pub-
lished several manuscripts for this and other journals. He had translated Newton’s
Optiks into French in the Mémoires de Trévoux in 1723 and was keenly interested
in Newton’s and Kircher’s experimental scientific work on the colour-tone anal-
ogy. Castel, however, wanted to use the colour-tone analogy as both a scientific
and philosophical basis for his declaration of a new art of colour-music and the
invention of an instrument to bring that analogy to human experience – the
clavecin oculaire, or ocular harpsichord. Philosophically, he believed that a rela-
tionship did exist between colour and musical tone and that it was a natural
one: that is to say, it came from the basis of a fundamental principle of nature.
In this belief, Castel was heavily inf luenced by Kircher’s theory that the analogy
between sound and light holds because both belong to the same physical pro-
cesses of vibrational phenomena. This theory of Kircher’s, Castel acknowledges,
led him to the idea of an ocular harpsichord.62
Castel accepted that Newton had discovered a scientific and mathematical
proof for a colour-tone analogy, but this only served to consolidate the theory
that such a principle existed hidden in the depths of nature. Therefore, Castel
agreed with a colour-tone scale in principle. The proposed ocular harpsichord,
by contrast, was to be an ordinary harpsichord adapted in its mechanism so that
the keyboard could play colours “in all their harmony.”63 It was, however, once
Castel had started to go about building the instrument that he encountered prob-
lems with the practicalities of investigating and implementing Newton’s colour-
tone analogy into his design. He realised that at a very fundamental level, he did
not actually agree with the choices of colours to musical tones and the details of
Newton’s colour-to-tone correlations. One can only imagine how difficult this
must have been for Castel to resolve in his own work, considering the fact that
he wished to base his whole ocular harpsichord project on a solid fundamental
natural-scientific basis.
Initially, Castel worked with Newton’s alignment of colours to the musi-
cal tones but ended up devising an entirely different colour-to-tone correspon-
dence. There were many aspects of Newton’s analogy with which he did not
agree, such as, the starting colour and tone of the colour-tone scale; how many
distinct colours there were; and he noted that Newton’s spectra do not occur at
all distances from the prism, and so, are a special case. It is arguably the case that
Castel (and many others) misunderstood the scale Newton used, which was an
20 Maura McDonnell

older (Dorian) modal scale.64 Castel nonetheless did not agree that the tone C
should correspond to the colour violet because violet was a colour that can come
from mixing red and blue or that the tone D should be the fundamental tone and
the starting point for the scale. Castel, in other words, was trying to resolve the
sequence of colour in the spectrum from red to violet to the principles associ-
ated with mixing colours, assigning the most stable primary colours in dominant
positions in relation to music and the compound colours produced from mixing
the primary colours to intermediary positions in music. In the music scale, the
most stable intervals were the tonic, dominant fourth, and so, in his arrangement
Castel sought to align these musical configurations to the three primary colours
of red, blue and yellow (see Figure 1.5).65
For Castel, blue had to be the fundamental colour-tone because it was a more
stable colour in terms of mixing. It also was the colour of the sky, where we
see all colours in nature against it, and therefore it was a basse fundamentale of
nature.66 The tone C was a fundamental tone in music according to Castel, and
so his colour-tone scale started on the note C and not on D, as in Newton’s scale.

FIGURE 1.5 Louis-Bertrand Castel. Diagram comparing Newton’s division of the

spectrum into seven distinct colours and his division of the spectrum into
twelve distinct colours that are based on an arrangement of identifying the
primary colours and the other intermediary colours from the spectrum.
Source: Public Domain. https://archive.org/details/gri_c00033125008507283/page/n441/mode/2up
 Inventing Instruments 21

Castel used the C major scale and not the Dorian scale, which Newton used in
his analogy.
Castel went on to devise a chromatic colour-tone scale system, comprising
12 colours representing an analogy with the 12 tones of the chromatic musi-
cal scale, ordered with reference to what he calls “natural colours,” “artifi-
cial colours” and “natural order of tones,” that is, primary colours and those
colours that result from a mixing of primary colours in correspondence with
the notes of the music.67 Maarten Franssen remarks that Castel believed that
the discernment of the 12 musical tones and colours are discerned from a
continuum of all possible sounds within the octave and of all possible colours
within the spectrum. Castel’s final chromatic colour-to-tone result consisted
of the notes of the C major scale ordered to the following colours in succes-
sion: blue, celadon, green, olive, yellow, fallow, nacarat, red, carmine, violet,
agate, violaceous.68
Castel’s ocular harpsichord was conceived of as not only an extension of this
colour-tone analogy but also as a means to make the alliance between colour and
music physically present in a display of the harmonic form of colour suggested
in the colour-tone analogy, and thus the device was conceived to demonstrate
a scientific principle. This demonstration could subsequently both support the
theory underpinning the instrument and enable the instrument to make visual
the theories of colour by enabling the actual musical keys of a keyboard to facili-
tate the ‘playing’ of the corresponding colour. This must have been extremely
exciting for Castel, for, as he writes, the major advantage and possibility of such
a device would be that it “gives to the colours, apart from their harmonic order,
a certain vivacity and lightness which on an immobile inanimate canvas they
never have.”69 Yet from the beginning in his thinking and thoughts about this
instrument, Castel observed and noted other aesthetic possibilities for this instru-
ment that could bring a kind of moving art of colour, hitherto unseen, into
existence. This is why Castel believed and imagined that by arranging colours
in an analogous way to the arrangement of musical tones, such could be appreci-
ated “so that a whole new form of art would emerge, a music of colour.” 70 From
this time onwards the ocular harpsichord, as it progressed as an idea, was also
intended to be designed to facilitate the display and performance of colour in
a colour harmony that was equivalent to music harmony, what Castel named a
colour music. An unknown author writes in 1757 that the ocular harpsichord
contained the pre-figured possibility of an appeal to the visual senses in analo-
gous fashion to the auditory senses that comes from listening to music; referring
to Castel, the author writes:

He imagined that if colours could be made to play, and that an instrument

could be framed, which should, like the Harpsichord, have as tones and
semi-tones, its tonics and quints, etc, the variegated play of colours would
have an effect on the eyes somewhat approaching to what the sounds have
on the ears.71
22 Maura McDonnell

Several scholars have given an account of Castel’s ocular harpsichord and its impact
and legacy on the science and music community of his time and after,72 and it is
noted in the literature that no actual, definitive design blueprint was ever provided
and that the evidence for the actual construction of an instrument is not very clear.
However, we do know that Castel was engaged in the ocular harpsichord project
in its design, in arguments for its proof, in the build of demonstration models and
in demonstrations of these models for a large period of his life, from 1725 to 1754.
Castel announced the invention of such an instrument in 1725 in the Mercure de
France,73 but again this is only in terms of a proposal and a question into the pos-
sibility of what this ocular harpsichord will produce.74 Explaining what this instru-
ment is and stressing what it can and will demonstrate, Castel writes:

What is a clavecin? It is a set of stretched strings which sound in their

length, and in their thickness, a certain harmonic proportion. They are
sounded by means of a little tangent which picks them, producing the
diverse tones and chords of the music. Now the colours follow the same
harmonic proportion. Let us take just as many of them as needed to form
a complete keyboard and dispose of them in such a manner that by apply-
ing the fingers in a certain way, they will appear in the same order and the
same combination as the corresponding sounds do to their own touches.
When I say dispose, I mean only that they should be exposed to the air, not
actually placed on a canvas since it is necessary that one (colour) should
be able to appear without the other, or with such others as the player may
bring forth [. . .]. It is necessary that in moving the fingers as for an ordi-
nary clavecin, the pressure of the keys should cause colours to appear, in
their combinations and chords, in a word, in all their harmony, which cor-
responds precisely to that of the music.75

In 1726, Castel continues to develop his argument for such an invention by pro-
viding a set of geometrical demonstrations of the universality of its statements.76
These geometrical arguments were a set of propositions rather than being mathe-
matical proofs,77 and the geometrical arguments were further extended in 1735.78
He wrote his manuscript L’optiques des Couleurs in 1740,79 addressing the problem
of how to devise a colour scale system that facilitates or reproduces a mapping
to the range of tones in music across octaves. He solved this by creating a series
of ‘chiaroscuro units’ (light and dark) for each colour that results in the possibil-
ity of having the lighter or darker colour of each primary colour available at the
octave positions of the musical scale.
In 1734, Castel presented a more advanced model of the instrument, even
though he did say that it is “necessarily imperfect.”80 He provided a vague account
of this model and the music-to-colour correspondences that he employed. The
colours could be played individually or in combination, and the interval between
each individual colour corresponded to the interval between the tones. Playing
 Inventing Instruments 23

colours in combination was considered analogous to music chords, as they would

exhibit the same harmony effects except through colour. Colours across the
octaves had the same hue but either lighter or darker, depending on the octave
placement of the key being pressed. Castel referred to the playing of the instru-
ment as being a movement in colours, like a melody that comprises a movement
of notes. He described how such a system of colours might work in the instru-
ment and how practice is needed to master the playing of the colour according
to the colour you want to select:

Do you want blue? Put your finger on the first key to the left. Do you
want the same only I degree lighter? Touch the 8th note. If you want it 2
degrees, or 3 degrees . . ., touch the 15th, or 22nd, or 29th, or the last to
the right. If you want blue-green, touch the first black to the left. Do you
want red, and which red? Crimson-red? That is the 4th black. You have
only . . . to know your clavier and know that blue is C and red is G etc.
This you can acquire with three days practice.81

The unknown author of the publication “Explanation of the Ocular Harpsi-

chord Upon Shew to the Public,” published in 1757, just after Castel’s death,

FIGURE 1.6 A drawing by Charles-Germain de Saint-Aubin depicting Louis-Bertrand

Castel playing the clavecin optique model that Castel exhibited in 1730.
Source: Public Domain. https://waddesdon.org.uk/the-collection/item/?id=17198
24 Maura McDonnell

but believed to be Castel’s pupil, a friend and assistant,82 describes the 1730s
trial demonstration model as having consisted of coloured slips of paper behind
cartridge paper that were raised and revealed when the finger pressed the keys of
the harpsichord. A fully realised model at a public exhibition in London is also
described. In this model, coloured slips of paper were replaced with glass and
lamps. Mason notes that this demonstration model “comprised a box with the
usual keyboard in front and about 500 candles lit from behind a series of 50 glass
shields which faced back toward the player and viewer. When a key was pressed
the glassed opening was supposed to shine from the transmitted rays of a bit of
‘colour lightning.’”83
Castel did gain some celebrity from pursing his project of writing about the
design of the ocular harpsichord and also gained some ridicule from those who
did not support it as an instrument or support his suggestion of a new colour
music art. A caricature drawing by the artist and draftsman Charles-Germain de
Saint-Aubin (1721–1786), for instance, depicts Castel performing on what is
considered to be the 1730 version of his clavecin oculaire invention in 1740–c 175784
(see Figure 1.6). Unfortunately, this appears to be the only graphic we have of
Castel’s ocular harpsichord: it is not ideal, but it is an image from its time.85
What is of most interest here is the aesthetic extension of the colour-tone
analogy that Castel writes about in the passages of his writings that depict
more imaginative possibilities for the instrument, which were often dismissed
and ignored by commentators as being just that – his imagination and not
of any relevance to the field of physics, philosophy or music. Luckily, for
audio-visual performance and visual music artists today, such passages are
an absolute delight, and they provide great evidence that Castel really did
see the possibilities of a new art that was more than a mechanical art of com-
bined harmonious colours and sounds, but one that could marry music with
an intensely imaginative mobile visual display.86 For example, in his early
experiments with prisms, Castel really enjoyed the colour light spectacle that
ensued, recalling:

One day, when the sun was shining brightly, having shut all windows of
a room and having placed four or five prisms in front of some holes that
I had made in the shutters, making them turn incessantly, I watched on
the opposite wall a moving tapestry which, without any other concert
of harmony, presented me with the most agreeable spectacle that I could
remember having ever seen or heard.87

Castel refers to the life of movement that could be obtained by using various
ref lective jewels, light sources and mirrors:

For example, one could make the colours themselves from true jewels . . .
the greens with emeralds, the reds with garnets, rubies, carbuncles, etc.
And what brilliance and lustre would not such a spectacle bring to light in
 Inventing Instruments 25

every part, sparkling like the stars, with jacinths followed by amethysts, then
rubies, etc., by the light of torches in an apartment furnished with mirrors? It
would be an infinitely brilliant object, a sort of immobile decoration where
everything would be varied. But what would it be if animated and given a
type of life through movement, a regular, measured, harmonic, and lively
movement? It would be charming, an enchantment, a glory, a paradise!88

In this description, Castel refers to a play of all sorts of figures, from human, to
nature, to animals, to allegorical figures and geometric figures, even thinking
of a play of landscapes and scenes. It is this idea of “a play of ” that is most inter-
esting, as it suggests the possibility of a moving image, “a type of life through
movement,” to accompany a music concert, for, as he writes:

One could make a play of all sorts of figures, human and angelic animals,
f lying creatures, reptiles, fishes, four-footed beasts, even geometric fig-
ures. One could, by a simply play demonstrate all the concord of Euclidian
elements. One could make a play of fantastic figures, of hippographs, of
centaurs, etc., allegorical figures, muses, dryads, naiads, etc. Or one could
make a play of f lowers, taking the rose for the colour rose, the coxcomb for
the purple, the violet for the violet, jonquils for the yellow, marigolds for
the gold, so arranged that each stroke of the hand on the keyboard would
represent a f lower-bed, and the result of playing would be a moving diver-
sity of animated f lower-beds.89

It is of course without doubt that the origins of Castel’s ocular harpsichord idea
had, in his own time, a variety of purposes and reasons to be invented, the
most pertinent one being as a demonstration and proof of a scientific reasoning
about the validity of a colour-tone analogy. As an instrument, however, it was
between science and art. Castel wanted to make the play of colours with such an
instrument as rational as possible in order that some universal principles could
be ascertained in its wonderment, and thus advance a science of colour music in
analogous fashion to advances in either visual art or music science. It was none-
theless left to other inventors after Castel to succeed in devising such practical
designs and plans for a physical instrument that explored a colour-tone analogy
and colour music.

From Ocular Harpsichords to Colour Music Instruments

In his essay, De novo musices, quo oculi delectantur, genere (On a new kind of music,
enjoyed by the eyes), Johann Gottlob Krüger (1715–1759) presents a blueprint
diagram for an instrument in 1743.90 Krüger’s proposed design was both in direct
response to Castel’s ideas about an ocular harpsichord and a departure from them
in detail. He devised a colour-tone analogy scale that matched a sequence of
eight colours to the eight tones in the scale of C major; the colours were red,
26 Maura McDonnell

orange, yellow, green, blue, purple, violet and back to red. The colour music
scale was different from Castel’s and more like Newton’s scale reversed. Krüger
also did not agree that a physical analogy between light and sound made the
perception of it pleasurable, nor agreed with the approach that Castel had taken
in the presentation of the colours. For Krüger, it was the distinguishing of conso-
nance from dissonance which supported the pleasing aspects of sound and image,
and this was a psychological phenomenon.91 He found some of Castel’s colour
harmony results, such as the colours that corresponded to the interval B-C, as
dissonant.92 He also did not agree with the lack of facility for the blending of the
colours in Castel’s instrument, as Castel’s colours followed a more melodic path.
This only enabled the colours to be played in sequence, and hence there was no
opportunity to see the harmonic aspects of colour. Krüger was also interested in
providing a solution to facilitate a more harmonic sense of colour and to facilitate
the simultaneous mixing of colour. In his instrument design plan, he wanted to
provide a means for blending colour tones into chords of colour. Krüger came to
call his instrument an ocular harpsichord and writes:

Anyone observing the beautiful colours that arise in the prism is easily led
to the insight that it might be possible to delight the eye by the alternation
and blending of the seven colours just as much as the ear by the seven tones.
I have sketched such a machine . . . which is not unworthy of the name of
ocular harpsichord.93

Krüger’s ocular harpsichord design provided a way to mix colours when the
relevant musical key was pressed by having a means to control the size of the
display of coloured glass in front of a light beam coming from a candle in cor-
respondence with whether the pitch was low or high sounding. It thus was able
to control the projection of coloured light into a concentric circular form (see
Figure 1.7).94
The circular form, projected onto a background wall, was where the colour
blending, and thus the harmony chords, were produced. Thus, Krüger had facili-
tated a way to control the projection of coloured light to control the placement
and output of the light into a singular form (the circle), with colour placements
and colour blending at the various concentric circles. His instrument facilitated
this by making a series of peep holes, covered with different coloured glass and
lenses, that were sized and combined for each key of the keyboard and lighted
from behind using candles. The beams of light were projected onto one point.
When the key was pressed, a lever pushed a circular piece of glass onto the
beam, thus making a circular projection of coloured light on the wall. Franssen
describes how colour chords were formed by having different sized glass circles
for each key of the keyboard:

The diameters of the windows decreased as the corresponding tones got

lower, enabling the simultaneous projection of different coloured circles to
 Inventing Instruments 27

FIGURE 1.7 Johann Gottlob Krüger’s diagram depicting the design of the ocular
harpsichord (1743).
Source: Public Domain. https://books.openedition.org/aaccademia/docannexe/image/2146/img-2.jpg

visualize a colour chord, showing the root of the chord as a primary colour
along the circumference of the projected circle and an array of increasingly
superimposed colours towards the centre.95

Krüger’s instrument design consisted of a colour-tone analogy, with the keys of

the keyboard assigned to colours in a colour scale; however, his planned execu-
tion did not present the colours in a linear sequence aligned in a similar manner
to the linear keys of the keyboard, but he sought to present the colours together
within the confines of the circular form. The circular form provided a method
for displaying colour chords and was the most effective way to display his interest
in an analogy of colour with music harmony.

Edmé-Gilles Guyot: Musique Oculaire

Edmé-Gilles Guyot (1706–1786) was a scientist and inventor who was interested
in exploring and experimenting with the practical application of mathematics.
28 Maura McDonnell

He manufactured scientific instruments but also experimented with exploring

magical and optical tricks and in building instruments to explore such opti-
cal effects. His designs inf luenced the subsequent development of pre-cinema
devices, such as magic lanterns and phantasmagorias. What is of interest here is
a very exciting design for a musique oculaire, an optical toy based on the analogy
of light and sound that is designed to play optical colours and inf luenced by Cas-
tel’s Optique des couleurs. This design was described in one of the chapters of his
four-part book Nouvelles récréations physiques et mathématiques (1770), a book about
“all kinds of ‘recreations’ based on scientific experiments to amuse family and
friends.”96 Guyot also provides an illustration of the proposed instrument97 (see
Figure 1.8). The instrument is a practical demonstration of a colour-tone anal-
ogy, but it is also designed with the intention of playing a “colorific musique”
(colour music) and by its mechanism also explores an optical illusion.
The instrument consists of a hollow cylinder with a lamp suspended from the
top and a handle that turns the cylinder, which raises it up to reveal the colours.
In his book, Rational Recreations, William Hopper (1782) refers to the inf luence
of Castel’s treatise on the colour-tone analogy in the design of Guyot’s instru-
ment in a section titled “Colorific Music.”98 The colours representing both a

FIGURE 1.8 Edmé-Gilles Guyot’s illustration of his musique oculaire invention in his
book, Nouvelles récréations physiques et mathématiques (1770).
Source: Public domain.
 Inventing Instruments 29

colour-tone analogy and a particular music tune are painted onto thin paper
onto horizontal lines, with six measures of musical time per line and a total of 30
horizontal lines. As Hooper writes, this cylinder is “covered with double pieces
of very thin paper, painted on both sides with the colours that are to represent
the musical notes.”99 The colour score is thus read from left to right in succession,
“in conformity to the notes of the tune that is to be expressed.”100 The dura-
tion of the musical notes is represented in the length that each colour occupies
within the measure. This paper is then pasted onto the inside cylinder of the
instrument. The cylinder is 18 inches long, with the top end open and the bot-
tom attached to a freely turning screw, with a wheel attached to turn the screw
and adjust the height position of the cylinder, allowing it to be levered up and
down. This movement is to facilitate the presentation of the colours in time. The
cylinder subsequently is placed inside an encasing, a box with eight small aper-
ture openings. The openings then allow the inside cylinder to come into view
as it is levered up and down on its axis. By placing this colour music sheet into
the moving cylinder, the apertures on the cylinder are designed to only show
the colourific representation of the musical note or notes that are happening at
a particular time in the music tune. The instrument needs to move up or down
in order to read the successive horizontal lines of colour information. This strip
of paper represents the colours of the music being expressed. Colours are also
represented in various shades to represent higher and lower pitches of the music,
as can be seen in the topmost graphic in the illustration of the instrument.

Bainbridge Bishop: Color Organ

Bainbridge Bishop (1837–1905) was one of the first early colour-tone analogy
instrument inventors who actually did design and build a number of experimen-
tal colour organ instruments over a period of five years from 1875 to 1880. He
named the instrument a “color organ,” and some of his designs facilitated the
dual playing of both the colour display and the music at the same time. The organ
consisted of a cabinet with a screen that was designed to be attached to the music
instrument keys as well, and it was placed on top of the organ, and with various
mechanisms, the keyboard and the colours could be played together. It was to be
placed in front of a sunny window, or an electric light was to be used behind it,
to act as illumination (see Figure 1.9).
Bishop’s instruments were not trying to provide a rational proof for a colour-
tone analogy that could be universally applied, as Castel had hoped in his
elaborate scientific discussions of his ocular harpsichord, nor was its purpose
to demonstrate a colour-tone analogy theory. These instruments, rather, were
really trying to facilitate a more nuanced colour display and to create a colour
harmony. Colour was to be worked with in different ways, focusing on the
interaction of colour with materials such as coloured background screens, the
addition of an analogy of graduated colour and musical octaves, resulting in
an expanded use of colour beyond the mapping of a tone to a simple colour.
30 Maura McDonnell

FIGURE 1.9 One of Bainbridge Bishop’s colour organs discussed in his essay “A
Souvenir of The Color Organ, with some suggestions in regard to The
Soul of The Rainbow and The Harmony of Light” (1893).
Source: Public domain. https://rhythmiclight.com/1998/books/HarmonyOf Light.pdf

Bishop based all these studies and interactions with colour on a consideration
of music harmony and in particular what he mentions many times: “the sensa-
tion of musical tone.”101 He found solutions in his colour studies for getting this
sensation, as it were, via colours. Bishop had noted that just a singular colour
itself did not give the sensation of tone; he found, rather, that what works well
for tone sensation (as well as sense of musical harmony) is to use “combinations
of colours softened by gradations into neutral shades or tinted greys, with the
edges of the main colours blending together, or nearly together, rendered the
sensation of musical chords very well indeed.”102 Bishop came to this solution
by observing the behaviour of the spectrum colours and prisms against various
neutral backgrounds and also through observations of colours in nature. These
inspired this association of a background colour and musical keys. He noted
that the edges of the main colours in the spectrum blended together, so he
also wanted to emulate these crossover blendings at the edges of colours in his
instrument designs. Two of the instruments have some documentation about
them, one an essay written by Bishop himself in 1893, titled “A Souvenir of The
Color Organ, with some suggestions in regard to The Soul of The Rainbow
and The Harmony of Light,” the other a patent filed with the US Patent Office
in 1877103 that provides similar schematic diagrams and descriptions for another
 Inventing Instruments 31

colour organ. Alas, he also explains, that the three main instruments he built
were all burned down.
Bishop was very concerned at first to explore a harmony of colours in his work
by both “applying the intervals and harmony of music” and the “idea of painting
music.”104 He did his colour studies research, and like other inventors, he devised
a unique colour–to–musical tone scale. He agreed with the seven main colours of
the spectrum, but also added in the colours for the semi-tones. He thus devised
his own version of a chromatic colour-tone analogy scale of colours. He provides
a snippet of a score denoting his scale of colours in his essay. He focused especially
in demarcating the colour mixtures, as a relationship between previous and next
music tones and their equivalent colours, such as the first three colours in cor-
respondence with the first three notes of the chromatic music scale starting on C
is Red, Orange-red, Orange. Bishop also wanted to find a way to work colours
in their correspondence with the octaves of the music scale. For the octaves, he
used both darkness to lightness of colour for the mapping to the various pitches
across the ranges and deployed volume and depth for how much room the colour
would occupy depending on its place in the equivalent of the musical octave; for
example, he writes: “a lighter red for the upper C of the octave, and doubling
the depth and volume of colour in each descending octave.”105 In the instrument
discussed in the essay, he applied these analogies of depth and volume to variable
sizes of coloured glass, whose colours match his colour-tone analogy scale. Bishop
spent a great deal of time working with the nuances of a musical key, starting out,
at first, with a preference for C major, but then moving more towards A minor.
In his earlier design, he provided two screens that could be used alternatively for
the two keys of C major and A minor. The red screen was to be the backdrop for
C major and the yellow one for A minor.
What is really noteworthy about Bishop’s invention is that there is evidence
that his instruments were actually constructed and that they were of interest to
new audiences. The showman P. T. Barnum exhibited one of Bishop’s colour
organ instruments in the Barnum Museum in New York in 1881.106 Unfor-
tunately, no original instruments have survived, but the patent, his essay and
several newspaper articles provide us with a lot of information about how he saw
the instrument contributing to an art of colour light music.

Alexander Wallace Rimington: Colour-Organ

Alexander Wallace Rimington (1854–1918) was an artist, inventor and profes-
sor of fine arts at Queens College in London, UK. He was also interested in the
subject of colour-music and in identifying the principles by which colour-music
compositions could be produced, as well as the emotional inf luences of colour
and sound. Rimington, in other words, conceived of colour-music as being an
important new type of art activity in its infancy and that there was a need for a
colour-music art. In this regard, he saw his role in this art as of pivotal impor-
tance, in that he could devise instruments “for bringing it into practical being
32 Maura McDonnell

and to develop, through a long series of experiments, some of the principles

upon which Colour-Music compositions can be produced.”107
Like inventors before him, therefore, Rimington set about inventing instru-
ments to make possible a demonstration of a colour-music. He focused on the
element of time for colour, referring to it as “a mobile colour,” and saw this as
taking place in the presentation of gradations of colour and not just in the static
appearance of a single colour. The instrument design was thus to be based on his
own systematic colour-tone analogies and colour-music principles and had the
aim of facilitating the mobile colour. Rimington writes:

Of special interest to artists is the introduction of the element of time into

the gradated effects of colour by means of mobile colour instruments . . .
when colour is produced by the colour-organ under conditions of grada-
tion, whether it be towards intensification or of weakening of the colour,
it grows, or wanes, under the eye of the spectator and can be quickened, or
retarded, at the will of the executant.108

FIGURE 1.10 Alexander Wallace Rimington, Colour-Organ.

Source: Public Domain. https://en.wikipedia.org/wiki/Alexander_Wallace_Rimington#/media/File:
Rimington’s_Colour-Organ_2.jpg
 Inventing Instruments 33

Rimington constructed several colour-organs and devised a variety of methods

for the performer (executant) to perform mobile colour. His first colour-organ
instrument was built in 1893. A keyboard like that of the piano or organ is given
to the performer to play the mobile colour. Various control features are also
added to allow for more control of the colour in performance by means of stops
(see Figure 1.10). In his subsequent 1912 publication “Colour-Music: The Art of
Mobile Colour,” Rimington gives us an account of his life’s work ideas related
to colour-music and an explanation of the principles of the colour-organ instru-
ment. He explains the principles underpinning the design and construction of
the instrument, which is to be done:

a By dividing the spectrum-band similarly to the musical octave

b By giving the colour-organ a keyboard like that of the organ or the pianoforte
c By arranging for the general control of the whole keyboard by means of
stops somewhat like those of the organ
d By providing higher and lower octaves in the colour scale of relatively paler
and deeper intensity, somewhat analogous, though not strictly correspond-
ing to, the higher and lower octaves of the musical scale, though, of course,
in the colour the wavelengths remain the same.109

The spectrum band of colour is divided similarly to the musical octave. Higher
and lower octaves of a colour scale are devised based on comparable adjustments
of the hue to either its paler or deeper intensity. Rimington acknowledges that the
wavelength of each colour remains the same across the higher and lower octaves
of colour, based on varying its intensity, which is unlike music, where the wave-
lengths at higher and lower octaves are different. What was of more importance to
him, however, was not to seek a strict correspondence of colour and music on the
basis of a strict physical analogy. He wanted rather to provide a means and method
to facilitate mobile colour based on musical notation and keyboard devices. He
explains how he works with a spectrum band of colour in his invention:

The complete spectrum-band, greatly lengthened by sufficient distance,

was thrown upon the screen by two bisulphite of carbon prisms – the
source of white light being an unenclosed arc-lamp. An opaque diaphragm
was then interposed close to the screen with an extremely narrow slit in it,
and the fine slice of colour rays passing through it was made to correspond
in position on the spectrum-band and approximately as to its rate of vibra-
tion to the notes of the musical scale in their relative intervals. The narrow
ribbon of colour, thus cut off from the spectrum-band, was then accurately
matched and apparatus design for producing it in larger quantities in the
colour-organ and placing it under the control of the corresponding musical
note upon its keyboard. The keys of the colour-organ were then connected
with suitable mechanism for allowing these several colours to appear on
the screen as soon as the key is depressed.110
34 Maura McDonnell

Rimington did not want to invent a new colour-music score, as the music score
itself provided the precedents of laying down the foundation for a colour-music
art. The musical score system was a convenient one to base the colour score on.
Instead, Rimington devised an adaptation of music notation to be used, since
the notation method for the colour-music was to be played on his colour-organ,
where the duration of a colour and the colour-to-tone analogy could be easily
performed. From the music score, the time value of notes, the musical key-
board, diminuendo, crescendo and allegro were assigned to his colour scale. The
colour-organ inventions were thus designed to be able to make this colour scale
work in relation to its colour-music score.
Rimington – in his colour-organ inventions, his unique approach to working
with the spectrum in the mechanics of his invention, his attention to colour-
music as an art and his thinking through of a method of composition and devis-
ing a colour-music score – presented the ultimate statement on the importance of
this new art, which he named colour-music. Problems to be solved were not just
about proving whether a colour-tone analogy was valid but also focussed on how
best one could design a system that can enable colour and music to work side by
side. Rimington attended to strategies for making such a system. His publica-
tion, “Colour-Music: The Art of Mobile Colour,” was written many years after
he had worked with his inventions and theories, and it is a statement on how
much importance he attributed to this new art field. He mentions in his writing
that he wrote the publication because of the widespread interest in the subject of
colour-music and to address and respond to the many enquiries he had person-
ally received asking him about colour-music and his thoughts and ideas on it.111
With Rimington, the possibilities for exploring the putting together of a mobile
colour with music by devising a systematic approach had just begun, paving the
pathway for future artists, musicians and inventors to explore this new territory
of colour and visual performance tightly aligned to the musical expression in the
music performance.

Conclusion
The colour-tone analogy was investigated and applied to matters in natural phi-
losophy and early modern science from antiquity to the 18th century. The first
foray, however, into building an instrument to demonstrate this colour-tone
analogy was attempted by Louis-Bertrand Castel. His imaginings for what was
possible, nonetheless, did not sit well with the scientific community because
in his efforts to build an instrument that would demonstrate the geometry of a
colour–to–musical tone connection, he had incorporated a new type of value,
that of an artistic value. The mechanical means to put sound and colour together
continued to be laborious and difficult. Yet despite this, scientific inventors
did try, and many sought to create a quantifiable relation between colours and
sounds, what we could call a mapping approach today. For many, the initial
focus of the invention was to demonstrate the scientific reasoning about the
 Inventing Instruments 35

analogy of colour and (musical) sound, and so the focus was on the plans for such
a device and not necessarily on the building of the instrument, as was the case
with Castel. However, Castel’s invention did suggest other artistic possibilities
to him, and along with subsequent theorists’, they were not always overly con-
cerned with an exact quantifying of colour according to quantities from music,
but were interested in the realm and promise of artistic possibilities for a new
type of aesthetic experience that could be obtained from the combining of visual
and auditory sensation. Many inventors refer to a new type of colour-music art
that seemed to be full of promise. The new mobility of colour and the element
of time that became afforded to colour combinations facilitated by these instru-
ments suggested many future aesthetic possibilities and opened up new fields of
investigations and many technical and artistic pathways. These developments
would continue to take hold in the 20th century and on into our audio-visual
performance culture of today where we continue to chase ways to connect the
visual with music through a combination of technical and technological means.

Notes
1 Jörg Jewanski, “Color-Tone Analogies,” in See This Sound: Audiovisuology: Compendium:
An Interdisciplinary Survey of Audiovisual Culture 2010 (Vienna: Ludwig Boltzmann,
2010), 339–349.
2 Dieter Daniels and Sandra Naumann, See This Sound: Audiovisuology: Compendium: An
Interdisciplinary Survey of Audiovisual Culture (Vienna: Ludwig Boltzmann, 2010), 8.
3 Morris Kline, Mathematics in Western Culture (London: Oxford University Press, 1953), 74.
4 Oliver Darrigol, “The Analogy between Light and Sound in the History of Optics
from the Ancient Greeks to Isaac Newton. Part 1,” Centaurus, 52(2), 2010, 119.
5 Eduard Jan Dijksterhuis, The Mechanization of the World Picture: Pythagoras to Newton
(Princeton, NJ: Princeton University Press, 1961), 6.
6 Ralph McInerny, The Logic of Analogy (The Hague: Martinus Huhoff, 1971), 3.
7 Oliver Darrigol, “The Analogy between Light and Sound in the History of Optics
from the Ancient Greeks to Isaac Newton. Part 1,” Centaurus, 52(2), 2010, 118.
8 Eduard Jan Dijksterhuis, The Mechanization of the World Picture: Pythagoras to Newton
(Princeton University Press, 1961), 6.
9 Morris Kline, Mathematics in Western Culture (London: Oxford University Press, 1953), 76.
10 Rolf G. Kuehni, “Development of the Idea of Simple Colors in the 16th and 17th
Centuries,” Color Research and Application, 32(2), 2007, 92.
11 Eileen Reeves, “Color by Numbers: The Harmonious Palette in Early Modern
Painting,” in The Language of Nature: Reassessing the Mathematization of Natural Phi-
losophy in the Seventeenth Century (Minneapolis: University of Minnesota Press, 2016),
https://manifold.umn.edu/read/untitled-7ca18210-217d-40f2-83fe-b0add1d84ede/
section/7e0954bc-3fc8-466c-84ce-2291ea796fc4.
12 Wilton Mason, “Father Castel and His Color Clavecin,” The Journal of Aesthetics and Art
Criticism, 17(1), 1958, 103.
13 Aristotle, On Sense and the Sensible, 353BC, http://classics.mit.edu/Aristotle/sense.1.1.html.
14 Richard Sorabji, “Aristotle, Mathematics, and Colour,” The Classical Quarterly, 22(2),
1972, 295.
15 Eileen Reeves quoting Aristotle in “Color by Numbers: The Harmonious Palette in
Early Modern Painting,” The Language of Nature, 2016, 233.
16 Oliver Darrigol, “The Analogy between Light and Sound in the History of Optics
from the Ancient Greeks to Isaac Newton. Part 1,” Centaurus, 52(2), 2010 and Oliver
36 Maura McDonnell

Darrigol, “The Analogy between Light and Sound in the History of Optics from the
Ancient Greeks to Isaac Newton. Part 2,” Centaurus, 52(2), 2010.
17 Rolf G. Kuehni, “Development of the Idea of Simple Colors in the 16th and 17th
Centuries,” Color Research and Application, 32(2), 2007, 92.
18 Rolf G. Kuehni, “Development of the Idea of Simple Colors in the 16th and 17th
Centuries,” Color Research and Application, 32(2), 2007, 92.
19 Morris Kline, Mathematics in Western Culture (London: Oxford University Press,
1953), 107.
20 Morris Kline, Mathematics in Western Culture (London: Oxford University Press,
1953), 106.
21 H.L. Resnikoff, “Differential Geometry and Color Perception,” Journal of Mathematical
Biology, 1, 1974, 99.
22 H.L. Resnikoff, “Differential Geometry and Color Perception,” Journal of Mathematical
Biology, 1, 1974, 99, quoting Descartes, 1638.
23 Nejc Sukljan, “Renaissance Music between Science and Art: The Case of Gioseffo
Zarlino,” Musicological Annual, 56(2), 2020, 183–206.
24 Niels Hutchison, “3. Classic Codes: Colour,” Colour Music in the New Age: de-mystifying
De Clario, www.colourmusic.info/colour.htm.
25 Nejc Sukljan, “Renaissance Music between Science and Art: The Case of Gioseffo
Zarlino,” Musicological Annual, 56(2), 2020, 199.
26 Rolf G. Kuehni, “Development of the Idea of Simple Colors in the 16th and 17th
Centuries,” Color Research and Application, 32(2), 2007, 95.
27 Niels Hutchison, “3. Classic Codes: Colour,” Colour Music in the New Age: de-mystifying
De Clario, www.colourmusic.info/colour.htm.
28 Maarten Franssen, “The Ocular Harpsichord of Louis-Bertrand Castel,” Tractrix 3,
1991, 19.
29 Jörg Jewanski, Colour Tone Analogies,https://blog.tomaskorinek.com/color-tone-analogies/.
30 Kircher quoted in Maarten Franssen, “The Ocular Harpsichord of Louis-Bertrand
Castel,” Tractrix, 3, 1991, 19.
31 Kircher quoted in Maarten Franssen, “The Ocular Harpsichord of Louis-Bertrand
Castel,” Tractrix, 3, 1991, 19.
32 Patricia Fara, “Newton Shows the Light: A Commentary on Newton (1672) ‘A Let-
ter . . . Containing His New Theory about Light and Colours . . . .,” Philosophical
Transactions of the Royal Society, 2015, https://royalsocietypublishing.org/doi/10.1098/
rsta.2014.0213.
33 Richard S. Westfall, “The Development of Newton’s Theory of Color,” Isis, 53(3),
September 1962, 339–358.
34 Patricia Fara, “Newton Shows the Light: A Commentary on Newton (1672) ‘A Let-
ter . . . Containing His New Theory about Light and Colours . . . .,” Philosophical
Transactions of the Royal Society, 2015, https://royalsocietypublishing.org/doi/10.1098/
rsta.2014.0213.
35 Sir Isaac Newton, “The First Book of Opticks. Part 1 (1718),” The Newton Project,
2009, www.newtonproject.ox.ac.uk/view/texts/diplomatic/NATP00045.
36 Patricia Fara, “Newton Shows the Light: A Commentary on Newton (1672) ‘A Let-
ter . . . Containing His New Theory about Light and Colours . . . .,” Philosophical
Transactions of the Royal Society, 2015, https://royalsocietypublishing.org/doi/10.1098/
rsta.2014.0213.
37 John Gage, Colour in Art (London: Thames & Hudson world of art, 2006), 7.
38 Renzo Shamey and Rolf G. Kuehni, Pioneers of Color Science (Switzerland: Springer,
2020), 101.
39 Renzo Shamey and Rolf G. Kuehni, Pioneers of Color Science (Switzerland: Springer,
2020), 100.
40 “[O]r in the string, which are afterwards, by different stoppings and strikings produced;
which string (by the way) is a pretty representation of the shape of a refracted ray to the
 Inventing Instruments 37

eye; and the manner of it may be somewhat imagined by the similitude thereof: for the
ray is like the string, strained between the luminous object and the eye, and the stop or
fingers is like the refracting surface, on the one side of which the string hath no motion,
on the other a vibrating one,” Hooke quoted in Olivier Darrigol, 2020, 86.
41 Thomas L. Hankins, “The Ocular Harpsichord of Louis-Bertrand Castel: Or, the
Instrument That Wasn’t,” Osiris, 2nd Series, 9, 1994, 144.
42 Thomas L. Hankins and Robert J. Silverman, Instruments and the Imagination (Princeton,
NJ: Princeton University Press, 1995), 74–75.
43 Thomas L. Hankins and Robert J. Silverman, Instruments and the Imagination (Princeton,
NJ: Princeton University Press, 1995), 75.
44 Olivier Darrigol, A History of Optics: From Greek Antiquity to the Nineteenth Century
(Oxford: Oxford University Press, 2012), 108.
45 Olivier Darrigol, A History of Optics: From Greek Antiquity to the Nineteenth Century
(Oxford: Oxford University Press, 2012), 91.
46 Penelope Gouk, quoted in Thomas L. Hankins, “The Ocular Harpsichord of Louis-
Bertrand Castel: Or, the Instrument That Wasn’t,” Osiris, 2nd Series, 9, 1994, 237.
47 Olivier Darrigol, A History of Optics: From Greek Antiquity to the Nineteenth Century
(Oxford: Oxford University Press, 2012), 94.
48 Wilton Mason, “Father Castel and His Color Clavecin,” The Journal of Aesthetics and Art
Criticism, 17(1), 1958, 104.
49 First Book of Opticks, Part 2, PROB. III. PROB. 1, EXPER. VII and this diagram
is added in the 1716 revised edition of Opticks, www.newtonproject.ox.ac.uk/view/
texts/diplomatic/NATP00046.
50 First Book of Opticks, Part 2, PROB. III. PROB. 1, EXPER. VII.
51 First Book of Opticks, Part 2, PROB. III. PROB. 1, EXPER. VII.
52 David Briggs, Part 7: The Dimensions of Hue in the Dimensions of Color, www.huevalue-
chroma.com/071.php.
53 John Gage, Colour in Art (London: Thames & Hudson World of Art, 2006), 34.
54 H.L. Resnikoff, “Differential Geometry and Color Perception,” Journal of Mathematical
Biology, 1, 1974, 101.
55 John Gage, Colour in Art (London: Thames & Hudson, 2006), 36.
56 Olivier Darrigol, A History of Optics: From Greek Antiquity to the Nineteenth Century
(Oxford: Oxford University Press, 2012), 97.
57 Newton quoted in Renzo Shamey and Rolf G. Kuehni, Pioneers of Color Science
(Switzerland: Springer, 2020), 103–104.
58 Olivier Darrigol, A History of Optics: From Greek Antiquity to the Nineteenth Century
(Oxford: Oxford University Press, 2012), 98.
59 Renzo Shamey and Rolf G. Kuehni, Pioneers of Color Science (Switzerland: Springer,
2020), 103–104.
60 Olivier Darrigol, A History of Optics: From Greek Antiquity to the Nineteenth Century
(Oxford: Oxford University Press, 2012), 117.
61 For intricate accounts of how Castel’s ocular harpsichord was received in his con-
temporary time, see Maarten Franssen, “The Ocular Harpsichord of Louis-Bertrand
Castel,” Tractrix, 3, 1991.
62 Wilton Mason, “Father Castel and His Color Clavecin,” The Journal of Aesthetics and Art
Criticism, 17(1), 1958, 107.
63 Maarten Franssen, “The Ocular Harpsichord of Louis-Bertrand Castel,” Tractrix, 3,
1991, 21.
64 Wilton Mason, “Father Castel and His Color Clavecin,” The Journal of Aesthetics and Art
Criticism, 17(1), 1958, 9.
65 Diagram from Louis-Bertrand Castel, L’optique des couleurs: fondée sur les simples observa-
tions & tournée sur-tout a la pratique de la peinture, de la teinture, & des autres arts coloristes
(Paris: Briasson, 1740), https://archive.org/details/gri_c00033125008507283/page/
n441/mode/1up.
38 Maura McDonnell

66 Louis-Bertrand Castel, 1735, quoted in Maarten Franssen, “The Ocular Harpsichord

of Louis-Bertrand Castel,” Tractrix, 3, 1991, 25.
67 Louis-Bertrand Castel quoted in Wilton Mason, “Father Castel and His Color Clave-
cin,” The Journal of Aesthetics and Art Criticism, 17(1), 1958, 111.
68 Maarten Franssen, “The Ocular Harpsichord of Louis-Bertrand Castel,” Tractrix, 3,
1991, 25.
69 Maarten Franssen, “The Ocular Harpsichord of Louis-Bertrand Castel,” Tractrix, 3,
1991, 21.
70 Maarten Franssen, “The Ocular Harpsichord of Louis-Bertrand Castel,” Tractrix, 3,
1991, 19, references Mercure de France, November 1725, 2552–2577.
71 Unknown, Explanations of the Ocular Harpsichord Upon Shew to the Public (London: S.
Hooper and A. Morley, 1757).
72 See Maarten Franssen, “The Ocular Harpsichord of Louis-Bertrand Castel,” Tractrix, 3,
1991, and Wilton Mason, “Father Castel and His Color Clavecin,” The Journal of Aes-
thetics and Art Criticism, 17(1), 1958 and Thomas L. Hankins, “The Ocular Harpsichord
of Louis-Bertrand Castel: Or, the Instrument That Wasn’t,” Osiris, 2nd Series, 9, 1994,
141–156.
73 Louis-Bertrand Castel, “Clavecin pour les yeux, avec l’art de peindre les sons, et toute
sortes de pieces de musique Lettre écrite de Paris le 20 Fevrier 1725 parle R.P.Castel,
Jesuite a M.Decourt, a Amiens,” Mercure de France, November 1725, 2552–2577 refer-
enced in Thomas L. Hankins, “The Ocular Harpsichord of Louis-Bertrand Castel: Or,
the Instrument That Wasn’t,” Osiris, 2nd Series, 9, 1994.
74 Thomas L. Hankins, “The Ocular Harpsichord of Louis-Bertrand Castel: Or, the
Instrument That Wasn’t,” Osiris, 2nd Series, 9, 1994, 150.
75 Louis-Bertrand Castel quoted in Wilton Mason, “Father Castel and His Color Clave-
cin,” The Journal of Aesthetics and Art Criticism, 17(1), 1958, 109.
76 See both Thomas L. Hankins, “The Ocular Harpsichord of Louis-Bertrand Castel: Or,
the Instrument That Wasn’t,” Osiris, 2nd Series, 9, 1994, 150 and Maarten Franssen,
“The Ocular Harpsichord of Louis-Bertrand Castel,” Tractrix, 3, 1991, 8 and 13.
77 Thomas L. Hankins, “The Ocular Harpsichord of Louis-Bertrand Castel: Or, the
Instrument That Wasn’t,” Osiris, 2nd Series, 9, 1994, 15.
78 Mason explains that ‘the whole theory was developed at length in the last six volumes
of the Journaux de Trévoux of 1735, under the title “Nouvelles experiences d’optique et
d’acoustique, in Wilton Mason, “Father Castel and His Color Clavecin,” The Journal of
Aesthetics and Art Criticism, 17(1), 1958, 107.
79 Louis-Bertrand Castel, Optique des couleurs, fonde é sur les simples observations, et tourne é
surtout a’ la pratique de la peinture, de la teinture et des autres arts coloristes (Paris: Briasson,
1740), https://archive.org/details/gri_c00033125008507283/page/n441/mode/1up.
80 Wilton Mason, “Father Castel and His Color Clavecin,” The Journal of Aesthetics and Art
Criticism, 17(1), 1958, 108.
81 Louis-Bertrand Castel, Esprit, saillies, et singularites du pere Castel (1763). As quoted by
A. Klein, Colour-Music: The Art of Light (London: Lockwood, 1926), 184.
82 Wilton Mason, “Father Castel and His Color Clavecin,” The Journal of Aesthetics and Art
Criticism, 17(1), 1958, 115.
83 Wilton Mason, “Father Castel and His Color Clavecin,” The Journal of Aesthetics and Art
Criticism, 17(1), 1958, 115–116.
84 In an online collection of Saint-Aubin’s drawings, descriptions and curatorial commen-
tary provide us with a more detailed description of the mechanism of this instrument:
A white haired cleric dressed in a black cassock and a four-peaked biretta with a
tassel on top sits at a slim keyboard instrument with cork-screw legs. He is shown
in right profile. While his hands press down on keys from the lower of its two
keyboards, his left foot pumps a long pedal to which a pulley system has been
attached. The pulley system, which passes through the body of the instrument,
appears to activate the release of liquid from a large syringe mounted on top of its
 Inventing Instruments 39

soundboard. The arc of water squirted by the syringe is traced with a dotted line: it
splashes down onto the face of the seated man. He looks straight ahead at a line of
seven upright planks of various heights inserted into the instrument. Each is differ-
ently coloured and inscribed with one of the sol-fa syllables. Behind them on the
soundboard of the instrument are seven mushroom-shaped pads. A painter’s pallet,
bowsaw, scroll, musical score, set-square and measuring stick are gathered at his
feet. Retrieved from: https://waddesdon.org.uk/the-collection/item/?id=17198.
85 Retrieved from: https://waddesdon.org.uk/the-collection/item/?id=17198.
86 See McDonnell’s discussion of the relevance of Castel for visual music in Maura
McDonnell, Finding Visual Music in Its Twentieth Century History (PhD thesis, Trinity
College, 2020), 20–29.
87 Louis-Bertrand Castel, 1726 quoted in Maarten Franssen, “The Ocular Harpsichord of
Louis-Bertrand Castel,” Tractrix, 3, 1991, 23.
88 Wilton Mason, “Father Castel and His Color Clavecin,” The Journal of Aesthetics and Art
Criticism, 17(1), 1958, 112.
89 Louis-Bertrand Castel, 1725 quoted in Wilton Mason, “Father Castel and His Color
Clavecin,” The Journal of Aesthetics and Art Criticism, 17(1), 1958, 112.
90 Maarten Franssen, “The Ocular Harpsichord of Louis-Bertrand Castel,” Tractrix, 3,
1991, 23.
91 Maarten Franssen, “The Ocular Harpsichord of Louis-Bertrand Castel,” Tractrix, 3,
1991, 37.
92 Maarten Franssen, “The Ocular Harpsichord of Louis-Bertrand Castel,” Tractrix, 3,
1991, 38.
93 See museum of imaginary musical instruments. Curators, Deirdre Loughbridge and
Thomas Patteson, 2013, http://imaginaryinstruments.org/ocular-harpsichord/.
94 See museum of imaginary musical instruments. Curators, Deirdre Loughbridge and
Thomas Patteson, 2013, http://imaginaryinstruments.org/ocular-harpsichord/.
95 Maarten Franssen, “The Ocular Harpsichord of Louis-Bertrand Castel,” Tractrix, 3,
1991, 38.
96 Maarten Franssen, “The Ocular Harpsichord of Louis-Bertrand Castel,” Tractrix, 3,
1991, 35.
97 https://gallica.bnf.fr/ark:/12148/bpt6k15124383/f305.item and https://gallica.bnf.fr/
ark:/12148/bpt6k15124383/f298.double.
98 William Hooper, Colorific Music in Rational Recreations, 1782, 162–168.
99 William Hooper, Colorific Music in Rational Recreations, 1782, 162–168.
100 William Hooper, Colorific Music in Rational Recreations, 1782, 162–168.
101 Bainbridge Bishop, A Souvenir of the Color Organ, with Some Suggestions in Regard to the
Soul of the Rainbow and the Harmony of Light, with Marginal Notes and Illuminations by the
Author (New York: The De Vinne Press, 1893).
102 Bainbridge Bishop, A Souvenir of the Color Organ, with Some Suggestions in Regard to the
Soul of the Rainbow and the Harmony of Light, with Marginal Notes and Illuminations by the
Author (New York: The De Vinne Press, 1893).
103 Bainbridge Bishop, Patent no. 186,298 in Michael Betancourt, Visual Music Instrument
Patents (Rockville, MD: Wildside Press LLC, 2004).
104 Bainbridge Bishop, A Souvenir of the Color Organ, with Some Suggestions in Regard to the
Soul of the Rainbow and the Harmony of Light, with Marginal Notes and Illuminations by the
Author (New York: The De Vinne Press, 1893).
105 Bainbridge Bishop, A Souvenir of the Color Organ, with Some Suggestions in Regard to the
Soul of the Rainbow and the Harmony of Light, with Marginal Notes and Illuminations by the
Author (New York: The De Vinne Press, 1893).
106 Ralph K. Potter, Excerpt from “New Scientific Tools for the Arts,” The Journal of Aesthetics
and Art Criticism, 10(2), 1951, www.centerforvisualmusic.org/library/CVMPotterexc.htm.
107 Wallace Rimington, Colour-Music: The Art of Mobile Colour (London: Hutchinson &
Co, 1912), vii.
40 Maura McDonnell

108 Wallace Rimington, Colour-Music: The Art of Mobile Colour (London: Hutchinson &
Co, 1912), 109–110.
109 Wallace Rimington, Colour-Music: The Art of Mobile Colour (London: Hutchinson &
Co, 1912), 47–48.
110 Wallace Rimington, Colour-Music: The Art of Mobile Colour (London: Hutchinson &
Co, 1912), 109–110.
111 Wallace Rimington, Colour-Music: The Art of Mobile Colour (London: Hutchinson &
Co, 1912), v.
2
MOVING TOWARDS THE
PERFORMED IMAGE (COLOUR
ORGANS, SYNESTHESIA AND
VISUAL MUSIC)
Early Modernism (1900–1955)

Steve Gibson

Introduction
Continuing on from the early Live Visual experiments with colour organs in the
18th and 19th centuries, as discussed in Chapter 1, this chapter explores the early
period of the 20th century in which artists began to experiment with new tech-
nologies available to them. Covering the startling expansion of Live Visual prac-
tices in the early 20th century, we focus on the new developments that helped
defined the visual as a performed medium. Both the creation of new instruments
engendered by advances in technologies and new forms that made use of idiom-
atic or expanded use of those technologies established a trajectory for the audio-
visual avant-garde over the course of the following 100 years. The development
of film as an artform, of both the narrative and experimental varieties, was also
key to the development of an aesthetic of audio-visual cross-fertilisation.
On the technological side, ideas stemming from an interest in synesthesia
began to inform the creation of both interfaces and individual pieces. Build-
ing from Rimington’s experiments with the colour organ in the late 19th cen-
tury, figures such as Scriabin began to formalise models for combining images
with sound. While the science surrounding the study of synesthesia was far
from established in this era, experiments derived from anecdotal descriptions
of sound-visual cross-modal perception were increasingly fruitful. Instruments
such as the luce (light keyboard) made the performance of Live Visuals possible,
and Scriabin’s development of a method for combining colour and tonality – in
part informed by his interest in synesthesia – began to establish a formal model
for matching sound and image.
With the development of both film theory and practice in the 1920s and
1930s, further techniques and forms were created that had a profound inf luence
on the evolution of Live Visuals. Oskar Fischinger, Len Lye and Norman McLar-
en’s experiments with drawn sound and image on film stock produced a startling
DOI: 10.4324/9781003282396-4
42 Steve Gibson

synchronicity between the visual and aural. Editing with a handwork and preci-
sion that was previously impossible, they established a new form of audio-visuals:
the visual music film. While both Fischinger and McLaren produced linear short
films, their techniques of image and sound matching have informed audio-visual
artists to the present day.
The chapter concludes with a detailed analysis of Lye and McLaren’s best-
known works, discussing their formal attributes and the implications for the later
development of Live Visuals and audio-visual art in general.

The Technology of Live Visuals in Early Modernism

The early modernist era was characterised by a startling advance in a number
of technologies that were utilised in all artforms. The development of film as
an artform in the early 20th century engendered advancements in camera and
projection technology that radically altered the way in which visual stories were
consumed, arguably displacing theatre as the dominant medium of performed
and/or recorded narrative. While film is obviously not necessarily a live artform –
though in its early inception it was partially live by virtue of the live music
often accompanying screenings – the idea of a constructed narrative, edited for
viewing over a ‘theatre-like’ time frame (i.e. 90 to 120 minutes) obviously had
profound implications for the development of most, if not all, technologically
produced media that followed it, including electronic and tape-based music,
video production, animation and ultimately Live Visuals. Theories of early
cinema, including Vsevolod Pudovkin’s (1893–1953) attentive observer model,
established a manner of rendering apparently disjunct scenes into a coherent
narrative:

The sum of the shooting script is divided into sequences, each sequence
into scenes, and, finally, the scenes themselves are constructed from a
whole series of pieces (script scenes) shot from various angles. . . . The film
technician, in order to secure the greatest clarity, emphasis, and vividness,
shoots the scene in separate pieces, and joining them and showing them,
directs the attention of the spectator to the separate elements, compelling
him to see as the attentive observer saw.1

It is hard to overstate the inf luence and reach of this idea for the development
of a technological means of rendering an audio-visual narrative. The attentive
observer model informs much of early film development and makes its way (often
unnoticed) into almost all forms of early modern narrative film. As a means of
general editing, it is not a stretch to state that it is the common method for edit-
ing narrative productions from the early 20th century to the present.
Film editing theories such as Pudovkin’s were clearly enabled by the devel-
opment of early film cameras and editing technologies, many of which had
been developed prior to Pudovkin, including by Edwin S. Porter (1870–1941)
 Moving Towards the Performed Image 43

at Edison in the early 20th century, among others. While the development of
film technology is obviously complex and beyond the scope of our discussion
here, there can be no doubt that much of the work in various audio-visual
forms was engendered by the advances in film technology in the early 20th
century. 2 More specifically, it is easy to trace many of the processes, effects
and techniques used in contemporary Live Visuals practice (e.g. luma keying,
colour correction, compositing) back to early film technologies, through video
production, and on to contemporary Live Visual software such as Modul8 and
Resolume.
As mentioned in the introduction earlier, much of the technology that advanced
in the early 20th century had an extremely explicit inf luence on the production
of visuals in the performance domain. As Chapter 1 has made clear, a number
of devices were proposed and realized prior to 1900, including most obviously
Louis-Bertrand Castel’s clavecin oculaire, or ocular harpsichord, and Alexander
Wallace Rimington’s colour-organ. An enormous number of variations on Rim-
ington’s colour-organ were developed in the early 20th century. As described by
Teun Lucassen, some devices from this period included:

Preston Millar’s, Chromola, 1911. Twelve differently colored lights were

operated by a keyboard of 15 keys (the first three lights were repeated on
the remaining keys).
Thomas Wilfred, Clavilux, 1922 . . . this device did not make any actual
music, but it produced imagery solely based on light and color. The home
version of the Clavilux (Home Clavilux or Clavilux Junior) looked like an
old-fashioned TV set.
Alexander Lászlò, Farblichtmusik, 1925. His device consisted of a few
switches above his piano, controlling a few projection lights and a slide
projector lightning [sic] the stage above the piano. . . .
Oskar Fischinger, Lumigraph, late 1940s . . . this was one of the first
devices ever to use a touch screen. The touched place by the operator
was the only place to show light. This allowed the user to create his own
imagery.3

Some of these instruments were used sparingly at the time, while others, such
as those used by Alexander Lászlò (1895–1970, see Figure 2.1), were extremely
popular: “Beginning in the mid-1920s in Europe and especially in Germany con-
certs with audiovisual instruments were extremely popular and Lászlò became at
true celebrity.”4 Other instruments that were created in this era included Leon
Theremin’s Etherophone, Fredrick Bentham’s Light Console and Cecil Stokes’s
Auroratone. These and other devices are documented in great detail in Adriani
Abbado’s excellent Visual Music Masters.5
While most of these technologies no longer exist, or are at best preserved in
museums, their inf luence was far-reaching. The sheer number of visual perfor-
mance devices created in the early modern period points to the desire to facilitate
44 Steve Gibson

FIGURE 2.1 Alexander László’s colour light piano, consisting of a keyboard and small
projectors.
Source: From Graef, Otto A. 1926. Farblichtmusik. Neue Musik-Zeitung 47(17), 397. Public domain.
[email protected]

the performed image. The fact that so many of these instruments had musical
form (i.e. a keyboard) also illustrates the close-knit relationship between the
aural and the visual in these experiments. We can see resonances in the contem-
porary era of MIDI controllers and control surfaces that are used to simultane-
ously control audio and visuals in real time.

Live Visuals and Synesthesia in Early Modernism

“If synaesthesia represents the unity of the sense, the dream of synaesthesia is the
unification of the arts.”6
The study of synesthesia in the early 20th century had a profound effect on
how the matching of sound and visuals was achieved.

How does it feel to hear music in color, or to see someone’s name in color?
These are examples of synesthesia, a neurological phenomenon that occurs
 Moving Towards the Performed Image 45

when a stimulus in one sense modality immediately evokes a sensation in

another sense modality. Literally, “synesthesia” means to perceive (esthesia)
together (syn).7

The history of synesthesia and its inf luence on art (and science) are well doc-
umented elsewhere, including in Cretien van Campen’s excellent The Hidden
Sense: Synesthesia in Art and Science and in the Museum of Contemporary Art LA
book Visual Music: Synaesthesia in Art and Music Since 1900. For the purpose of
our discussion, we can rely on van Campen’s description of synesthesia earlier as
key to describing how audio-visual artists over the past 100 years conceptualised
the idea of joined sensory information in their work.
An early proponent of the concept of synesthesia as a tool for matching
sound and visuals was Russian composer Alexander Scriabin (1871–1915). Scri-
abin began imagining the mix of coloured lights within his symphonic works
as early as 1908.8 While in recent years a number of precise scientific studies
have confirmed the subjective experience of synesthesia as perceptually real,9 in
Scriabin’s era there were only anecdotal descriptions of the effects of synesthesia.
Therefore, despite the now-debunked claims that Scriabin experienced the con-
dition himself,10 any matching of sound to image was, at best, a simulation of the
effects of synesthesia for Scriabin.11

FIGURE 2.2 Scriabin’s colour scheme for the luce used in Prometheus: Poem of Fire.
Source: Creative Commons. https://en.wikipedia.org/wiki/Clavier_%C3%A0_lumi%C3%A8res#/
media/File:Scriabin-Circle.svg
46 Steve Gibson

For his 1910 work Prometheus, the Poem of Fire, Scriabin initially used a primi-
tive light keyboard called a tastiera per luce (or simply luce) to ‘perform’ the visual
material according to the relationship of musical key to colour, as shown in Fig-
ure 2.2. Colours were not specifically associated with notes, but rather particular
tonalities: the luce was used to define and amplify particular tonal relationships
in the music. For the initial performance in 1911 the luce part was scrubbed by
Scriabin due to its ineffectiveness in realisation. Later performances with the
luce in 1915 were a limited success and were never properly repeated in his life-
time. New performances have been attempted, including one presented at Yale
University in 2010.12 In this contemporary version a MIDI keyboard was used
instead of a luce and included some of the additional effects (lightning, fireworks,
etc.) that were specified in Scriabin’s later revised score but not used in the 1915
performances.
There is some argument to the merit of the visual aspect of Prometheus, as
summed up by Evan Norcross Flynn:

Consider these two arguments regarding use of the luce in performance:

In support, the luce both triggers and illuminates changes, filling in for
what the sound has disguised, in opposition, the lights appear dull in
contrast to the music due to what seems like arbitrarily chosen colors
which are dependent on mystic chord fundamentals. Does the luce mat-
ter? Do Scriabin’s colors matter? The majority of performers and schol-
ars would say that the music stands alone perfectly well. . . . However,
one cannot deny that the performance with the luce is a spectacular
feat.13

Regardless of the artistic merit of Scriabin’s use of Live Visuals in Prometheus,

there is no doubt of the historical impact of the use of both the luce as a visual
instrument and synesthesia as a model for sound-image matching. Parallel to
Scriabin, a number of painters, composers and filmmakers also delved into the
notion of a unification of the senses based on the simulation of synesthesia (many
of these are described in the aforementioned Visual Music: Synaesthesia in Art and
Music Since 1900), but Scriabin was arguably the most visible of the earliest mod-
ernist figures to endeavour to unearth the means, the technology and the form
for live audio-visual performance. If we look at the history of visual music from
Scriabin’s time to the present, there are a range of artists from Fischinger to alva
noto that owe a debt to Scriabin’s embryonic experiments with the technology
of Live Visual performance and his efforts to find analogues between aspects of
the aural and the visual.

Early Modernism, Synesthesia and Colour Music

Following on from Scriabin, a number of artists and musicians were interested in
the use of coloured light projections in a live performance situation:
 Moving Towards the Performed Image 47

Kandinsky’s theatre piece Der gelbe Klang (The yellow sound, 1911) included
elaborate stage directions for coloured lighting which assumed prime
importance in the painter’s plotless stage composition. Der gelbe Klang was
never produced but Kandinsky published his manuscript in Der Blaue Reiter
Almanach. Schoenberg incorporated comparably important colored-light
projections as well as hand-painted set designs, in the conception of his
opera Die glückliche Hand.14

In addition, the extraordinary number of composers and musicians from the

early to mid-20th century claiming to experience the effects of synesthesia
(occasionally somewhat dubiously) was remarkable: Debussy, Gershwin, Charles
Ives, Varèse, Messiaen, Schoenberg, Leon Theremin and, of course, Scriabin “all
appear to have possessed this faculty to one degree or another.”15 This interest in
synesthesia had the obvious effect of generating numerous colour music theories
as well as several pieces that explored notions of (visual) colour related to musical
parameters (not just notes/tonalities but also timbre, dynamics and other musical
aspects).
As Chapter 1 has already made clear, there were an enormous number of
theories of colour and sound tracing all the way from Pythagoras and Aristotle
to Newton and on to Rimington in the late 19th century. As previously out-
lined in this chapter, Scriabin created a colour scheme for his piece Prometheus
(see Figure 2.2). This was somewhat distinct from early colour-pitch schemes,
though there are some commonalities with Scriabin’s mappings and Bainbridge
Bishop’s colour scale from 1877 and Rimington’s scale from 1893 (for example,
all three begin with C matched to a shade of red).16 Other composers, including
Arnold Schoenberg (1874–1951), were interested in musical colour (often solely
associated with musical timbre) as well as the relation of musical elements to
visual colours. For his aforementioned Die glückliche Hand Schoenberg “designed
the sets, the costumes and the lighting. In the overture to the opera Schoenberg
designated an actual ‘colour crescendo’ between measures 124 and 153 from
“schwarch rötlich (pale red) to bläuliches Licht (bluish light).”17 Schoenberg was
also an accomplished painter so he had some visual advantage here. Schoenberg’s
usage of colour in relation to musical dynamics points to possibilities beyond the
simple matching of pitch to visual colour and suggests the more complex audio-
visual mappings that appear in the work of visual music artists discussed later in
this chapter.
Olivier Messiaen (1908–1992) is arguably the composer from the early to
mid-20th century with one of the best claims for actually experiencing synesthe-
sia. Certainly, Messiaen employed the effects of colour in his compositions (often
as simple colour analogies rather than lighting or visual effects). From an early
period in his life he claimed to see colours related to sound: “And I realized that
I also connected colours to sounds, but intellectually, not with the eyes. In fact,
when I hear or read music, I always see colour complexes in my mind that go
with the sound complexes.”18 A number of Messiaen’s works have direct colour
48 Steve Gibson

music descriptions, sometimes directly referred to in the titles (e.g. Chromochromie

from 1960) and at other times in the description of the musical ‘colours’ such as in
Trois petites liturgies de la présence divine for Ondes Martenot, Piano, Female Choir,
Percussion and String Orchestra (1945):

The music of TROIS PETITES LITURGIES is above all a music of

colours. The ‘modes’ that I use there are harmonic colours. Their juxta-
position and their superposition give: blues, reds, blues striped with red,
mauves and greys spotted with orange, blues spiked with green and circled
with gold, purple, hyacinth, violet, and the glittering of precious stones:
rubies, sapphire, emerald, amethyst – all that in draperies, in waves, in
swirling, in spirals, in interlaced movements. Each movement is assigned
to one ‘kind’ of [divine] presence. . . . These inexpressible ideas are not
expressed – they remain in the order of a dazzle of colours.19

Messiaen’s evocation of colour in Trois petites liturgies de la présence divine is exclu-

sively made in the musical domain (i.e. there is no visual element to the piece),
but it is striking in its description of colour to musical modes and also in the
description of musical elements in relation to coloured objects (i.e. “precious
stones”). Other works of Messiaen’s used actual visual representations of colours
in live performance: “Fête des belles eaux (1937) for six ondes martenots (early elec-
tronic instruments) was performed at the 1937 Paris World’s Fair while colored
lights were projected onto arabesques of water in a nearby fountain.”20
Other composers in the early to mid-20th century worked in colour music,
including Edgard Varèse (1893–1965, covered in detail in Chapter 3) and Leon
Theremin (1896–1993). Theremin was a self-described synesthete and was
renowned for the invention of the early electronic music instrument the Ther-
emin, but “he also invented the illumovox, a color-music device.”21 The illumivox
has been described by Theremin’s biographer as “a specially constructed device
in which a beam of light was projected through a rotating disk containing a strip
of gelatin tinted with a color spectrum.”22 Theremin used the illumovox in tan-
dem with the theremin in many performances, creating one of the first examples
of live performed audio-visuals using entirely electronic means, and arguably
pre-figuring later forms such as live audio-visual performance as described in
detail in Chapter 12.

Light, Environment, Transcendence

It is worth mentioning that live light experiments were continued in the early
20th century onwards in more direct and (often) very public ways by other art-
ists who may or may not have been specifically involved in the Western classical
music tradition. For example, American artist, architect and theosophist Claude
Bragdon (1866–1946) was interested in moving light performances out into the
public sphere, including in public outdoor venues. He organised a number of
 Moving Towards the Performed Image 49

these events in New York in the mid-1910s in order for light and sound to mix
to create “a transcendent experience”23 for the public. Bragdon was interested in
theosophy, which can roughly be defined as a quasi-spiritual, quasi-philosophical
movement interested in mystical experience. While the history of theosophy is
clearly beyond the scope of this book, it is important to note that many early
20th-century audio-visual artists (including the aforementioned Scriabin) were
interested in spiritual and mystical uses of audio-visual synchronisation, often
employing synesthetic means in order to heighten spiritual manifestations or to
evoke mystical experience. This is certainly an important aspect of much audio-
visual work and clearly connects with movements described in Chapter 3, such as
liquid visuals as practiced by groups such as Silver Wing Turquoise Bird.
Another key figure who was associated with Bragdon and light-based per-
formance was Danish artist Thomas Wilfred (1889–1968, Wilfred was a musi-
cian as well as a light artist). Wilfred designed his own specialist instrument
for light performance, called the clavilux (sometimes referred to as the lumia).
Wilfred considered his lumia quite distinct from a colour organ, as it was meant
perform light colours silently. He took this instrument on tour in the 1920s and
performed various light scores to classical performances, including to Nikolai
Rimsky-Korsakov’s Scheherezade. Wilfred continued development of the clavilux
in various forms in the 1920s and 1930s and composed multiple pieces for it,
which were generally made up of abstract patterns of light to be performed on
the lumia.24 Wilfred considered his work to be the ultimate expression of abstract
painting into an evolving, time-based medium, which we also can observe as a
key driver in the work of many of the visual music artists described next.

Visual Music
Visual music as a notion and artform quite literally burst into view in the early
20th century. Jeremy Strick goes so far as to argue that visual music exists as a
parallel (if under-recognised) modern artform:

Even if visual music is not a single mode through which music and visual
arts have interacted in the past Century, it is certainly the most consistent.
Indeed, the tradition of visual music might be said to be among the most
tenacious stylistic strains of the past one hundred years, continuing to find
new arenas for aesthetic exploration even as other, more famous move-
ments and styles eventually faltered.25

The idea of cross-fertilisation between sound and image has been explored by
figures as diverse as painters (Klee, Kandinsky), narrative filmmakers (Eisen-
stein), abstract filmmakers ( James and John Whitney, Jordan Belson), visual-
music filmmakers (Len Lye, Fischinger, McLaren) and composers (Messiaen,
Xenakis). That visual music still persists as a form is testament to the continued
vitality of the original notions explored in the early and mid-20th century.
50 Steve Gibson

Visual Music in Early Modernist Painting

Painting was the first artform to broadly explore the idea of visual music in direct
ways that have been well-preserved to the present. The term ‘visual music’ was
coined by Roger Fry (1866–1934) in 1912 as a description of post-impressionist
painting which “gave up all resemblance to natural form, and create a purely
abstract language of form – a visual music.”26 Painters, including Wassily Kan-
dinsky (1866–1944), were both inspired by music and worked intimately with
composers (Kandinsky considered modernist composer Arnold Schoenberg a
“kindred spirit”).27 Inspired by the musical concepts of rhythm and timbre, sev-
eral painters in the early 20th century sought to explore visual time (rhythm) and
colour (timbre) as primary concerns of their works. Kandinsky was also inspired
by the notion of synesthesia 28 and attempted to give visual form to existing musi-
cal forms in various paintings in the early 20th century, including such obviously
musically inspired works as Fuga (Fugue) from 1914.
Paul Klee (1879–1940) was another painter obviously inspired by music, and
he sought to more precisely formalise temporal and gestural aspects of music to
his paintings, including the ‘staccato’ (i.e. in musical terms: short notes with a
definite break between them) brushstrokes applied to paintings such as Memory
of a Bird (see Figure 2.3) from 1932.29 While a broader discussion of painting

FIGURE 2.3 Paul Klee, Swiss, 1879–1940, Memory of a Bird, 1932, Watercolour and
pencil on laid paper, 12–3/8 × 18–7/8 in. (31.4 × 47.9 cm).
Source: Norton Simon Museum, The Blue Four Galka Scheyer Collection. Accession Num-
ber: P. 1953.033. © Norton Simon Museum. Used by permission. www.nortonsimon.org/art/
detail/P.1953.033
 Moving Towards the Performed Image 51

and music cross-fertilisation is beyond the scope of this volume, there can be no
doubt that the genesis of visual music lies in the work of artists such as Kandin-
sky and Klee alongside other less well-known figures such as Marsden Hartley,
Robert Delaunay and Mikhail Matiushin.30
Painting was the first visual medium to recognise the possibilities inherent in
adapting musical concepts and applying them to the visual. Undeniably an obvi-
ous impediment to a more precise rendering of musical concepts (such as rhythm)
lay in the fixed time of the still painting. While paintings such as Mikhail Maty-
ushin’s (1861–1934) Painterly-Musical Construction of 191831 rendered an abstract
colour field with obvious rhythmic and metric structures that could be viewed
as visually representing changes in metre and tempo, there was no avoiding the
fact that the painting was fixed in time on a f lat plane. For a more precise visual
music to arise, there was a need to employ a visual medium that worked in time,
and film was the obvious choice.

Visual Music Film

“While Kandinsky could only suggest movement in a fragmented way through
repeated forms and lines, [Oskar] Fischinger was able to create an actual kinetic
art in which each motion sequence is complete.”32
The world of the filmmaker was one in which obvious formal synchronicities
could be made between the aural and the visual. Given the time-based nature
of both film and music, this was a seemingly inevitable development. Early
filmmakers such as Russian montage filmmaker Sergei Eisenstein (1898–1948)
had speculated on the relationship between the film’s visual material and its
soundtrack: his notion of contrapuntal montage, in which the film’s soundtrack
had a “non-naturalistic” relationship with the moving image linked through the
overall concept of montage, had obvious musical references with the melodic
counterpoint used in classical music from the Baroque era to the early modernist
period.33 However, it took a group of filmmakers in the 1920s to more precisely
realise the visual music film and to turn it into a specific artform that resonates
through to the present. The first filmmakers to explore this combination of
painting, film and music were “three artists in Berlin: Hans Richter, Viking
Eggeling and Walter Ruttmann.”34 These artists all used formal abstract paint-
ings and/or drawings rendered on film and accompanied by music in order to
create a new type of visual music film. This led to a number of experimental
visual music films, arguably the most inf luential of which were created by Oskar
Fischinger.

Oskar Fischinger
Oskar Fischinger (1900–1967) was an avant-garde visual music filmmaker
from Germany. “In the 1920s he used a wax-cutting machine he invented to
create animations that are still current today.”35 In this period he also hosted
52 Steve Gibson

multi-screen events that could be seen as precursors to multi-screen installations.

From 1927 to 1936 he lived and worked in Berlin and created a number of strik-
ing visual music films, including the series Film Studien (1929–32), which were
also released internationally. Prior to WWII he f led Germany for the United
States and worked in Hollywood, most famously working for Disney (which
ultimately proved disappointing for Fischinger, and he left after requesting his
name be removed from the credits of his work there).36 He ultimately tired of the
commercial Hollywood scene and concentrated on making a number of further
visual music films until 1947.37 The inf luence and impact of these films have
increased exponentially until the present.
Fischinger was deeply interested in music, and this had an obvious and
direct inf luence on his film output. Studie Nr 5 (from Film Studien) is some-
times referred to as one of the f irst examples of a music video, in which
hand-drawn images are animated in relative synchronisation with music. 38
Fischinger also produced a series of Ornament Sound Experiments (1932), hand-
drawn-shapes that were photographed and then printed on the soundtrack of
the film strip that then produced precisely timed sound tones. While these
experiments were limited (at least in length) at the time, the Ornament Sound
Experiments had a lasting inf luence on the generation of visual music film-
makers who came after Fischinger, including Len Lye and Norman McLaren.
The concept of using drawings for printing both the sound and the visual
elements onto the film strip allowed for a precision not possible in the con-
ventional film editing setups of the era. We can also draw a line from these
original experiments to the computer-assisted Live Visual performances of the
present, in which precise timing connections between the audio and video are
enabled by software.

Len Lye
Following on from Fischinger’s experiments, several visual music filmmakers
began to create their own works in the 1930s and 1940s. Chief among these
were Len Lye (1901–1980) and Norman McLaren (1914–1987). Both of these
filmmakers were inspired by Fischinger’s notion of using drawings printed on
the filmstrip, as well the precision with which he matched the drawn image with
the music soundtrack.
“Len Lye was among the first to intervene directly on the film with the most
varied techniques to produce animation.”39 Lye was initially a kinetic sculptor
who was raised in New Zealand. In the late 1920s he began to work on visual
music films.40 Lye expanded on Fischinger’s technique, updating it to include

a method of painting and staining directly on celluloid. . . . Films such as A

Colour Box (1935), Rainbow Dance (1936), Trade Tattoo (1937) and Swinging
the Lambeth Walk (1939) include rapid-fire nervous imagery . . . intricately
synchronized not to classical music but to jazz and Latin rhythms.41
 Moving Towards the Performed Image 53

Swinging the Lambeth Walk in particular was an inf luential film, as it established a
‘database’ of sorts for matching drawn images and particular instrumental melo-
dies. “In general Lye preferred to start from animated images and then searched
for music, sounds that were appropriate, in other words that were ‘in tune’ with
the visual part.”42
The British Council Film has made Swinging the Lambeth Walk publicly avail-
able on its Vimeo site.43 Throughout the film, one can see that the instrumental
melodies are precisely combined with visual shapes, often mimicking or matched
with the ‘groove’ of the music. For example, from 0:38 to 0:52 the drums are
rhythmically matched with red and black circles; the bass with red and black,
large, snake-like vertical lines; the guitar notes with thinner white and orange
horizontal lines; and the guitar chords with a series of coloured vertical lines.
These all come in rapid-fire succession, but because of the precision of the tim-
ing, they are easily recognized at exact cross-media matches, made even more
convincing because of the ways the lines and circles represent movement in tune
with the music. The piece continues with a wide palette of different visual forms
(as well as more realistic imagery), all in synchronization with the musical ele-
ments. Occasionally the visual field becomes polyphonic, when more than one
instrument is dominant, as in the section at 1:12 to 1:22 where the piano chords
are represented by moving coloured dots and the bass by green and red moving
vertical lines.
The genius of Swinging the Lambeth Walk lies in the exactitude with which
the visual material represents the musical elements: the precision of the rhythmic
editing is astonishing, even for viewers accustomed to computer-based audio-
visual editing. Working from a database of images to sounds, Lye establishes
an image-sound cross-modality with formal precision that has deep lessons for
contemporary live visualists. Certainly, one can see echoes of Lye’s visual music
techniques in contemporary Live Visual music such as the work of alva noto
(1965–). In the latter’s work, precise correspondences between the audio and
visual elements are controlled by programming between the MIDI software
Ableton Live and Live Visuals software TouchDesigner. Sound objects and musi-
cal control parameters are assigned precise visual analogues, with simultaneous
audio-visual control enabled by performance on a MIDI control surface and/or
keyboard. Therefore, the live performance in Noto’s work has an exactitude very
similar to precise editing technique used in Lye’s linear films (see Case Study 2 in
Chapter 12 for a detailed description of programmer Marcus Heckmann’s solu-
tions for audio-visual mapping in noto’s work).
While his work was interrupted by WWII, Lye continued to work on a num-
ber of visual music films post-war, and remarkably in 1957 he created a Live
Visual music performance with composer Henry Brant (1913–2008), entitled All
Souls Carnival:

The film was premiered in 1957 in the Carnegie Recital Hall in New
York. The music was performed live, with the images projected on a screen
54 Steve Gibson

behind the musicians. Lye and Brant decided to create the images and
the music separately so that the synchronization between them would
be a matter of chance. The meeting of the music with the images was
still striking because of the strong sense of affinity between the two
elements.44

Unfortunately, no complete document of this project exists, but it obviously

represents one essential step in the creation of live audio-visuals. The description
earlier makes it clear that All Souls Carnival does not have the precise audio-visual
timing of Lye’s linear films, since there was no attempt to synchronise the images
and music beforehand, but it instead represents a more general effort to match the
aesthetic qualities of the two artforms.
This loose audio-visual performance model, in which there was not a pre-
planned connection between the sound and the music, but rather an attempt
to create an “affinity between the two elements,”45 was continued in the 1960s
and 1970s by a number of artists (e.g. The Joshua Light Show, Single Wing Tur-
quoise Bird, etc.) working with coloured oils on overhead projectors, as well as a
number of lighting effects (see Chapter 3 for a further discussion of this). This is
also obviously an approach that many Live Visual artists (especially VJs) have fol-
lowed in their work, including the Swiss VJ collective Scheinwerfer, as discussed
in Case Study 1 in Chapter 12. The idea of partially planned Live Visuals loosely
matched with live music (but not precisely mapped) has natural antecedents in
works such as this piece. More broadly, the general inf luence of Lye, and in par-
ticular his use of both precise and aesthetic mapping of image to sound, and his
general interest in popular and experimental (rather than classical) music, deeply
resonates with the general approach to Live Visuals in the present.

Norman McLaren
Another significant visual music filmmaker to emerge in the early to mid-20th
century was Scottish-Canadian Norman McLaren (1914–1987): “Of Scottish
origin, McLaren moved first to London, then on to the United States and finally
to Canada, where he permanently settled and worked at the National Film Board
of Canada.”46 McLaren was heavily inf luenced by Fischinger’s technique of pho-
tos of drawings being inscribed directly on the film strip (and the soundtrack),
and he produced an enormous range of work from the abstract to the quasi-
narrative, including such films as Dots (1940), Begone Dull Care (1949), Blinkity
Blank (1955) and Synchromy (1971), among many others.
In general McLaren worked by drawing on a 35mm film strip, at times also
drawing on the film’s sound strip (e.g. Dots) and at other times using a primarily
composed soundtrack (e.g. Blinkity Blank). Dots (available on the National Film
Board of Canada’s Vimeo channel)47 is an excellent introduction to his work,
as the piece is short-form and demonstrates his mastery of hand-drawn audio-
visuals created in perfect rhythmic synchronisation. In this film all audio-visual
 Moving Towards the Performed Image 55

elements were hand-drawn directly on the film strip. McLaren described his
technique for creating this piece as follows:

I draw a lot of little lines on the sound-track area of the 35-mm. film.
Maybe 50 or 60 lines for every musical note. The number of strokes to
the inch controls the pitch of the note: the more, the higher the pitch; the
fewer, the lower is the pitch. The size of the stroke controls the loudness: a
big stroke will go “boom,” a smaller stroke will give a quieter sound, and
the faintest stroke will be just a little “m-m-m.” A black ink is another way
of making a loud sound, a mid-gray ink will make a medium sound, and a
very pale ink will make a very quiet sound. The tone quality, which is the
most difficult element to control, is made by the shape of the strokes. Well-
rounded forms give smooth sounds; sharper or angular forms give harder,
harsher sounds. Sometimes I use a brush instead of a pen to get very soft
sounds. By drawing or exposing two or more patterns on the same bit of
film I can create harmony and textural effects.48

Even from the perspective of the early 21st century Dots is a remarkable achieve-
ment, showing a complete mastery of visual music film, in all areas. Even more
remarkable, is that in 1940 the piece effectively pre-figures electronic music,
some 10 to 15 years before studios began working on electronically generating
sound, and 2 to 5 years before the tape-based sound editing and manipulation of
musique concrète was developed by Pierre Schaeffer and others.
After the credits end at 0:53, the film begins simply by outlining blue dots of
various sizes against a red background. The dots move from small and fast circles
towards larger and more irregularly shaped blobs. The pitch of generated sounds
becomes lower as the dots become larger (as described by McLaren earlier). The
film has a rhythmic regularity to it that can only be the result of precise timing
of the frames, so that the dots produce an almost dance-like feel (without becom-
ing too rigid). Each dot is accompanied by a particular sound, again exactly as
described by McLaren. The frequency of the dots increases at 1:09, and the visual
and music field become increasingly polyphonic, while at the same time main-
taining a relatively consistent rhythm. At 1:20 extremely fast dots appear, creating
32nd notes in the sound. Throughout, the dots dance around the screen, creating
almost realistic shapes, more imagined than precise, all ‘in tune’ with the sound
world. At 1:30 very large blobs appear, producing very bass tones, that have a sim-
ilar quality to triangle wave bass notes on a synthesizer. At 1:40 the internal tempo
increases and the dots appear almost as a series of consecutive 16th notes. The
visual field becomes completely polyphonic, but each shape and sound are easy to
follow due to the precise nature of the audio-visual synchronisation and the care
in which the visual field is addressed. At 2:00 the rhythm becomes more irregu-
lar, switching between dots played for a longer duration and fast-moving dots at
different lengths and tempos. There is a triplet feel here, which is undoubtedly
intentional. From 2:10 there is a f lourish of dots that lead to the ending at 2:20.
56 Steve Gibson

In Dots we are introduced to an apparently simple (on the surface) but master-
ful realisation of the visual music film. There is a complete unity between the
form of the visuals and the audio, and the technique of pen drawing on the film
shows an amazing f luidity that would be almost impossible to re-create using
computer-based editing.
McLaren continued to work through his lifetime on a series of visual music
films, primarily employing the technique of drawing on the film strip. He pro-
duced a diverse body of work in the late 1940s and 1950s, including Begone Dull
Care (1950), Now Is the Time (1953) and Neighbours (1953). The latter film, which
was perhaps his most well-known, applied techniques from his hand-drawn
films to a stop-motion animation using live actors. McLaren branched out from
quasi-abstraction to include more realist (or surrealist) and narrative elements.
While there are any number of films to talk about in relation to McLaren’s out-
put, one of his most justifiably well-known later works is the film Blinkity Blank
from 1955.

Blinkety [sic] Blank is a four-minute color film that was shot without a
camera. Right on the film itself MacLaren [sic] drew a number of designs
and abstract figures to create an erotic ballet of male and female elements
encountering each other. . . . Blinkety [sic] Blank is an absolutely unique
work which bears no resemblance to anything that has been made in sixty
years of filmmaking. In this “great little film” that’s only four minutes
long, there is all the fantasy of Giraudoux, the mastery of Hitchcock, and
the imagination of Cocteau.49

Blinkity Blank (see Figure 2.4) is unique in the era, as it combines McLaren’s
hand-drawn film technique (now with colour) precisely timed with Maurice
Blackburn’s (1914–1988) jazz-inf luenced modernist score, it employs a quasi-
realist story revolving around a male and female bird and it famously uses ‘inter-
mittent animation’ to create a stroboscopic visual field:

This experimental short film by Norman McLaren is a playful exercise

in intermittent animation and spasmodic imagery. Playing with the laws
relating to persistence of vision and after-image on the retina of the eye,
McLaren engraves pictures on blank film creating vivid, percussive effects.50

The film (which can be viewed at the National Film Board of Canada web-
site)51 employs the same precise timing between sound and visual as McLaren’s
previous work, but in this instance the musical score (with a few exceptions,
mostly in the credits) is not hand-drawn on the 35mm film, but rather an acous-
tic score using a small instrumental ensemble. As with his previous works, the
hand-drawn animations are precisely timed with the score, dancing in complete
synchronisation with the musical elements. By using the ‘persistence of vision’
effect, whereby images stay on the retina even when shown only intermittently,
 Moving Towards the Performed Image 57

FIGURE 2.4 McLaren, Norman, director. Blinkity Blank. Montreal: National Film
Board of Canada, 1955. Used by permission.
Source: https://help.nf b.ca/contact-the-nf b/?_gl=1*emvay9*_ga*NDIyMTQwMC4xNjIzMDY3
MDc3*_ga_EP6WV87GNV*MTYyMzA2NzA3Ni4xLjEuMTYyMzA2NzEyNS4w

McLaren also creates a unique effect that hints at a narrative without making it
explicit (until perhaps the end when the forms linger on screen for a bit longer).
It is worth noting that “McLaren knew about potential audiovisual mappings,
but chose not to follow them. In general, he adopted dark colors for low sounds and
light colors for high sounds, but felt free to disregard this principle if he did not find
it appropriate.”52 In other words McLaren was not a total formalist, and while he
generally employed a precise link between the rhythm of the film editing and the
rhythm of the music, he was not prone to making exact binding correspondences
between the sound elements and the visual elements. The only exception to this
rule was in Synchromy (1971) in which all audio-visual elements had precise cor-
respondences.53 While McLaren’s output was primarily, if not exclusively, in the
fixed domain of the linear film, there can be no doubt of his inf luence on the next
generation of visual music artists, including those such as Silver Wing Turquoise
Bird, who began to think of the visual as more of a performed medium.

Other Visual Music Filmmakers

In the period of roughly 1940–1955 (and beyond), there were a number of
other important visual music filmmakers creating work inf luenced by or
58 Steve Gibson

related to the work of Fischinger, Lye and McLaren. Photographer and set
designer Luigi Veronesi (1908–1998) “created animated films whose rhythm
was derived from music.”54 “The editing rhythm of Film n. 2, for example
was based on Fibonacci number sequence.”55 Filmmaker Harry Smith (1923–
1991), like Len Lye, was primarily interested in popular and folk music and
created some of his films by directly drawing on the film strip. “In works
such as Early Abstractions: Film No. 3 (Interwoven) (1949) his use of constantly
shifting, shimmering, vibrating forms exploding from the recesses of picto-
rial space in sync with the strains of Dizzy Gillespie give us the sense of deep
continuous space in a way that Lye’s films do not.”56 Visual music filmmaker
Hy Hirsh (1911–1961) was a cinematographer who worked in Hollywood and
“was interested in machines and technology; not only was he one of the first
to use an oscilloscope to create abstract images for films such as Eneri (1953) . . .
he also created his own optical printer, a special-effects machine that could
print one image onto another.”57

Conclusion
The artistic and technological experiments of these and other visual music film-
makers continued in the work of artists in the 1950s and beyond. These include
Stan Brakhage, John and James Whitney and many others, who continued to
expand on the notion of the visual music film, as outlined in Chapter 3. The
cross-fertilisation of sound and image explored in Fischinger, Lye and McLaren
had a profound effect on all this work as well as general audio-visual culture from
the 1950s to the present.
By 1955 new technologies of electronic sound production and tape-based
editing had advanced to the point that more complex matching of sound to
visuals was made more accessible (if not precisely simple). In the late modern
period that followed, large audio-visual events became increasingly possible. An
example was the Phillips Pavilion from Expo 1958 in Brussels, an event that
combined the architecture of Le Corbusier and Iannis Xenakis with the music
of Edgard Varèse diffused into a large pavilion. This work clearly continued a
trajectory that was begun in the early 20th century by pioneers such as Scriabin,
leading towards the present era of the performed image.

Notes
1 V.I. Pudovkin, “Film Technique,” in Daniel Talbot, ed., Film: An Anthology (Berkeley
and Los Angeles: University of California Press, 1969), 189–190.
2 The history of film technology is covered in other venues such as Ryan A. Piccirillo’s
excellent general overview, R.A. Piccirillo, “The Technological Evolution of Filmmak-
ing and Its Relation to Quality in Cinema,” Inquiries Journal/Student Pulse, 3(8), 2011,
www.inquiriesjournal.com/a?id=560.
3 For the full list see Teun Lucassen, “Color Organs,” in Human Media Interaction (Twente:
University of Twente, the Netherlands, 2008).
 Moving Towards the Performed Image 59

4 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 33.
5 See Adrano Abbado, “Luminous Instruments,” in Visual Music Masters (Milan: Skira
Editore, 2017), 28–41.
6 Jeremy Strick, “Visual Music,” in Kerry Brougher, Jeremy Strick, Ari Wiseman and
Judith Zilczer, Visual Music: Synaesthesia in Art and Music Since 1900 (London: Thames
and Hudson, 2005), 19.
7 Cretien van Campen, The Hidden Sense: Synesthesia in Art and Science (Cambridge, MA:
MIT Press, 2007), 1.
8 Anna M. Gawboy and Justin Townsend, “Scriabin and the Possible,” Music Theory
Online, 18(2), June 2012, https://mtosmt.org/issues/mto.12.18.2/mto.12.18.2.gawboy_
townsend.php.
9 Cretien van Campen, The Hidden Sense: Synesthesia in Art and Science (Cambridge, MA:
MIT Press, 2007), 5.
10 Anna M. Gawboy and Justin Townsend, “Scriabin and the Possible,” Music Theory
Online, 18(2), June 2012, https://mtosmt.org/issues/mto.12.18.2/mto.12.18.2.gawboy_
townsend.php.
11 Steve Gibson, “Simulating Synaesthesia in Real-Time Performance,” in Lanfranco Aceti,
Steve Gibson, and Stefan Müller-Arisona, co-eds., Live Visuals for Performance, Gaming,
Installation, and Electronic Environments (San Francisco: Leonardo Electronic Almanac,
2013), 214–229, www.leoalmanac.org/vol19-no3-simulating-synesthesia/.
12 Scriabin’s, Prometheus: Poem of Fire, Yale Campus YouTube page, www.youtube.com/
watch?v=V3B7uQ5K0IU.
13 Evan Norcross Flynn, Liberation of the Senses: An Exploration of Sound-Color Synesthesia in
the Music of Alexander Scriabin and Olivier Messiaen (Lawrence: The University of Kansas,
2014), 43.
14 Judith Zilczer, “Music for the Eyes: Abstract Painting and Light Art,” in Kerry Brougher,
Jeremy Strick, Ari Wiseman and Judith Zilczer, Visual Music: Synaesthesia in Art and
Music Since 1900 (London: Thames and Hudson, 2005), 73.
15 Olivia Mattis, “Scriabin to Gershwin: Color Music from a Musical Perspective,” in Kerry
Brougher, Jeremy Strick, Ari Wiseman and Judith Zilczer, Visual Music: Synaesthesia in
Art and Music Since 1900 (London: Thames and Hudson, 2005), 211.
16 Olivia Mattis, “Scriabin to Gershwin: Color Music from a Musical Perspective,” in Kerry
Brougher, Jeremy Strick, Ari Wiseman and Judith Zilczer, Visual Music: Synaesthesia in
Art and Music Since 1900 (London: Thames and Hudson, 2005), 213.
17 Olivia Mattis, “Scriabin to Gershwin: Color Music from a Musical Perspective,” in Kerry
Brougher, Jeremy Strick, Ari Wiseman and Judith Zilczer, Visual Music: Synaesthesia in
Art and Music Since 1900 (London: Thames and Hudson, 2005), 220.
18 Olivier Messiaen, Traité de rythme, de couleurs, et d’ornithologie, vol. 7 (Paris: Alphonse
Leduc, 2002), 7, translated by by Håkon Austbø in “Visualizing Visions: The Significance
of Messiaen’s Colours,” Music + Practice, vol. 2, www.musicandpractice.org/volume-2/
visualizing-visions-the-significance-of-messiaens-colours/, DOI: 10.32063/0201.
19 Olivier Messiaen, “Notice Analytique,” in The Score of Trois Petites Liturgies (Paris:
Durand, 1952), translated by by Håkon Austbø in “Visualizing Visions: The Significance
of Messiaen’s Colours,” Music + Practice, vol. 2, www.musicandpractice.org/volume-2/
visualizing-visions-the-significance-of-messiaens-colours/, DOI: 10.32063/0201.
20 Olivia Mattis, “Scriabin to Gershwin: Color Music from a Musical Perspective,” in Kerry
Brougher, Jeremy Strick, Ari Wiseman and Judith Zilczer, Visual Music: Synaesthesia in
Art and Music Since 1900 (London: Thames and Hudson, 2005), 225.
21 Olivia Mattis, “Scriabin to Gershwin: Color Music from a Musical Perspective,” in Kerry
Brougher, Jeremy Strick, Ari Wiseman and Judith Zilczer, Visual Music: Synaesthesia in
Art and Music Since 1900 (London: Thames and Hudson, 2005), 225.
22 A.V. Glinsky, The Theremin in the Emergence of Electronic Music (PhD. Dissertation, New
York University, 1992), 69. Quoted in Olivia Mattis, “Scriabin to Gershwin: Color
Music from a Musical Perspective,” in Kerry Brougher, Jeremy Strick, Ari Wiseman and
60 Steve Gibson

Judith Zilczer, Visual Music: Synaesthesia in Art and Music Since 1900 (London: Thames
and Hudson, 2005), 226.
23 Judith Zilczer, “Music for the Eyes: Abstract Painting and Light Art,” in Kerry Brougher,
Jeremy Strick, Ari Wiseman and Judith Zilczer, Visual Music: Synaesthesia in Art and
Music Since 1900 (London: Thames and Hudson, 2005), 76.
24 Judith Zilczer, “Music for the Eyes: Abstract Painting and Light Art,” in Kerry Brougher,
Jeremy Strick, Ari Wiseman and Judith Zilczer, Visual Music: Synaesthesia in Art and
Music Since 1900 (London: Thames and Hudson, 2005), 76.
25 Jeremy Strick, “Visual Music,” in Kerry Brougher, Jeremy Strick, Ari Wiseman and
Judith Zilczer, Visual Music: Synaesthesia in Art and Music Since 1900 (London: Thames
and Hudson, 2005), 18.
26 Roger Fry, An Important Event of the Season: Recent Paintings of Alfred Maurer of Paris (New
York: Folsom Galleries, 1913), cited in Judith Zilczer, The Aesthetic Struggle in America,
1913–18: Abstract Art and Theory in the Stieglitz Circle (PhD dissertation, University of
Delaware, 1975), 56.
27 Judith Zilczer, “Music for the Eyes: Abstract Painting and Light Art,” in Kerry Brougher,
Jeremy Strick, Ari Wiseman and Judith Zilczer, Visual Music: Synaesthesia in Art and
Music Since 1900 (London: Thames and Hudson, 2005), 32.
28 Judith Zilczer, “Music for the Eyes: Abstract Painting and Light Art,” in Kerry Brougher,
Jeremy Strick, Ari Wiseman and Judith Zilczer, Visual Music: Synaesthesia in Art and
Music Since 1900 (London: Thames and Hudson, 2005), 31.
29 Ilene Susan Fort, “Oskar Fischinger, Modernist Painter,” in Cindy Keefer and Jaap Gul-
demond, eds., Oskar Fischinger 1900–1967: Experiments in Cinematic Abstraction (Amster-
dam and Los Angeles: EYE Filmmuseum and Center for Visual Music, 2012), 56–57.
30 For examples see Judith Zilczer, “Music for the Eyes: Abstract Painting and Light Art,”
in Kerry Brougher, Jeremy Strick, Ari Wiseman and Judith Zilczer, Visual Music: Syn-
aesthesia in Art and Music Since 1900 (London: Thames and Hudson, 2005), 33–58.
31 Judith Zilczer, “Music for the Eyes: Abstract Painting and Light Art,” in Kerry Brougher,
Jeremy Strick, Ari Wiseman and Judith Zilczer, Visual Music: Synaesthesia in Art and
Music Since 1900 (London: Thames and Hudson, 2005), 51.
32 Kerry Brougher, “Visual Music Culture,” in Kerry Brougher, Jeremy Strick, Ari
Wiseman and Judith Zilczer, Visual Music: Synaesthesia in Art and Music Since 1900,
organized by Kerry Brougher (London: Thames and Hudson, 2005), 110.
33 Kristin Thompson, “Early Sound Counterpoint,” Yale French Studies, (60), 1980, 115–
140, www.jstor.org/stable/2930008, DOI: 10.2307/2930008.
34 Kerry Brougher, “Visual Music Culture,” in Kerry Brougher, Jeremy Strick, Ari Wise-
man and Judith Zilczer, Visual Music: Synaesthesia in Art and Music Since 1900, organized
by Kerry Brougher (London: Thames and Hudson, 2005), 100.
35 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 46.
36 William Mortiz, “Fischinger at Disney, or Oskar in the Mousetrap (Excerpt),” The Cen-
ter for Visual Music, 1977, www.centerforvisualmusic.org/OFMousetrap.htm.
37 Jaap Guldemond, Marente Bloemheuval, and Cindy Keefer, “Oskar Fischinger, an
Introduction,” in Cindy Keefer and Jaap Guldemond, eds., Oskar Fischinger 1900–1967:
Experiments in Cinematic Abstraction (Amsterdam and Los Angeles: EYE Filmmuseum and
Center for Visual Music, 2012), 10–11.
38 Oskar Fischinger, Studie nr. 5 (excerpt), The Center for Visual Music Vimeo channel,
https://vimeo.com/258726727.
39 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 49.
40 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 49.
41 Kerry Brougher, “Visual Music Culture,” in Kerry Brougher, Jeremy Strick, Ari Wise-
man and Judith Zilczer, Visual Music: Synaesthesia in Art and Music Since 1900, organized
by Kerry Brougher (London: Thames and Hudson, 2005), 111.
42 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 49.
 Moving Towards the Performed Image 61

43 Len Lye, Swinging the Lambeth Walk, 1940, The British Council Film Vimeo channel,
https://vimeo.com/92722063.
44 The Len Lye Foundation Official Website, All Souls Carnival, 1957, www.lenlyefoundation.
com/films/all-souls-carnival/32/.
45 The Len Lye Foundation Official Website, All Souls Carnival, 1957, www.lenlyefoundation.
com/films/all-souls-carnival/32/.
46 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 51.
47 Norman McLaren, Dots, 1940, The National Film Board of Canada Vimeo Channel,
https://vimeo.com/32645760.
48 W.E. Jordan, “Norman McLaren: His Career and Techniques,” The Quarterly of Film
Radio and Television, 8(1), 1953, 1–14.
49 François Truffaut, The Films in My Life, translated by Leonard Mayhew (New York:
Simon & Shuster, Inc. 1978), 269.
50 Norman McLaren, Blinkity Blank, 1955, The National Film Board of Canada Official
Website, www.nfb.ca/film/blinkity-blank/.
51 Norman McLaren, Blinkity Blank, 1955, The National Film Board of Canada Official
Website, www.nfb.ca/film/blinkity-blank/.
52 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 52.
53 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 52.
54 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 52.
55 L. Caramel and A. Madesani, Luigi Veronesi e Cioni Carpi all Cineteca Italiana, quoted in
Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 52.
56 Kerry Brougher, “Visual Music Culture,” in Kerry Brougher, Jeremy Strick, Ari Wiseman
and Judith Zilczer, Visual Music: Synaesthesia in Art and Music Since 1900, organized by Kerry
Brougher (London: Thames and Hudson, 2005), 117.
57 Kerry Brougher, “Visual Music Culture,” in Kerry Brougher, Jeremy Strick, Ari
Wiseman and Judith Zilczer, Visual Music: Synaesthesia in Art and Music Since 1900,
organized by Kerry Brougher (London: Thames and Hudson, 2005), 112.
3
LIQUID VISUALS
Late Modernism and Analogue Live Visuals
(1950–1985)

Steve Gibson

Introduction
This chapter focuses on the period from the 1950s to the 1980s and provides
an overview of the dramatic expansion of audio-visual culture and Live Visuals
practice during this timeframe. With the development of new tools, along with
experimental ideas, the concept of the performed image became a reality. The
year 1955 is also a particularly important date, as it can be seen as the start of rock
and roll music. It is also roughly the time in which electronic music expanded
dramatically, both in modernist circles (Stockhausen, Cage, etc.) and in more
popular forms (Louis and Bebe Barron’s score for Forbidden Planet being a key
exemplar).
Technologically, beginning with analogue tools such as film loops and over-
head projection systems and moving towards video in the 1970s and 1980s, the
moving image became a potent force for live performance, not only in so-called
serious art culture but also in popular culture. In addition, the concept of visual
music as established by Fischinger and McLaren (as discussed in Chapter 2) con-
tinued to have wide impact, creating offshoots in the visual music films of John
Whitney and Larry Cuba.
Simultaneous to the earlier developments, the expansion of electronic music
in the 1950s brought new tools for performing and/or distributing sound by
electronic means. Tape-based editing as employed by musique concrete composers
Pierre Schaeffer and Pierre Henry and synthesised sounds by Karlheinz Stock-
hausen using newly developed studios in Cologne pointed to a new form of music
that had natural affinities with the visual music of Fischinger and McLaren.
These were organically combined in a dramatic fashion by Edgard Varèse in
his piece Poeme électronique. This piece was composed for the Phillips Pavilion
in the Brussels Expo of 1958 and combined audio by Varèse and images from
DOI: 10.4324/9781003282396-5
 Liquid Visuals 63

Le Corbusier. Poeme électronique prefigured both the music video and the audio-
visual installation as forms.
The 1960s saw a dramatic development of mixed media performance, engen-
dered both by new technologies and by collaborations between popular and art
culture. The Fluxus group of artists began exploring mixed media in a perfor-
mance context, often employing absurdist ideas arguably inherited from Dadaism
and frequently built on ephemeral, conceptual strategies (as opposed to object-
based and craft-oriented techniques). This in turn inf luenced the psychedelic
movement, which exploded with liquid light show performances by The Joshua
Light Show and Single Wing Turquoise Bird. Employing repurposed overhead
projectors mixed with oils and later adding film loops, these groups created
elaborate live visual performances improvised to the music of Frank Zappa, the
Grateful Dead and others.
Pop art live multimedia reached its apex in the 1960s with Andy Warhol’s
collaboration with the Velvet Underground, The Exploding Plastic Inevitable, now
possibly the most well-known mixed-media event of the era, that combined
lights, projections and live music. This performance set the stage for the develop-
ment of multimedia performance using Live Visuals. Parallel to this, expanded
cinema (as outlined by Gene Youngblood in his seminal text of 1970 and prac-
ticed by Guy Sherwin, Tony Hill, Jeffrey Shaw and others) began to conceive of
film as a performed medium, with the filmmaker often interacting in some way
with the film materials in a live context.
The chapter concludes with a look at the developments in the 1970s and 1980s
that moved out of the analogue domain and into the digital. With the develop-
ment of video editing devices and video synthesizers (both analogue and digital)
and the rise of the personal computer, visual effects were easier to achieve in real
time, and artists were able to start working more precisely with visual material
in a performance context. Looking at installations such as Myron Krueger’s Vid-
eoplace, Stephen Jones’s work with video synthesizers for the experimental synth
group Severed Heads and large-scale multimedia performances such as Laurie
Anderson’s United States Live, as well as Merrill Aldighieri’s live video mixes, this
chapter argues that the hybrid form of contemporary Live Visuals was prefigured
and borne out of these experiments.

The Technology of Live Audio-Visuals in Late Modernism

As illustrated in Chapter 2, the number of new technologies that were developed
in the early part of the 20th century was startling; however, it could be argued
that many of these technologies became quickly obsolete, or at best led to the
development of newer, better (and more enduring) versions of similar devices.
It could equally be argued that the late modernist period (here roughly desig-
nated as 1955–1985) was an even more prolific era of innovation in audio-visual
technology, and most importantly many of the technologies developed in this
64 Steve Gibson

era (e.g. synthesizers, videotape recorders) had widespread use and a relatively
long life span.
As Léon McCarthy argues broadly in Chapter 8 “aesthetics are profoundly
coupled to technological processes and inventions.”1 This is especially evident
in a form such as Live Visuals, in which the technological means and processes
are so intricately linked with the concerns and contents of the medium and are
simultaneously subject to change at a rapid rate. Given this, it is no accident that
the precipitous rise of various audio-visual technologies in the late modernist era
was accompanied by an equally precipitous rise in the number of audio-visual
forms, from electronic music to expanded cinema, music video and multimedia
performance. Unquestionably these forms were ultimately enabled by techno-
logical advancements, so much so that they would be impossible to have been
established without their relevant technologies.
Key to the development of many of these new audio-visual forms were two
key advances: the development of various electronic music devices, from simple
oscillators and tone generators in the 1950s to the synthesizers of the late 1960s
and 1970s, and the advancement of film and video technology from simple 8 and
16mm film, to 35mm and 70mm, to VHS and Beta videotape recorders in the
1970s and 1980s, and finally to various devices for editing and/or manipulating
visual material (i.e. videotape editors, video synthesizers).

The Technology of Electronic Music in Late Modernism

The early development of electronic music is well documented in various vol-
umes, including Paul Griffiths’s A Guide to Electronic Music, among many others;
however, for our purposes it is worth noting a few of the key developments that
had a broad impact on audio-visual culture and are therefore relevant to the his-
tory of Live Visuals. The importance of musique concrete and the work of Pierre
Schaeffer (1910–1995) and others at Radio-difusion-Télévision Française from 1948
onwards cannot be overstated. Utilising the new post-war recording tools, such
as a reel-to-reel tape deck, as well as simple editing by means of razor blades,
Schaeffer pioneered a completely new way of creating sound through the col-
lage of found and recorded sound. Using taped recordings of “railway trains or
the piano . . . the recordings were transformed by his playing them at different
speeds, forwards or in reverse, isolating fragments and superimposing one sound
on the other.”2 The techniques used by musique concrete have rough parallels with
visual music filmmakers such as McLaren and Lye, particularly in the use of a
recorded medium for hand-crafted editing. It is also easy to trace the inf luence of
musique concrete into such diverse musical forms as sample-based music (i.e. hip-
hop), but also to visual forms such as experimental film and onwards to scratch
video.
Similarly, the Elektronische Musik of Karlheinz Stockhausen (1928–2007) and
others at Cologne Nordwestdeutscher Rundfunk from 1952 onwards had a profound
inf luence on the development of electronic music and arguably all audio-visual
 Liquid Visuals 65

culture of the next 50 years. Music “generated exclusively by electronic means”3

was the goal of Stockhausen and others in Cologne, and the impact of their early
work with electronically generated sound had a profound impact on audio-visual
culture of the late 20th and early 21st centuries. Given the intimate connection
between electronic music and contemporary Live Visuals, it is not an exaggeration
to state that Live Visuals would not exist in their current form without the initial
work by Stockhausen and others to produce work entirely electronically. In the
1950s any number of similar electronic music studios arose, and key composers of
the era such as Cage, Varèse and Berio began to work with electronically generated
sound, concrète tape-based music or some combination of the two forms.

FIGURE 3.1 A music studio at the College for Music (Musikhochschule) in Cologne,
Germany, has two analogue synthesizers still in action: a Moog modular
synthesizer from the late 1960s and, in the background, an ARP 2500
from 1970.
Source: Photo by Maximilian Schönherr. GNU Free Documentation License. Creative Commons
Attribution-Share Alike. https://commons.wikimedia.org/wiki/File:Moog_und_ARP_Synthesizer.
jpg
66 Steve Gibson

A key follow-on development from Elektronische Musik that changed music pro-
duction dramatically was the invention of the synthesizer in the late 1950s. The
first commonly available device was the RCA synthesizer, the Mark 2, which
gained some traction in the late 1950s and 1960s; however, it wasn’t until Bob
Moog (1934–2005) and Don Buchla (1937–2016) released more stable modular
synthesizers in the early 1960s that the synthesizer really took off as a genuinely
viable device for a broader group of musicians and composers (see Figure 3.1).4
Moog’s original synthesizers were much easier to use than the RCA, as they
were (relatively) easily tuned and included a keyboard. Through the invention
and popularisation of the synthesizer, electronic music was truly established
as a separate form: over the next 50 years it continually expanded to become
arguably the most dominant musical form of the 21st century. Again, given the
intricate connection between electronic music and Live Visuals, the technol-
ogy of the synthesizer is key to understanding the history of the performed
image.

The Technology of Audio-Visuals and Visual Music

in Late Modernism
In the 1940s and 1950s there was an equal, if less publicly celebrated, explosion
of visual technologies that were often (though not always) used in tandem with
the audio technologies mentioned earlier. Key to the development of new visual
devices and strategies “were the brothers John Sr. (1917–1995) and James Whit-
ney (1921–1982), who in different ways created works that left indelible marks on
the history of visual music.”5 The brothers (alone or together) worked with many
key technologies of the late modern era from analogue devices such as an optical
printer in the 1940s to computers in the 1960s and beyond.
A good example of the brothers’ early work can be seen in James Whitney’s
film Yantra (1950–1958)6 with an electronic score by Henk Badings (1907–1987).
This work represents a logical continuation of the hand-drawn visual music
films of McLaren and Lye. Unlike McLaren’s work, though, the music is not
drawn on the film strip, but is composed by electronic means. Also, unlike both
McLaren and Lye, the combination of the audio and the visuals is not linked
by precise correspondences between the mediums (though there are attempts
at synchronisation at key moments), but rather on the creation of particular
synchronous moods. The piece represents both a precursor to the psychedelic
visual era of the 1960s and a masterful summing up of the hand-drawn visual
music film.
In succeeding years both brothers moved into increasingly computer-based
audio-visual compositions, using repurposed technology such as WWII ana-
logue computers:

In Los Angeles in the late 1950s, John Whitney started purchasing junk:
“mechanical junk excreted from army depots across the country. . . . Junk
 Liquid Visuals 67

such as brand new thirty-thousand dollar antiaircraft specialized analog

ballistic problem solver computers dating back to World War II.” 7 He
transformed this military-spec surplus into a machine for creating experi-
mental animation – literally and metaphorically retooling a device that
had itself served to remake human vision for modern war. A twin of this
machine would enable John’s brother James to create the 1966 film Lapis.8

Key to note here is that the brothers used repurposed technology to create this
work. Repurposing of ‘other-intended’ technology as described earlier also reso-
nates with the work of musique concrete composers described previously, but also
foreshadows the work of succeeding Live Visualists, from experiments with oils
on overhead projectors in the 1960s, to the use of the film projector as a live
performance device in expanded cinema, to the scratch video of the 1980s and
1990s. Without a doubt, the history of audio-visuals and Live Visuals in the late
modern period is one in which almost many, if not most, key advancements were
made by repurposing existing technologies.
James Whitney’s Lapis is described in particularly striking terms by expanded
cinema pioneer Gene Youngblood (1942–2021) as “cybernetic cinema.” 9 In
general terms “cybernetics became part of the foundation for an emerging dis-
course of both human-machine interaction and computational representation.
Youngblood’s rhetoric situated the Whitneys’ films not simply as works made
with a computer but as works engaged with this larger field.”10 Key to this
characterisation is the notion of a larger body of technologically created audio-
visual art production, in this case under the banner of cybernetics. While the
visual field of Lapis superficially recalls the hand-drawn visual music of Lye
and McLaren, its sensibility is much more in keeping with 1960s psychedelia
and California-based cybernetic art. This is evident both in the choice of music
(Ravi Shankar) and in the use of the analogue computer to generate complex
mandalas.11 Certainly watching Lapis in the present era, one can see the visual
composition as key to the development of a genuine style for audio-visual cul-
ture, and (acknowledged or unacknowledged) traces of the Whitneys’ work
can be felt in the work of club visualists, as well as more experimental live
audio-visualists.
John Whitney had a long and varied career following this and is now consid-
ered one of the pioneers of computer graphics audio-visuals:

Whitney had an opportunity to work on the new high-powered digital

computers between 1966 and 1969, when he was awarded a fellowship
as artist-in-residence at IBM . . . John Whitney’s computer films grew
continually more intricate in their exploration of a genuine aesthetic goal:
the establishment of a secure basis for harmonic events in audio-visual
presentation. In the later 1980s, Whitney concentrated on developing a
computerized instrument on which one could compose visual and musical
output simultaneously in real time.12
68 Steve Gibson

FIGURE 3.2 Le Corbusier and Edgar Varese, Poeme électronique, Phillpis Pavilion,
Brussels World’s Fair, 1958.
Source: Photo by Maximilian Schönherr. GNU Free Documentation License. Creative Commons
Attribution-Share Alike. https://creativecommons.org/licenses/by-sa/3.0/

Touching on so many aspects of audio-visual culture, the inf luence of the

Whitneys on the work that followed in the succeeding decades was direct and
profound.
Running roughly parallel to the Whitneys’ experiments of the 1950s was the
work of architect Le Corbusier (1887–1965), his then student Iannis Xenakis
(1922–2001) and composer Edgar Varèse (1883–1965), who collectively produced
their installation Poeme électronique for the Phillips Pavilion in the Brussels Expo
of 1958 (see Figure 3.2). Le Corbusier, of course, was extremely well-known as
an architect at this point, and Varèse had already built a long career composing
both acoustic and electronic music:

The two conceived a production that included architecture, light, music,

space and color. . . . Varèse’s music, made with electronic and concrete
sounds, was not in strict relation with the images, indeed it had been
 Liquid Visuals 69

conceived to provide a contrast with the images. . . . At the same time,

Varèse was able to explore the spatial effect of sound, since he worked with
fourteen audio tracks piped through three hundred and fifty loudspeakers.
From this point of view, one can claim that the work certainly contributed
to the dialogue between music and the visual arts.13

Key to note in relation to Poeme électronique is its conception as an installation

to be experienced in a public venue. While the audio and visual materials were
fixed and played back on a projection and sound system, they were meant to be
experienced in this particular venue, with a substantial number of speakers and
in an imposing architectural space. The work is obviously not ‘live’ per se, yet
the experience of it required the live context in order for it to be fully appreci-
ated. In addition, it is key to note the future relationship between Live Visuals
and architecture, which is effectively pre-figured by the Phillips Pavilion. It can
be seen as perhaps the defining starting point for considering the relationship
between audio-visuals and architecture. See Chapter 14 for a further discussion
of this.

Fluxus, Happenings and Liquid Psychedelia in the 1960s

If the work in the 1950s described earlier paved the way for the visual to be
moved into a live domain, then the 1960s can be described as the era in which
visuals became genuinely live. This was not only due to a solidification of tech-
nological advances, such as those described previously, but also to an increas-
ing willingness to consider the visual as a performance medium in more direct
terms. It is also at this point that the culture of late modernism collided with
popular culture, and the results of this combination were felt on a number of
fronts from popular music (e.g. The Beatles, The Velvet Underground) to visual
and performance art (Warhol, Nam June Paik).
Key to the development of a live visual performance approach in the 1960s
was the Fluxus movement, and more specifically the ‘happenings’ that emerged
from the movement. The first happening can be traced to Paris in 1958: The-
atre Is in the Street by Wolf Vostell (1932–1998), which was “the first artwork to
incorporate a TV set.”14 The happening became a fixture of the New York–based
Fluxus movement, spearheaded by Allan Kaprow (1927–2006) and others. Key
to Fluxus-based happenings was a mixed-media approach to live performance
that broke “down the barriers between artists and audiences, involving the pub-
lic in the work itself.”15 In this way Fluxus happenings cannot only be viewed as
nascent Live Visuals experiences but also as forerunners to interactive art. Hap-
penings became a fixture in 1960s America and opened up late modernist art to
ideas of improvised performance, visual spontaneity and absurdity.
Possibly the most well-known of the Fluxus artists was Nam June Paik (1932–
2006), who was educated as a composer and an early colleague of Stockhausen.
He was one of the first artists to work explicitly and regularly with the television
70 Steve Gibson

FIGURE 3.3 Nam June Paik – Shuya Abe: Paik-Abe Video-Synthesizer, 1969/92.
Changing Channels, MUMOK, Vienna, 2010. Photo: Zs. Gyenes. Used
by permission.
Source: www.kunsthalle-bremen.de/en/view/static/page/contact-us

as both an exhibition and stage device. His work in the 1960s included TV Cello,
a stack of TVs repurposed as a string instrument (see Léon McCarthy’s further
discussion of this in Chapter 8). Key to Paik’s work was the use of visual devices
in the performance context (though most Fluxus artists bristled at the term ‘per-
formance’ at the time).16
Paik was also a keen inventor, and his work with engineer Shuya Abe (1932–)
resulted in the Paik-Abe Video Synthesizer (see Figure 3.3):

From 1969 to 1971, together with television technician and specialist Shuya
Abe, Paik constructed a video synthesizer that made it possible for him to
edit seven different sources simultaneously – in real time. Seven cameras
are calibrated to receive seven colors, each perceiving/photographing only
a single color. The equipment is enhanced by a button for mixing, and a
small clock that reverses the colors – from ultraviolet to infrared.17

While the device itself remained a curiosity (though it was later used by one
of Paik’s students, Jim Wiseman),18 both in its form and intent, the device is an
obvious precursor to other video synthesizers such as the Fairlight Computer
 Liquid Visuals 71

Video Instrument (released in 1985). In addition, the ability of the Paik-Abe

Video Synthesizer to mix up to seven channels of video live was key to the con-
ceptual idea of video mixing, which was later realised in the video mixers of the
1980s and 1990s and the VJ software of the 21st century. Paik did actually use
the device, with the first ‘performance’ being a broadcast event at a Boston area
television station:

The Synthesizer debut[ed] in a four-hour broadcast television show called

“Video Commune – The Beatles from Beginning to End” on WGBH,
channel 44 on August 1, 1970. Paik took advantage of a licensing agree-
ment that WGBH had which gave them rights to air all Beatles songs.
So he created four hours of a wildly colorful broadcast performance to a
soundtrack of Beatles music.19

If Fluxus represented an advancement in both the idea of the visual as a perfor-

mance medium and as a site for technological advances such as those made by
Paik, Pop Art represented a consolidation of late modernism and popular cul-
ture, as well as providing the one of the clearest examples of genuinely impro-
vised Live Visuals. Arguably the key figures, particularly for a discussion of Live
Visuals, were Andy Warhol (1928–1987) and The Velvet Underground. Warhol
was obviously very well established in the New York art scene in the 1960s and
was a central figure (along with Robert Rauschenberg and others) in the Pop Art
movement. The Velvet Underground were an interesting mix of popular musi-
cians (e.g. Lou Reed), as well as figures associated with the New York avant-
garde (e.g. John Cale, who played in minimalist composer La Monte Young’s
ensemble).
Warhol’s collaboration with The Velvet Underground (and others) from
1966 to 1967, named The Exploding Plastic Inevitable (see Figure 3.4), consisted
of “installations of monitors, film and slide projectors, lights and live music
by Nico and the Velvet Underground, and other performances. These events
certainly contributed to further the idea of multimedia works that were not
pre-established.”20 The Exploding Plastic Inevitable is noteworthy due to its impro-
vised, non-repeating and ‘not pre-established’ form. The visuals were f luid and
changed as desired in response to the music or the mood of the audience. While
obviously the film footage used was pre-shot, it could be reconfigured differ-
ently in each instance of the event. This mode of operation is key to Live Visuals
and audio-visual performance, as it is a preferred method of performance for
many live visual artists (including VJs) as well as live audio-visual performers.
Various video documents of The Exploding Plastic Inevitable demonstrate its obvi-
ous connection to these later forms in its use of multiscreen projections, complex
lighting setups and a theatrical performance staging.21
Running roughly parallel to the happenings and Pop Art mixed-media events
in New York were a series of psychedelic live audio-visual events in Los Angeles,
San Francisco and later elsewhere. These ‘psychedelic light shows’ echoed the
72 Steve Gibson

FIGURE 3.4 Andy Warhol’s multiple-screen projection environment – The Exploding

Plastic Inevitable, with the music of The Velvet Underground and Nico.
Photos © Ronald Nameth, 1966–2020. All rights reserved. Reprinted
by permission.
Source: ArtSiteIn KB, Stockholm, Sweden.

general approach of the Fluxus group, while simultaneously they expanded on

both the technology used and the audience reach:

The psychedelic light show rejected the materialist art object in favour of
the ephemeral Happening, the commercial gallery for the underground
space. . . . In these events, painting, film, color organs, and music came
together. . . . As early as 1952 . . . Seymour Locks taught a course called
“Light and Art” in which he demonstrated to his students the possibility of
creating motion painting by swirling colored liquids in a dish and casting
the “painting” on the wall by means of an overhead projector while a jazz
group improvised a musical accompaniment.22

The psychedelic light show was quickly adopted at various venues, particularly in
San Francisco, and was often accompanied by the music of the electronic music
composers associated with what later become the San Francisco Tape Music
Center, including high-profile figures such as Pauline Oliveros (1932–2016)
and Morton Subotnik (1933–).23 These shows eventually branched out to other
 Liquid Visuals 73

locales and cities, including London and New York: “In London, artists such as
Mark Boyle and Joan Hills, as well as Gustav Metzger, created light shows as
art events and for rock concerts ranging from Soft Machine, Pink Floyd, and
the Who.”24 In New York, a number of artist/techies, including Joshua White,
William Schwarzbac, and Tom Shoesmith, consolidated into the Joshua Light
Show.25
The Joshua Light Show was for many the public face of Live Visuals culture
in the 1960s, appearing with popular musical figures as Jimi Hendrix, Frank
Zappa and Janis Joplin. For their performances they used “liquid projection
using colored oil and water on overhead projectors,”26 clearly continuing the
Whitney brothers’ (and others) proclivity for repurposing existing technology.
These oils were mixed live and projected large-scale, creating spectacular, non-
repeatable visuals performances in sync with the psychedelic music of the era (see
Figure 3.5). With these shows Live Visuals unquestionably entered the main-
stream of popular culture.
Simultaneous to the Joshua Light Show’s events in New York, Silver Wing
Turquoise Bird in Los Angeles were performing alongside The Grateful Dead,
Steve Miller Band and The Velvet Underground. Silver Wing were “a more
film-based light-show group . . . composed of a number of filmmakers and
artists directly aware of the work of Fischinger, the Whitneys and [ Jordan]

FIGURE 3.5 The Joshua Light Show with Frank Zappa and the Mothers of Invention,
The Mineola Theater, Long Island, New York, 20 December 1967.
Used by permission.
Source: [email protected]
74 Steve Gibson

Belson.”27 Silver Wing applied a more cinematic approach to their live perfor-
mances than the Joshua Light show and used “film, strobes, and a vast array of
slides that could be animated over other images.”28 Their lionisation by Gene
Youngblood as “a combination of Jackson Pollack, and 2001”29 ties them more
directly to expanded cinema, as does the fact that they also performed in galleries
and museums as experimental performing artists in addition to the shows with
popular music groups. In this respect Silver Wing connected Live Visuals to both
art culture (and in particular to expanded cinema) and to popular culture.

Expanded and Live Cinema

Gene Youngblood’s exhaustive volume Expanded Cinema, published in 1970,
traced various innovations in the conception of what could constitute cinema
(beyond its limitations in pure drama), from synesthesia to cybernetics, through
to TV and intermedia. Covering artists as diverse as Jordan Belson (1926–2011)
and Stanley Kubrick (1928–1999), the book proposed generally that cinema had
become much than a simple playback medium, that in fact there were mul-
tivarious approaches to cinema, including through television, multiprojection
environments and intermedia. While the complete history of expanded cinema
outlined by Youngblood is beyond the scope of this book, some key concepts he
discussed are crucial to the understanding of later Live Visuals movements.
One of these is obviously Youngblood’s prescient understanding of the pro-
found implications of computer hardware and software for the future: “the digi-
tal computer opens vast new realms of possible aesthetic investigation.”30 This is
obviously a given in contemporary practice, but in 1970 this would have been
considered a somewhat radical idea, particularly as most computing power at the
time was in the hands of the US military and specialist technology companies
such as IBM. On the other hand, undoubtedly at this time there was a move
to integrate the arts and computing in specific ways, to humanise the power of
the computer, including at the Bell Labs programme “Experiments in Art and
Technology” started by Billy Klüver (1927–2004). The work of Bell Labs was
followed up by artist/programmers such as Myron Krueger (1942–), discussed
later. Unquestionably the evolution of audio-visual culture more broadly over
the following 50 years was deeply reliant on the aesthetic possibilities of comput-
ers that Youngblood so shrewdly foresaw. More specifically, the contemporary
forms of Live Visuals such as VJing, projection mapping and Live Cinema would
be demonstrably impossible without the computing advances that happened
between 1970 and the present.
Equally important was Youngblood’s demonstration of all the ways in which
cinema had or was transcending its limits as a dramatic medium as consumed in
the film theatre. Covering an astonishing array of material from the art cinema
of Stan Brakhage, Michael Snow and others (intended to be viewed in a gallery),
to the installations and performances of Nam June Paik and Stan VanDerBeek
 Liquid Visuals 75

(to be experienced in a live setting), and on to holography, Youngblood charts

the course of how cinema and the cinematic expanded into a non-dramatic and
eventually a live medium. Undoubtedly Youngblood’s idea inf luenced a genera-
tion of filmmakers, many of whom produced work in which the performance of
the film was the work. Combined with inf luences from Fluxus, a new genera-
tion of artists began to think of film as a performance medium, often with the
filmmaker as the key performer within the projected artwork.
While several filmmakers took onboard and actively used the banner of
expanded and live cinema in the 1970s, key among them were British-born art-
ists Guy Sherwin (1948–) and Tony Hill (1946–). These filmmakers transformed
film into a live, performative medium in which they or the audience interacted
with the projected film, usually in a gallery or similar space. In this way the live
cinema of the 1970s connected the art film culture of the 1950s and 1960s (the
Whitneys, Stan Brakhage) with the Fluxus, Pop Art and psychedelic light per-
formances of the 1960s. A key early work of live cinema is Sherman’s Man with
a Mirror from 1976:

The performance involves Sherwin holding up a mirror which is painted

white on the back, acting as a screen. The image is projected from a Super
8 projector onto the screen/mirror. The projected image, which shows
Sherwin holding a similar mirror/screen in a park was shot in 1976. The
intention was to perform the piece later in the year it was made. However
as time has passed and Sherwin has been asked to repeat the performance,
it has taken on new meaning as the performer has aged and time has moved
on.31 (Note: BD films has made a performance of this work available on
YouTube: www.youtube.com/watch?v=DX1-xuCNIe).

This live cinema performance is key to understanding the history and tech-
nology of early live cinema, as well as later developments connected to it,
including the work of contemporary live cinema artists. In the performance
Sherwin holds a mirror up to the projected image of himself, moving it around
the room and changing its focus and angle. It becomes a sort of low-tech
visual effects system for compositing an image on a figure, as well as a ghostly
ref lection on the nature of the changing self. In this manner, its ‘lo-techness’
is quite similar to the use of overhead projectors by the psychedelic light show
artists of the previous ten years and ties broadly into the repurposing common
in much historical and contemporary Live Visuals production. Live cinema is
generally characterised not only by the liveness of the cinema experience but
also by the inclusion of repurposed devices (some simple, some much more
complex) often used to distort the film image in some way in relation to the
performance itself.
Another live cinema artist known for repurposing devices and including both
himself and the audience in the live experience is Tony Hill. His Floor Film from
76 Steve Gibson

FIGURE 3.6 Floor Film by Tony Hill, 16mm film installation, 30 minutes, 1975. Images
taken during a screening with a young audience on the screen watching
the film. ©Tony Hill 2020. Used by permission.
Source: Mail conversation with artist available on request

1975 (see Figure 3.6) is similar in its ‘liveness’ to Sherwin’s work as described
earlier but also includes the audience as part of the live film experience:

The 1975 original of this unique film was projected via a large, overhead
mirror onto a screen which formed the f loor of a small room. The audi-
ence watched the film either by standing on the screen or by viewing
through the mirror. Seen through the mirror the audience members in the
room become part of the film. Those standing on the screen experience
situations such as walking on water, the screen catching fire and other
unusual events.32 (Note: A recent 2016 upgrade of this film installation can
be seen on Vimeo: https://vimeo.com/198752309).

In Floor Film audience members can superimpose themselves over a film projected
onto the f loor, often moving in front of or alongside figures and/or objects in
the film scene. The audience therefore becomes a direct part of the performance
event, in essence foreshadowing the role of audience later seen in interactive
art of the 1990s and following. Hill is also well known for his technical rigs,
which often involve repurposing or developing technologies for distinct filmic
 Liquid Visuals 77

purposes. These rigs are often used to have an immersive experience of the film
event.33 For more information about Tony Hill’s work please see Chapter 16,
Interview 1, for a detailed interview with him.
The experience of live cinema continued from the mid-1970s to the present.
Both Hill and Sherwin (as well as other expanded cinema artists such as Mal-
colm Le Grice) have created multiple works since these initial projects, often
with updated technologies and techniques. In addition, many succeeding artists,
from Laurie Anderson to The Light Surgeons, followed on from the approaches
of these early works and incorporated film in a live and performative context,
albeit in more multimedia contexts. In the early 21st century live cinema is now
a recognised, distinct form that overlaps audio-visual and live visual performance
in some ways, but also retains unique features that can be tied back to these ini-
tial experiments. See Chapter 12 for more on contemporary live cinema work.

Moving Towards the Digital I: Electronic Art and

Interactive Installation
With the development of digital computers in the 1970s, visuals slowly became
more digital, both in their production and dissemination. In addition, the devel-
opment of various electronic music devices, including monophonic and poly-
phonic synthesizers, accelerated as the decade wore on, becoming somewhat
ubiquitous in popular culture by the end of the decade. In this era, the idea of
electronic art (later digital art and media art, all terms that were used interchange-
ably) was effectively born as a medium. Essentially created by artists with one foot
in the technological world and the other in the art world, electronic art of 1970s
became key to the development of audio-visual culture over the next 50 years.
Arguably the birthplace of electronic art was The Kitchen in New York, which
has operated from 1971 to the present day as a place for artists to experiment
with various forms of technology and to present live electronic work. Among the
founders of The Kitchen were Steina (1940–) and Woody Vasulka (1937–2019),
whose piece Violin Power (1970–1978) used the violin “as an instrument that con-
trolled real-time processing of images during performances,”34 effectively presag-
ing later audio-visual work from Laurie Anderson (1947–) to alva noto:

The Vasulkas used audio synthesizers as a starting point: the core of their
system was in particularly the idea of the oscillator, that is the wave genera-
tor that could be used for both audio and video. By regulating a synthesiz-
er’s control tension, they could in fact simultaneously modify the sound’s
pitch and the image’s size.35

This direct use of the synthesizer as a simultaneous control device for audio and
visuals stands as a key development for VJ/DJ culture and present-day audio-
visual performance, particularly as the keyboard synthesizer (along with the
MIDI control surface) remains a key interface device for live audio-visual per-
formance to the present day.
78 Steve Gibson

Another figure who was key to the development of electronic and interactive
art in the 1970s was Myron Krueger (1942–):

Myron Krueger was the first artist to focus on interactive computer art as
a composable medium. In the process, he invented many of the basic con-
cepts of virtual reality. He pioneered the development of unencumbered,
full-body participation in computer-created telecommunication experi-
ences and coined the term “Artificial Reality” in 1973 to describe the
ultimate expression of this concept.36

Krueger had a distinct advantage over many other electronic and interac-
tive artists, as he was trained as a computer scientist (he received his PhD in
computer science from the University of Wisconsin-Madison). He therefore
had the technical background to develop his own computer systems. Begin-
ning in 1969 (with Daniel Sandin and others), he “developed the audiovisual
interactive environment glowf low,”37 which allowed the audience to interact
with light and sound in real time. His key project, however, was Videoplace
(1974). 38
Videoplace was a key piece of interactive art for Live Visuals, as it used a
camera-based system to ‘transport’ the outline of users’ figures into a simple
computer world, in which they could interact in real time with visual figures:

“Videoplace” takes up the closed-circuit video installation prefigured

in 1969 in Nam June Paik´s “Participation TV II”39 and modifies it:
In Paik´s installation observers could use a control panel to manipu-
late video images recorded by video cameras and projected on moni-
tors. Krueger replaces Paik´s interface (control panel and cameras) by
a series of programmes transforming the recordings of a black/white-
surveillance camera. The camera mounted underneath the projection
surface records observers and their operations “against a brightly backlit
sheet of translucent plastic.” On a computer with parallel active “spe-
cialized processors” the software gathers the camera´s input as a “binary
image” transforming the observers´ contours in a field with ones and
zeros for recognised/non recognised observer operations. The software
registers motions of heads, hands, fingers, legs and feet.40 These data are
used by programmes transforming and colouring the observers´ con-
tours in different manners.41

Krueger laid the philosophical, technical and conceptual foundation for inter-
active art with Videoplace. Importantly, he established that the quality of interac-
tion could provide a means for discussing the aesthetics of the work, but he also
illustrated how user input could be fed directly into a real-time visual world,
allowing the audience to be directly part of a live visual environment. Videoplace
became not only a touchstone for the idea and realisation of interactive art, it
 Liquid Visuals 79

also established the groundwork for virtual and immersive environments more
generally. A number of key artists – from Jeffrey Shaw (1944–) in the 1980s
to Rafael Lozano-Hemmer (1967–) in the 1990s and 2000s – followed up on
Krueger’s ideas and strategies, creating ever more complex works in which the
user interacted with graphics, video and eventually 3D worlds in real time. For
more information and a direct discussion of the use of Live Visuals in interactive
and immersive environments see Chapter 13.

Moving Towards the Digital II: Video Synthesizers

and Music Video
Parallel to the developments noted earlier, a number of video synthesizer
prototypes were developed in the late 1960s and 1970s. These were visibly
inspired by the various music synthesizers (such as the Moog) that were devel-
oped in the years preceding this. Early video synthesizer devices included
Myron Krueger collaborator Daniel Sandin’s (1942–) Image Processor (1971–
1974) and Stephen Beck’s (1950–) Direct Video Synthesizer (1971). Both devices
were analogue and were indebted to the modular music synthesizers of the
day, which used patch cables to connect sound modules together. “Daniel
Sandin created the Image Processor . . . a modular and programmable ana-
log computer for the treatment of video images that followed the Paik-Abe
Video Synthesizer. When connected to a camera, the Image Processor could
manipulate the video signal in various ways.”42 For an example of the layout
of Image Processor see Figure 3.7. Various visual processors are connected by
BNC cables, allowing the operator to alter the video signal projected on the
attached TV. This modular approach clearly inspired future developers of Live
Visuals devices and software, including most obviously Garagecube’s Modul8
VJ software, which allows the user to connect a number of image processing
modules in software.
A number of audio-visual pieces were developed by both Sandin and Beck,
including Sandin’s Spiral PTL (1980), Beck’s Illuminated Music (1972) with com-
poser Warner Jepson and Union (1975), “the latter dedicated to Jordan Belson,
with whom Beck collaborated.”43 Similar to the early 20th century, in which
numerous variations of the colour organ appeared, in the 1970s, a number of
variations on the video synthesizer were developed. These included Bill Hearn’s
VIDIUM (1972), David Jones’s Jones Colorizer (1974–1975) and Laurie Spiegel’s
(1945–) video and sound computing system VAMPIRE (1974–1976), developed
with Ken Knowlton of Bell Labs44:

Spiegel could draw forms with a graphic tablet and simultaneously, through
other devices, modify the image’s parameters, such as size, color and tex-
ture, and then record them. The tools used for the audio, for instance
filters, reverbs and so on, could also be used for images, and so Spiegel had
her hands on a full-f ledged audiovisual instrument.45
80 Steve Gibson

FIGURE 3.7 Sandin’s Image Processor, exhibited at School of the Art Institute of
Chicago (SAIC), with complex oscillators patch as written up by James
H. Connolly.
Source: Photo by Rosa Menkman, 2015, www.f lickr.com/photos/r00s/17016562038/. Creative
Commons.

FIGURE 3.8 Fairlight Computer Video Instrument, an early digital video synthesizer.
Fairlight is now owned by BlackMagic. Used by permission.
Source: www.blackmagicdesign.com/support/
 Liquid Visuals 81

Arguably the first (relatively) well-known commercial video synthesizer was the
Fairlight Computer Video Instrument (CVI), “an early video synthesiser devel-
oped in Australia in the 1980’s. It was intended to be a ‘video version’ of the
iconic Fairlight CMI (Computer Music Instrument).”46 The Fairlight CMI was
developed in the late 1970s and became a fixture in popular music in the 1980s.
It was the first commercial music device to allow both sampling and a primitive
version of computer sequencing. It was used extensively by artists such as Peter
Gabriel, Kate Bush, Jean-Michel Jarre and Art of Noise. The Fairlight CVI had
a similar look to the CMI (see Figure 3.8) and allowed for the application of a
number of video effects to source footage in real time. It became a ubiquitous
device for videos shows on MTV in the early days of the music video. Accord-
ing to http://fairlightcvi.com/, the following videos utilised the Fairlight CVI:

Crowded House – Now We’re Getting Somewhere (1986)

A Flock of Seagulls – Who’s That Girl? (1985)
Jean-Michel Jarre – Zoolookologie (1985)
Oscar (Sky Mitchel) Lariosa – I’m Just an Alien47

A cursory look at any of the videos (available on YouTube) will reveal the quality
of the processing capabilities of the CVI. It was capable of rudimentary compost-
ing, as well as a range of 8-bit effects, most of which are now quite dated look-
ing. In the better examples (e.g. Jean-Michel Jarre’s Zoolookologie), the effects are
relatively seamlessly used, and the compositing (while somewhat rudimentary)
is achieved somewhat believably considering the era. It is clear that a lot of work
remained to be done before digital video synthesis could be considered an aes-
thetically satisfying tool.
Undoubtedly the development of video as a format in the late 1970s and
1980s contributed greatly to the interplay between sound and image. The rise
of the music video via MTV (and others) ushered in an era in which music
was often discovered because of its visual representation. While early-era MTV
music videos are obviously not live in any sense, they plainly created a techni-
cal and conceptual language for the cross-pollination between sound and image
and undoubtedly inf luenced the next generation of scratch video artists and VJs.
Videos such as David Bowie’s Ashes to Ashes (1980) or Cabaret Voltaire’s Senso-
ria (1984) reveal sensibilities that owed a considerable amount to earlier audio-
visuals, montage and visual music, but also pointed to the future of Live Visuals
in their use of extreme effects, fast cuts in sync with the music and precise audio-
visual timing.
The inf luence of MTV video, audio-visual culture and work with video syn-
thesizers made its way into the live music domain as well. One of the more
striking examples of this was in the work of Stephen Jones (1951–) for Australian
electronic music group Severed Heads. Jones began working with video in the
mid-1970s and “and acted as the technical attendant for the Nam June Paik &
Charlotte Moorman exhibition at the Art Gallery of NSW (AGNSW) in April
82 Steve Gibson

1976.”48 In the late 1970s he worked extensively with video systems and live
audio-visual productions for both art and popular figures. He developed his own
video synthesizer in 1979, and in 1982 he joined Severed Heads as a full-time
member of the band, arguably becoming the first resident VJ for a touring band:
“Well-known as the video component of the important Australian electronic
music group Severed Heads, Jones worked with them for ten years, producing
and touring a large number of their videos. As a member of the band, he used his
custom-built video synthesiser as part of the live stage show.”49 Jones toured with
the Severed Heads throughout the 1980s and produced Live Visuals that were
genuinely performance visuals:

Unlike most other bands who incorporate visuals into their show, Jones’
contribution was very much a live performance. Rather than just project-
ing images with maybe some cross-fading and other simple manipulations,
Jones with his videosynthesiser was able to do much more sophisticated
image manipulation, such as changing key levels governing how images
were overlaid and, perhaps more significantly, the videosynthesiser (as its
name implies) generated moving patterns, often triggered by the audio. So
no two performances would have the exact same visuals, even though they
used the same base visual material.50

Having experienced one of their performances in 1986,51 I can confirm the

uniqueness of this pairing: Jones was on stage with the band (Tom Ellard and
Gary Bradbury were the musicians) and was unmistakeably on equal footing
to the musicians (as opposed to the visual performers in the psychedelic light
shows of the 1960s, who were normally positioned off-stage). Jones was able
to randomly select a large bank of found and abstract images, which he could
access in real time. These could also be composited and effected in real time: in
essence, Jones used his video synthesiser to perform the task that a VJ does in
the present era.
A historical document of Jones demonstrating his video synthesiser on Aus-
tralia’s ABC Television in 1986 is available at Tom Ellard’s YouTube page: www.
youtube.com/watch?v=MX0goKMpB4Y&feature=emb_logo. Tom Ellard has
also made some of Jones’s work with the video synthesizer available on Vimeo,
including the video for the song We Have Come to Bless this House – https://
vimeo.com/35074914 – and video of a live performance of “Petrol” from 1982 –
https://vimeo.com/39454294.
As these examples demonstrate, Jones was a prototype for the Live Visuals
performer in the 1980s, and he laid the groundwork for visuals performers for
the next generation. While other bands began to work with stage visuals in the
1980s, most of these acts did not foreground the live visualist as an equal member
on stage, and most other acts used various switching video devices to create live
mixes on stage that were relatively pre-set for each performance (see Chapter 4
for a further discussion of this).
 Liquid Visuals 83

Moving Towards the Digital III: Multimedia

As technology advanced in the late 1970s and early 1980s and audio and video
devices became more affordable and therefore ubiquitous, live music perfor-
mance became more ‘multimedia.’ The term has its original in the 1960s and
1970s performance, but entered common usage in the 1970s and 1980s when
artists such as Laurie Anderson began using multiple mediums consistently and
in a series of performances and projects. Prior to Anderson, a number of music
artists, most notably those associated with progressive rock in the 1970s, began
to incorporate film as well as theatrical elements in their performances. One
key work in this regard was Genesis’s live production of The Lamb Lies Down on
Broadway (1974–1975) in which lead singer Peter Gabriel took on the role of the
lead character from the concept LP. The stage performance of this was extremely
elaborate with extravagant lighting, baroque use of theatrical props and a com-
plex multiscreen slide projection show by artist Jeffrey Shaw:

The band Genesis commissioned The Lamb Lies Down on Broadway to

provide a pictorial multimedia backdrop to their concert performance of
the same name. Using three pairs of programmable synchronized Kodak
Carousel slide projectors, more than two thousand slides were projected
onto three screens installed over the full length of the stage behind the
band. . . . A custom-developed laser-wand enabled Peter Gabriel to point
a dynamic cone of light into the audience. A number of inf latable event
structures were also designed for this concert. Taken together, the slide
projections, laser projections, inf latable costumes and kinetic stage lighting
were a groundbreaking experiment in total theatre, merging live perfor-
mance with multimodal staging that translated Peter Gabriel’s sophisti-
cated musical narrative into a visceral, memorable multimedia experience
for the audience.52

The visual record of this event is somewhat limited (there are no official video
or DVD releases of the original 1975 performances), but the pictorial evidence
at Jeffrey Shaw’s webpage53 suggests a highly complex stage setup with multiple
screens and complex lighting that clearly has echoes in later Live Visuals per-
formances, including those of Laurie Anderson (described later), scratch video
performances (described in the next chapter) and live audio-visual performance
in general from the 1990s onward. Other bands of the era also used increasingly
complex stage setups, including Pink Floyd for their performances of The Wall
(1980–1981), which used a very elaborate stage setup including an on-stage Wall.
This was later turned into a well-known film in 1982 by filmmaker Alan Parker.
With the advent of punk in the late 1970s, this level of theatricality was gen-
erally frowned upon, and most bands stripped down their stage setups. A few
years later, in the early 1980s, with the rise of new wave and the integration
of the 1970s avant-garde into popular culture, artists such as Laurie Anderson
84 Steve Gibson

began to conceptualise complex stage shows as the de facto mode for live per-
formance. Genesis’s and Pink Floyd’s events were effectively one-offs for a single
tour (though Pink Floyd resurrected The Wall after the fall of the Berlin Wall
in 1990). Anderson, on the other hand, made a career out of being primarily
known as a multimedia performing artist: her work was best experienced in a
live setting in which the viewer was immersed in a complex multimedia experi-
ence that included music performance, dance, spoken work, theatrical props and
multiscreen projection.
Laurie Anderson was a classically trained violinist who emerged from the New
York performance art scene in the late 1970s. In 1982 she released Big Science,
featuring the track O Superman, which surprisingly went to No. 2 on the British
charts. Nonesuch Records has made the video for O Superman available at their
YouTube page: www.youtube.com/watch?v=Vkfpi2H8tOE.54 The video for this
piece became an iconic representation of the avant-garde artist in the 1980s, but
it also revealed a demonstrable pop sensibility, albeit one clearly in sync with new
wave rather than conventional rock or pop. The video was a representation of
Laurie Anderson’s multimedia renditions of the piece in performance. She plays
keyboard and sings via a vocoder. The video uses simple lights and projections
which Anderson interacts with by means of her hands and later in the video vari-
ous other body parts. The effect is quite similar to the work of live cinema artists,
as Anderson inserts herself within the projections in real time. The video also
evidences some rudimentary visual effects (compositing, multiple screens), par-
ticularly toward the end, but again represented within the context of Anderson
performing live. In live performances of the piece, one of which I witnessed, she
effectively re-created the video on stage, with all of the live video elements.
Prior to the success of O Superman, Anderson primarily played in small arts
venues, but after its surprise rise in the charts, she moved to larger venues and
ever more elaborate multimedia spectacles, such as her epic performance United
States Live (released on five vinyl records in 1983 and four CDs later) or her
video release of Home of the Brave from 1986 (available now on DVD). The latter
featured not only a group of highly skilled music performers (such as Adrian
Below and David Van Tiegham) but also had an incredibly elaborate (for the era)
projection setup, with complex imagery accompanying each live performance,
along with theatrical props, and very particular guest appearances, such as one
from cut-up writer William S. Burroughs. Both United States Live and Home
of the Brave set the stage for multimedia audio-visual performances of the suc-
ceeding two decades. Using a combination of analogue and digital projection
systems, complex lighting, as well as live electronic music, the work of Laurie
Anderson in the early 1980s was key to the development of the performed
audio-visuals. Following on from her, a generation of artists emulated the form
(if not necessarily the look and sound) of Anderson’s work, resulting in a gen-
eration of live video performances that dominated in the 1980s and 1990s.
Amongst a number of artists (Captain Beef heart, Frank Zappa, etc.) also
associated with the American avant-garde in the 1970s and 1980s, well-known
 Liquid Visuals 85

electronic avant-gardists The Residents used increasingly multimedia elements

in their work in the early 1980s and beyond. For example, their performance
of The Mole Show from 1983, included their trademark theatrical presentations
(including an array of costumes), as well as multiscreen video, several danc-
ers and a complex stage setup where the band played behind a burlap scrim. 55
These were followed by other more elaborate stage performances, as well as a
series of specially produced CD-ROMs and DVDs in the 1990s. Other groups
in Europe and the UK associated with either avant-garde or new wave, such
as Cabaret Voltaire, Kraftwerk and Gary Numan, also used complex stage
setups and projections in their works in the 1980s and continued to do so in
the coming decades. It was in this era that the idea of audio-visual multimedia
performance was born, and it finds echoes through to the present day where
almost every major touring band has some sort of projection element.

The Rise of the VJ: Merrill Aldighieri

An unquestionably key development that occurred in the early 1980s was the estab-
lishing of the club-based visual jockey (VJ) as a clearly defined performer. The first
artist to use the term VJ was Merrill Aldighieri (birthdate unknown–) in relation
to her “pioneering work creating an improvised live video feed to interpret the DJ’s
music at the legendary club Hurrah”56 in New York in 1980–1981. While almost
forgotten in the late 1980s and 1990s, with the rise of VJing in the 21st century, her
work has come to the forefront again. Over the course of two years at Hurrah she
created live visual performances for an incredible number of punk, post-punk and
new wave acts, which have been collected into a documentary called V.J. Diaries:

Her historic filming of the live performances of punk, new wave, jazz and
industrial music form a diary marking her time at this seminal club. . . .
More than 200 hours of live music performances were recorded by Aldigh-
ieri during a year at Hurrah (1980–1981). Excerpts from 30 different live
shows include legendary bands like New Order, Mission of Burma, Maga-
zine, Gang of Four, The Psychedelic Furs, A Certain Ratio, and Bush
Tetras, as well as interviews with Cynthia Sley, Jim Fouratt, Anita Sarko,
Dee Pop, Ron Jagger, Bill Laswell, Jim Jarmusch, Stephanie Kaye, Alan
Vega, George Wrage and many more, blended with rare archival footage
and visually daring animation inspired by her nightly VJ improvisations.57

Aldighieri had been trained as a filmmaker and moved to New York in the
1970s, where she worked on The Muppet Show. In 1980 she began to work at the
Hurrah nightclub:

HURRAH was the first club to make a video installation as a focal point of
the club environment, but until I came they were just turning it on occas-
sionally [sic] to play films. I asked if I could experiment to create a real-time
86 Steve Gibson

constant f low of visuals to work with the DJ’s music so when my film played,
the f low would not stop. When they offered me my first paycheck, the word
VJ was born as we looked for how to note what I was doing.58

Her approach was aided by an enormous bank of video clips she had at her
disposal and were mixed in real time to various DJs, as well as bands and per-
formers. As such she became demonstrably the first club VJ, and while this may
be apocryphal, it has been claimed that the “MTV founders came to this club
[HURRAH] and Merrill introduced them to the term and the role of ‘VJ’,
inspiring them to have VJ hosts on their channel the following year.”59
Regardless of the accuracy of this claim, there is no doubt that the Aldighieri
was the first to perform as a visualist in a club atmosphere, and therefore can reason-
ably be dubbed the godmother of visual jockeying. “Extracts from the LIVE AT
HURRAH video archive made by Merrill Aldighieri that will serve as the basis for
a series of documentaries on her pioneering experience in music-video in the early
80s” are available in a collection at 2 Live at HURRAH.60 Certainly her influence
lived on (either accidently or by design) in the scratch video of the succeeding years,
and naturally in the club VJing came to the forefront in the 21st century.

Conclusion
The period from 1955 to 1985 was undoubtedly a period of substantial change,
both in the arts and sciences. In general, it was a period of considerable tech-
nological advancement, though these developments were achieved primarily
through analogue means (with some obvious exceptions, as outlined earlier).
Towards the middle of the 1980s, the digital became more prominent when the
MIDI specification was solidified in 1985. Combined with the broad dissemi-
nation of the personal computer in the late 1980s by Apple, IBM and Atari,
this marked a significant shift from the previous primarily analogue work
in which much live interaction was done by hand or with devices that were
extremely large, often unreliable and had no capacity to save configurations.
Clearly the work of artists as diverse as the Whitneys, Silver Wing Turquoise
Bird and Laurie Anderson fed into the next generation of live visual and live
audio-visual performers, at least in terms of concept; however, as the 1980s
moved on, analogue devices were slowly replaced by digital ones. The result was
that a certain repeatability was achievable that was previously elusive. In addi-
tion, digital editing allowed for greater audio-visual precision, which was taken
up by the next generation of live visualists, audio-visual artists and live cinema
performers. This leads us to our next discussion on scratch video and rave.

Notes
1 Léon McCarthy, “Live Visuals: Technology and Aesthetics,” Chapter 8 in Steve Gibson,
Stefan Arisona, Donna Leishman and Atau Tanaka, Live Visuals: History, Theory, Practice
(London: Routledge, 2023).
 Liquid Visuals 87

2 Paul Griffiths, A Guide to Electronic Music (New York: Thames and Hudson, 1979), 12.
3 Paul Griffiths, A Guide to Electronic Music (New York: Thames and Hudson, 1979), 13.
4 Paul Griffiths, A Guide to Electronic Music (New York: Thames and Hudson, 1979), 18–19.
5 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 55.
6 James Whitney, Yantra, 1957, Chapadão do Formoso YouTube page, www.youtube.
com/watch?v=nvWwlZSXaR0.
7 John Whitney, Digital Harmony: On the Complementarity of Music and Visual Art (King-
sport, TN: Kingsport Press, 1980), 184.
8 Zabet Patterson, “From the Gun Controller to the Mandala: The Cybernetic Cinema of
John and James Whitney,” Grey Room, 36, Summer 2009, 36–57 (Grey Room, Inc. and
Massachusetts Institute of Technology, 2009), 37.
9 Gene Youngblood, Expanded Cinema (New York: E.P. Dutton & Co., 1970), 194.
10 Zabet Patterson, “From the Gun Controller to the Mandala: The Cybernetic Cinema of
John and James Whitney,” Grey Room, 36, Summer 2009, 36–57 (Grey Room, Inc. and
Massachusetts Institute of Technology, 2009), 38.
11 James Whitney, Lapis, 1966, Chapadão do Formoso YouTube page, www.youtube.com/
watch?v=kzniaKxMr2g.
12 William Moritz, “Digital Harmony: The Life of John Whitney, Computer Animation
Pioneer,” Animation World Magazine, (2.5), August 1997, www.awn.com/mag/issue2.5/
2.5pages/2.5moritzwhitney.html.
13 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 74.
14 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 76.
15 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 76.
16 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 77.
17 Nam June Paik, Videa ‘n Videology 1959–1973 (New York: Emerson Museum of Art,
Syracuse, 1974), 55.
18 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 82.
19 George Fifield, “The Paik/Abe Synthesizer,” The Early Video Project, website, 2000,
http://davidsonsfiles.org/PaikAbeSythesizer.html.
20 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 78.
21 A documentation of Exploding Plastic Inevitable can be seen on tomorrowpictures.tv, http://
tomorrowpictures.tv/radio-tv-film/VNE4lSYAACcNQSvL/exploding-plastic-inevitable.
22 Kerry Brougher, “Visual Music Culture,” in Kerry Brougher, Jeremy Strick, Ari Wise-
man and Judith Zilczer, Visual Music: Synesthesia in Art and Music Since 1900, organized
by Kerry Brougher (London: Thames and Hudson, 2005), 159.
23 Kerry Brougher, “Visual Music Culture,” in Kerry Brougher, Jeremy Strick, Ari Wise-
man and Judith Zilczer, Visual Music: Synesthesia in Art and Music Since 1900, organized
by Kerry Brougher (London: Thames and Hudson, 2005), 159–160.
24 Kerry Brougher, “Visual Music Culture,” in Kerry Brougher, Jeremy Strick, Ari Wise-
man and Judith Zilczer, Visual Music: Synesthesia in Art and Music Since 1900, organized
by Kerry Brougher (London: Thames and Hudson, 2005), 161.
25 Kerry Brougher, “Visual Music Culture,” in Kerry Brougher, Jeremy Strick, Ari Wise-
man and Judith Zilczer, Visual Music: Synesthesia in Art and Music Since 1900, organized
by Kerry Brougher (London: Thames and Hudson, 2005), 161.
26 Kerry Brougher, “Visual Music Culture,” in Kerry Brougher, Jeremy Strick, Ari Wise-
man and Judith Zilczer, Visual Music: Synesthesia in Art and Music Since 1900, organized
by Kerry Brougher (London: Thames and Hudson, 2005), 162.
27 Kerry Brougher, “Visual Music Culture,” in Kerry Brougher, Jeremy Strick, Ari Wise-
man and Judith Zilczer, Visual Music: Synesthesia in Art and Music Since 1900, organized
by Kerry Brougher (London: Thames and Hudson, 2005), 166.
28 Kerry Brougher, “Visual Music Culture,” in Kerry Brougher, Jeremy Strick, Ari Wise-
man and Judith Zilczer, Visual Music: Synesthesia in Art and Music Since 1900, organized
by Kerry Brougher (London: Thames and Hudson, 2005), 166.
29 Gene Youngblood, Expanded Cinema, Introduction by R. Buckminster Fuller (New
York: P. Dutton & Co., Inc., 1970), 394.
88 Steve Gibson

30 Gene Youngblood, Expanded Cinema, Introduction by R. Buckminster Fuller (New

York: P. Dutton & Co., Inc., 1970), 183 and ff.
31 Guy Sherwin, Man with Mirror, 1976, BD Films YouTube page, www.youtube.com/
watch?v=DX1-xuCNIeg.
32 Tony Hill Films Official website, “Installations,” www.tonyhillfilms.com/installations.
33 Tony Hill Films Official website, “Rigs,” www.tonyhillfilms.com/rigs.
34 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 79.
35 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 78.
36 “Small Planet Myron Krueger,” The Interaction ’97: Towards the Expansion of Media Art
conference website, www.iamas.ac.jp/interaction/i97/artist_Krueger.html.
37 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 80.
38 See Myron Krueger, Videoplace 1975 for several videos documenting the project, https://
aboutmyronkrueger.weebly.com/videoplace.html.
39 Nam June Paik, Participation TV II, video-closed-circuit, three cameras combined with a
Paik/Abe synthesizer and four monitors, Galleria Bonino, New York 1971: Davis: Experi-
ment 1975, p. 189; Decker: Paik 1988, p. 65s., 151; Kacunko: Circuit 2004, p. 187s.
40 Myron Krueger, [Artificial] Reality (Boston: Addison-Wesley, 1991), 105ss.
41 Thomas Dreher, History of Computer Art, IASLonline Lessons/Lektionen in NetArt,
2013–14, http://iasl.uni-muenchen.de/links/GCA_Indexe.html.
42 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 80.
43 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 82.
44 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 82–83.
45 Adrano Abbado, Visual Music Masters (Milan: Skira Editore, 2017), 83–84.
46 VJzoo, Fairlight Computer Video Instrument “About” page, http://fairlightcvi.com/
wordpress/?page_id=2.
47 VJzoo, Fairlight Computer Video Instrument “Home” page, http://fairlightcvi.com/
wordpress/.
48 Steven Jones, Scanlines: Media Art in Australia Since the 1960s webpage, http://scanlines.
net/node/1704.
49 Steven Jones, Scanlines: Media Art in Australia Since the 1960s webpage, http://scanlines.
net/node/1704.
50 Severed Heads Biography, LTM Recordings official web page, www.ltmrecordings.com/
severed_heads.html.
51 See http://sevcom.nilamox.com/photo/1986%20The%20World/tour/index.html for
images from the 1986 Severed Heads tour.
52 Jeffrey Shaw, The Lamb Lies Down on Broadway, Jeffrey Shaw Compendium website, 1975,
www.jeffreyshawcompendium.com/portfolio/the-lamb-lies-down-on-broadway/.
53 Jeffrey Shaw, The Lamb Lies Down on Broadway, Jeffrey Shaw Compendium website, 1975,
www.jeffreyshawcompendium.com/portfolio/the-lamb-lies-down-on-broadway/.
54 Laurie Anderson, O Superman (Official Video) Nonesuch Records YouTube page, 1982
(remastered 2007), www.youtube.com/watch?v=Vkfpi2H8tOE.
55 The Residents, Mole Show (1983), the Residents Official Site, www.residents.com/historical/?
page=moleshow.
56 Merrill Aldighieri V.J. Diaries, Howl Arts website, 2017, www.howlarts.org/event/merrill-
aldighieri-v-j-diaries/.
57 Merrill Aldighieri V.J. Diaries, Howl Arts website, 2017, www.howlarts.org/event/
merrillaldighieri-v-j-diaries/.
58 Merrill Aldighieri, Merrill Aldighieri Resume, ARTCLIPS the Multi-Media Dream Palace of
Merrill Aldighieri, http://artclips.free.fr/cv_eng_merrill.htm.
59 VJing, HisoUR website, www.hisour.com/vjing-2984/.
60 See Live at Hurrah, video archive made by Merrill Aldighieri, www.youtube.com/playlist?
list=PLF17B5AD4D07B27C2. Also see www.howlarts.org/event/merrill-aldighieri-v-
j-diaries/ for a trailer of her documentary V.J. Diaries.
4
SCRATCH VIDEO AND RAVE
The Rise of the Live Visuals Performer
(1985–2000)

Léon McCarthy and Steve Gibson

Introduction
During the 1980s and 1990s the aesthetics and practice of Live Visuals entered the
mainstream, leading to better acknowledgement of the ‘Live Visuals performer’
as an important creative role. The rise of rave culture also created changes in
audience experience – being in a large, disinhibited dance-focused crowd now
demanded an alternative form of visual spectacle to replace the previous tradition
of the stage as the place of visual focus. Simultaneously the growth in stadium-
scale gigs also instigated the need to create more ambitious visuals, allowing
Live Visual performers to work on larger-scale and visually more innovative
productions.
These new contexts also provided live visualists new opportunities to pres-
ent contemporary socio-political subject matter to their audiences; themes such
as Thatcherism, Reaganomics, mass consumption and environmental degrada-
tion were commonly addressed within the audio-visual content of this period.
Interestingly, while mass media was a site for their criticism, for some artists, it
also became a channel through which they found a more mainstream voice – the
prime example being through the creation of music videos that were then seen
on MTV (as outlined in Chapter 3).
Developments in technology played an important role in driving these
changes. The emergent media technologies of the VHS video, the Panasonic
MX10 Digital Video Mixer and rudimentary computer-based video editing
greatly facilitated new forms of creation and production. As is argued in Chap-
ter 8, such experimental use of technology helped shape the visual aesthetic that
emerged during this era. By reappropriating broadcast content on VHS tapes,
irony and juxtaposition came to underpin the aesthetic; these approaches are also
reminiscent of the ‘montage’ technique used by Sergei Eisenstein (as discussed in
Chapter 2). By combining broadcast content with computer-generated graphics
DOI: 10.4324/9781003282396-6
90 Léon McCarthy and Steve Gibson

and text, visual artists could elect to be more narratively ambitious. Later hard-
ware developments afforded better real-time computer processing, which when
harnessed, allowed for the generation of ‘audio-responsive’ visuals. For Live
Visual performers, the development of better interactive interfaces made it pos-
sible to utilise all the new advantages of desktop computers and software into
their stage productions.
Parallel to these developments, the acid house movement began in Chicago
in the mid-1980s and moved quickly to the UK, culminating with the “Second
Summer of Love” in 1987.1 Acid house was a simple form of dance music, with
a pulse that was driven by the iconic Roland TB-303 synth. Acid house fed
directly into the establishment of rave culture, which was christened by former
Throbbing Gristle/Psychic TV singer Genesis P-Orridge in 1989.2 Rave usu-
ally consisted of illegal events in huge venues, where drugs were consumed and
hedonistic dancing took place to electronic music. Though initially driven by
acid house, other forms of electronic music such as drum and bass and techno
became common at raves in the 1990s. The rave, while not always employing
Live Visuals, often did have psychedelic qualities achieved through lighting that
brought to mind the equally psychedelic events of the 1960s (see Chapter 3 for
more on psychedelic events such as The Joshua Light Show’s performances and
Chapter 11 for more on the hedonism of rave culture).
This chapter will select and discuss some of the most pioneering live-visual
performers of the 1980s and 1990s as examples to illustrate what happened when
culture was mediated through emerging technology. The chapter will begin in
the UK with scratch video, stay in the UK to discuss rave culture, move to the
United States to discuss Emergency Broadcast Network (EBN) and then return
to the UK to consider Coldcut.

Scratch Video

Sampling Mass Media

Throughout most of the 20th century, commercial and theatrical moving-image
content was produced by a small professional elite for mass consumption, deliv-
ered according to a schedule outside the control of the consumer. This held true
whether the viewer was consuming content via the cinema screen or the home
TV set. Today, due to internet-based streaming services, we take the idea of ‘on-
demand’ for granted, but it was the arrival of cheap VHS video technology that
first brought the concept of on-demand to the masses.
During the 1970s video editing technologies had been in use by the broad-
cast industry; multiple video sources were queued in sync, ready to play content
through a video mixer via an edit decision list (EDL), with the output broadcast
as live or recorded for later delivery. As well as being able to cut and splice dis-
parate content together, video mixers made it possible to add colour filters, pat-
tern effects, cross fades, picture-in-picture and layering via chroma-key mixing.
 Scratch Video and Rave 91

While of a lesser fidelity than 35mm film, video editing made the process of
editing much faster and so had a particular utility for live TV broadcast.
During the 1960s and 1970s some artists such as Nam June Paik (also discussed
in Chapter 3) had experimented with video, recontextualising the meaning of
the original content in order to comment on the nature of mass media, distribu-
tion and consumption. Across music, fashion and art similar reappropriation pro-
cesses were emerging, contributing to the postmodern perception or belief that
content is not fixed; rather, it has multiple meanings depending on the context
of its consumption and the interpretation of the receiver.
For video artists in the 1980s, opportunities to work with new production
technologies were rare, as the cost of working with video was prohibitively
expensive; the mechanisms for producing video remained within the confines of
the TV studio. As outlined in Chapter 3, the release of cheaper VHS video decks
gave consumers the opportunity to record content that had been made for live
broadcast, breaking the reliance of the live schedule set by cable TV and usher-
ing in the concept of personally determined playback. Cheaper video technol-
ogy also made it easier for video artists to sample and recontextualise broadcast
content. It was when a group of London-based video artists began using VHS
decks in this way that their styles, processes and aesthetics converged to become
known as the ‘scratch video’ movement. Andy Lippman offers an apt descrip-
tion of this movement: “If television is our shop window on the world, scratch
has just chucked a brick through it, and is busy looting 30 years of goodies, with
abandon.”3
There are many ways in which scratch video had an impact on the develop-
ment of Live Visual performance practice, three of which will now be discussed:
the processes it established, the aesthetics that emerged and the contexts in which
it found an audience.

The Video Mashup

When a new media form emerges, it first tends to borrow techniques, tools
and semiotic principles from existing forms in a process that Phillip Auslander
termed “remediatisation.”4 As scratch video artists were using the same format
and technologies as TV video editors, they first explored some of the same tech-
niques and processes, including video effects, compositing, chroma-keying and
text overlays. However, unlike TV video editors, video artists were not restricted
to short formats and strict content codes; they had complete creative freedom to
explore the meaning inherent in the content they sampled.
Several stylistic approaches emerged from scratch video that went on to influence
music video, graphic design, motion graphics TV and ultimately Live Visual per-
formance practice. In terms of style, scratch video is most associated with the ‘video
mashup;’ a style that juxtaposes disparate video clips in order to create new mean-
ing. In the same way that Andy Warhol’s recontextualisation of popular graphics
lent itself to a form of irony, the video mashup similarly turned the meaning of the
92 Léon McCarthy and Steve Gibson

original content on its head, creating a condition whereby “broadcast TV is scoured

for arresting images and fed into video editing systems like shredding machines.”5
The juxtaposition of disparate video clips has echoes of Sergei Eisenstein’s
montage theory,6 establishing his theory in a postmodern context went on to
inf luence the 1980s music video, such as Cabaret Voltaire’s video for Sensoria
from 1984,7 as well as spilling over into visual advertising aesthetics. However, of
more relevance to this book’s theme is the inf luence that it had on visual perfor-
mance practice, a thread that can be woven from scratch video (VHS) through
to artists such as Eclectic Method (DVD) and onto the current approach of many
laptop-based VJs (digital video).
Rhythmic editing made such visuals a natural bedfellow to the dance music
that was emerging at the time, music that was based on the drum machine and
sequencer. For this reason, scratch video artists tended to perform in venues and
nightclubs such as London’s The Fridge and The Electric Ballroom or Manches-
ter’s Haçienda rather than art galleries (where the separate genre of video art
had tended to reside). The process of manipulating VHS tapes in real time with
vision mixers and effects meant such scratch video artists were already ‘live’ by
nature, and so the transition to the nightclub was seamless.
While most scratch video artists explored the video mashup, perhaps the
best-known early proponents of such a style were The Duvet Brothers. One of
their most well-known videos was the creation of a mashup to accompany the
New Order single “Blue Monday” (see Figure 4.1).8 From the opening sequence
entitled The Voice of Britain, they take a swipe at all that they saw was going
wrong with Britain: the rise of consumerism, the death of the working man and
the heavy hand of the police state. Their process was to juxtapose short loops
of archival footage against sampled TV content, with occasional text used to
emphasis the irony inherent in their visual narrative. Cultivation theory 9 details
how people who are regularly exposed to media (particularly TV) for long peri-
ods of time can come to absorb the artificial representations of social realities that
are presented, and in turn this can inf luence the viewer’s behaviour and beliefs.
By plundering TV content for arresting and/or provocative images, these artists,
through their use of juxtaposition, amplify the constructed nature of TV and its
programmed content.

Texture and Effect

Scratch video was not only about fast temporal editing and juxtaposition; like
earlier experimental film artists who explored the materiality of the filmstrip
(Norman McLaren, Stan Brakhage, etc.), some scratch video artists developed
an interest in the textural nature of analogue videotape. Anyone familiar with
consumer-grade VHS tape decks will remember the interference patterns that
would appear on screen, particularly when playback was at a non-standard rate.
While regular users and consumers would have seen this as an annoyance, artists
in turn often celebrated and foregrounded interference as part of the inherent
 Scratch Video and Rave 93

FIGURE 4.1 The Duvet Brothers, still from the video Blue Monday (1984); an example
of sampled footage chroma-keyed and composited with text. Used by
permission.

materiality of the medium. These artists might keep such artefacts in their edit or
might exaggerate the appearance of such with colour saturation filters and zoom
effects, all video processes that were readily available to video artists in the 1980s.
In Absence of Satan by George Barber, we see an example of a hyper-saturated
sequence that, due to the process applied, enhances any presence of interlacing
and interference (as can be seen in Figure 4.2).
By combining video effects, video artists could further reveal the nature of
the analogue video patterns on tape; in the hands of scratch video artists such
as George Barber, this often revealed a rough beauty unique to the medium.
Figure 4.3 shows a frame from the compilation The Greatest Hits of Scratch Video
Vol. 110 in which the artist is doing just this. This image also reveals another
technique used by many scratch video artists but initially pioneered by George
Barber – the use of the chroma-key effect to composite two video sources as a
moving image montage.
Iconography featured across the work of some scratch video artists; careful
use of the chroma-key effect gave some video montages the appearance of stop-
frame animations. Once content destined for the TV screen was reappropriated,
the irony inherent in the tropes typically used in mass media advertising would
94 Léon McCarthy and Steve Gibson

FIGURE 4.2 George Barber, still from the video Absence of Satan (1985), an example
of video effects heightening the interlacing artefacts inherent to the
medium.
Source: Courtesy of George Barber and LUX London. Used by permission.

FIGURE 4.3 George Barber, still from the video The Greatest Hits of Scratch Video Vol
1, 1980s, an example of composited footage treated with video effects
that heighten the materiality of the medium.
Source: Courtesy of George Barber and LUX London. Used by permission.
 Scratch Video and Rave 95

emerge and/or be placed centre stage in the work. A critical awareness of the
insidious nature of advertising began with Dada video artists and continued with
scratch video artists such as Jeffrey Hinton; in the following excerpt (taken again
from The Greatest Hits of Scratch Video), we see how he has looped segments of
late-night cable TV adverts in a way that points out the absurdity of such produc-
tions. This is a stylistic sensibility that can be seen later in the work of EBN and
many contemporary live-visual performers such as The Light Surgeons, Polar
Fantasy and Richard Curtis.

The Scratch Video Context

Scratch video artists were not initially given a voice or place in the traditional
gallery context; instead, these new formats were explored through site-spe-
cific installations, and performances. While the most well-known videos by
The Duvet Brothers were short-form mashups, they also explored longer-
form narratives, producing several multiscreen installations that took the form
of large TV walls erected on scaffold structures. As these installations were
live, scratch video artists began adapting the aesthetic for the live stage to
create Live Visual backdrops for bands such as Psycho Circus and Sigue Sigue
Sputnik.
The artists known as Gorilla Tapes explored a format reminiscent of the hour-
long TV documentary, with their most well-known work being their Invisible
TV production for Channel 4, a long form ‘mockumentary’ created from a mix
of sampled TV, archive footage and purposely produced content.11 These longer
formats were less suited to the repetitive use of short loops and instead featured
live original sound and the presence of actors. In many ways Invisible TV was
a precursor to what we now call live cinema, aligned with more recent works
such as SuperEverything* by The Light Surgeons (for a further discussion of live
cinema, see Chapter 12, and for an interview with Chris Allen of The Light
Surgeons, see Chapter 17).

Rave Culture
In the 1980s the emergence of the nightclub as a more mainstream environ-
ment created a demand for a new form of visual spectacle. It was for an audience
focused on dancing that Live Visual performers established their processes, aes-
thetics and narrative forms. It could be argued that it was the emergence of a new
medium (VHS) and an alternative context (nightclub) that created the breeding
ground for the development of Live Visual performance practice.

A Broadcast Movement
The scratch video movement did not emerge from the traditional gallery setting,
but rather from nightclubs and community dancehalls. This shift in context to
96 Léon McCarthy and Steve Gibson

a different type of audience was profound, as it meant scratch video artists had
to establish a semiotic language suited to mainstream communication. This was
not hard; by sampling mainstream TV content, they were piggybacking upon
the codes of mass media. It is no surprise that their hijacking and subversion of
the semiotics of mass media led the TV and advertising industries to reappropri-
ate its graphical stylisation to communicate to the youth culture of the 1980s
and 1990s.
It wasn’t only in London that a visual spectacle was being created in night-
clubs; as outlined in Chapter 3, in 1980 the New York nightclub Hurrah had
installed a TV wall and invited video artist Merrill Aldighieri to use it. She
went on to produce live video content to accompany the DJs who performed
there with a two-year residency leading to her being termed the world’s first
‘VJ.’ While the term was then appropriated by MTV to define their video hosts,
Aldighieri was the first individual to actually be cited as a live visualist.12 As
Chapter 3 has illustrated, she created visual performances for an enormous range
of musical artists from the post-punk scene. Other live visualists followed, such
as Stephen Jones for Severed Heads, and they helped to establish the notion of
the ‘performed image.’ This was an integral component of 1990s that in part was
driven by rave and techno culture, which in turn was dominated by DJ-based
electronic music alongside developments in stage performances with live music.

A Spectacle for Dance

Why were nightclub owners spending great sums of money on installing
cutting-edge VHS systems? A cultural shift was taking place in the entertain-
ment industry; dance-based electronic music and drugs that enhanced the expe-
rience of dancing were leading audiences away from live musical performances
to nightclubs. The absence of a group of musicians on stage meant a loss of visual
focus and traditional spectacle, so a form of visual projection was sought that
could augment the experience of the dancers. As we have seen in Chapters 2
and 3, Live Visuals had already been used as an accompaniment to live musical
performance. As outlined in Chapter 3, in the 1960s and 1970s visual artists such
as Silver Wing Turquoise Bird and The Joshua Light Show used oil lamps and
slide projectors to augment the spectacle of the band. In the 1980s the music,
the drugs and the environment had changed and so the visual spectacle changed
with it. Across all forms of dance-orientated electronic music (such as the afore-
mentioned acid house, techno, industrial and drum and bass), drugs were often
consumed in a nightclub or rave environment, so it was logical that Live Visuals
began to make up for the loss of the visual spectacle of live musicians.

A Setup for the Nightclub

Thirty-five millimetre cinema projectors were more than adequate for projec-
tion in dark environments, but it was on 16mm film that visual artists tended to
 Scratch Video and Rave 97

work, and these projections would have little impact in the darkness of a night-
club environment. TV walls had been experimented with by artists such as Nam
June Paik; these were brighter but required the use of expensive analogue video
sources. It was not until video artists began tinkering with VHS sources that an
opportunity arose to install TV walls in nightclubs.
As outlined earlier, video artists created bespoke videotape systems and used
video synthesizers and video mixers to suit the context of the nightclub (rather
than the TV studio), but in the nightclub it took time to replicate the same pro-
cesses that worked in the TV studio. It should then come as no surprise that many
video artists established long-running residencies in venues; they were integral
to developing the installed video systems and were one of the few who could
operate the same. This was the case with Aldighieri, but also many of the scratch
video artists in London.

Rave and Acid House

A DJ in a nightclub will often respond spontaneously to the crowd, so it cannot
be foreseen what track they will choose to play next. For the video artist inter-
ested in bringing themes to an audience, the challenge of spontaneity is one that
nightclub VJs still grapple with to this day. As they refined their practice and
style, it was no surprise that some video artists began to collaborate with the DJs.
As discussed in the introduction, Genesis P-Orridge from Psychic TV was the
first person to use the term rave in 1989.13 Psychic TV emerged from the breakup
of the punk-industrial band Throbbing Gristle. Both groups dealt in extreme
subjects ranging from politics to sexuality; by bringing a visual element to the
stage Psychic TV were able to broach such subjects directly. They emerged as
part of the UK’s acid house movement, which itself was heavily inf luenced by
the house scene in Chicago.
Rave itself as a movement embodied a collective hedonism and became a cul-
tural fixture in the late 1980s and 1990s in the UK and the United States. Venues
such as the Mud Club in London and the Haçienda in Manchester (nationally
known as ‘Madchester’) became key sites of rave music, but more commonly rave
events were illegal and organised secretly in abandoned post-industrial spaces
and outdoor settings (see Figure 4.4). The artist Jeremy Deller describes this act
of occupying disused factory spaces as “dancing on ruins,” a cathartic “death rit-
ual to mark the transition of Britain from an industrial to a service economy.”14
Rave culture ultimately created a moral panic in Britain15 but also in the United
States and Canada.16 It is not an exaggeration to say that the techno movement
of the 1990s almost assuredly owed its (original and continuing) prominence due to
the advent of the rave and rave culture. Acts such as Prodigy, Orbital, the Orb,
the Stone Roses and the Chemical Brothers became quasi-mainstream exemplars
of rave and techno culture, transcending their beginnings in the underground
scene of the 1990s. These acts also began experimenting with visual elements as
part of their musical performances, many of which were simply DJ-based sets
98 Léon McCarthy and Steve Gibson

FIGURE 4.4 Fantazia Summertime Rave, May 1992, Bournemouth Matchams Park
Altjunglist.
Source: CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/4.0>, via Wikimedia Commons.

with vocals and multicoloured lights (although acts such as Orbital controlled
their music via MIDI-enabled control interfaces, drum machines and sequenc-
ers, as well as live performance). It should also be said that rave also had its own
distinct visual style and identity – for example, the ubiquitous adoption of glow-
sticks or pacifiers and soothers alongside backpacks were common identifiers of
being part of the rave scene. The more dominant visual style was ‘trippy’ and
harkened back to the earlier psychedelic movement, though now with a techno-
logical f lavour. Running concurrently with rave were the inf luencing subcul-
tures of techno-utopian psychedelia (Timothy Leary) and cyberpunk (William
Gibson) as encapsulated by Mondo 2000 magazine.17
The moral panic about rave culture reached a zenith in 1994 with the advent
of Section 63 of the Criminal Justice Act in the UK,18 part 1 of which reads as
follows:

63 Powers to remove persons attending or preparing for a rave . . .

(1) This section applies to a gathering on land in the open air of [F120]
or more persons (whether or not trespassers) at which amplified music
is played during the night (with or without intermissions) and is such
 Scratch Video and Rave 99

as, by reason of its loudness and duration and the time at which it is
played, is likely to cause serious distress to the inhabitants of the local-
ity; and for this purpose –
(a) such a gathering continues during intermissions in the music and,
where the gathering extends over several days, throughout the
period during which amplified music is played at night (with or
without intermissions); and
(b) “music” includes sounds wholly or predominantly characterised
by the emission of a succession of repetitive beats19

While the new legislation had the partial effect of shutting down illegal raves,
the music of rave culture had entered the quasi-mainstream through groups such
as Prodigy, the Chemical Brothers and Orbital becoming superstars. The music
and visual culture of techno clearly survived this clampdown and set the stage
for the music and visual style of the 21st century. In addition, the performance
format of the rave (DJs supported by live visualists or light artists) was the ante-
cedent of the present club scenario (DJs supported by VJs). In addition, many of
the acts associated with the initial rave movement evolved to become the audio-
visual stars of the 21st century and helped establish the broader audience for Live
Visuals performance.

Emergency Broadcast Network

Going Live and Large

As discussed earlier, by the end of the 1980s a more defined form of Live Visual
performance practice was emerging. During that decade, scratch video had estab-
lished an aesthetic that was both accessible to artists and appealed to mainstream
audiences, while the growing popularity of dance-orientated clubs and venues
provided more opportunities for video artists to become live video perform-
ers. As the scene matured, its aesthetics were refined; narratives became more
elaborate, production standards rose and the scale of events increased. This wider
acceptance of the practice saw some video artists move beyond the realm of per-
formance into art galleries, public installations and motion graphics production.
Whereas some video performers returned to their original source of critique,
that of television and mass media, the collective EBN traced such a trajectory.

The EBN Aesthetic

In 1991 a group of graduates from Rhode Island School of Design released a
satirical video mashup of sampled cable TV content related to the US invasion of
Iraq. This group came to be known as Emergency Broadcast Network (EBN),
and according to author Holly Willis, much of modern VJing can be traced back
100 Léon McCarthy and Steve Gibson

to the practices EBN developed.20 EBN’s method of sampling video content was
heavily inf luenced by the scratch video movement; however, their work pushed
the boundaries of video editing with a tighter coupling of the sonic and visual
tracks. Audio loops of similar beats per minute (BPM) were aligned to form a
temporal grid over which disparate video loops were then aligned. They often
over-dubbed the lyrics in the original audio track with sampled lyrics from the
presented video clips. Another style of theirs was to intertwine custom-produced
clips of EBN’s frontman ( Joshua Pearson), which meant they could create and
exercise more directorial control over the perceived narrative. Despite the often-
disturbing nature of the video content, the result was a form of ironic audio-
visual entertainment.

Scale and Impact

Early scratch video work would not have been acceptable for live TV broadcast
due to the nature of its content and the quality of the productions (based as they
were on sampled VHS). However, the live arena was where many artists built
an audience. In 1991 EBN travelled with the Lollapalooza Festival displaying
their Gulf War anti-narrative on a bank of TVs rigged to a vehicle.21 EBN’s
provocative staging showcased the power of their medium to make statements
and communicate their intentions, which inf luenced many contemporary VJs to
make the shift from laptop performance to the more controlled and narrative-
orientated format of live cinema. To return to the words of Holly Willis: “Rather
than appearing as moving wallpaper designed to enliven a bands performance,
EBN demonstrated that VJing could be political in orientation.”22
This use of the medium as a site for political statement attracted the interest
of U2, who hired EBN to take part in producing the live stage show for the
1992–1993 Zoo TV world tour. By playing a central creative role in such a high-
magnitude production meant that other creative avenues opened up for them,
including various projects with MTV, both in terms of broadcast music videos
and live MTV Music Awards. In 1997 EBN rejoined U2 in a collaboration to
create a limited-run series for MTV called Zoo TV: The Television Program.23

EBN: Their Legacy

EBN are an example of a collective who emerged from the scratch video move-
ment to then cross into other forms and contexts: video art, live performance,
production design, music video direction and TV broadcasts. However, the
members of the collective were not just artists and performers; their experimen-
tation across formats led them to develop new processes and tools. Their music
album Telecommunication Breakdown was one of the first works released for the
‘enhanced CD’ format, and while it wasn’t a format to gain mainstream popular-
ity, such early works shaped how interactive multimedia work was produced for
DVD. Both during their time in EBN and in their later careers, members from
 Scratch Video and Rave 101

the collective developed bespoke tools; Brian Kane developed one of the world’s
first video sampling software Vujak,24 while Greg Deocampo went on to set up
the Company of Science and Art, whose legacy was the creation of AfterEffects
which has become one of the most popular software platforms for the produc-
tion of motion graphics. Not only is AfterEffects a tool suited to the needs of VJs
(animating, preparing and compositing loops), it is a tool used across the TV and
cinema production industries.

Coldcut

Going Digital
Video had been in use by the broadcast industry since the 1970s, but it was the
arrival of the cheaper VHS format that facilitated the development of scratch
video and the emergence of a live video performance practice. In a similar way
microchip technology always had the potential to change how Live Visuals were
produced and performed, but computers did not have an impact until the 1990s
when the desktop computer became readily available. While the most obvious
applications for computers were in business, some visionaries saw the creative
potential that lay within the microchip. As discussed in Chapter 3, the ana-
logue art of James Whitney foresaw such a future, and as the 1990s approached
tech-savvy multimedia artists increasingly started to use the computer for artistic
purposes. Matt Black and Jonathan More are a duo who work under various
monikers, most notably as Coldcut. Black and More helped define how the com-
puter could be a tool for Live Visual artists across both production, interaction
and performance.

Coldcut’s Music
Matt Black (originally a computer programmer) and Jonathan More (formerly an
art teacher) met through their similar tastes in hip-hop while both were DJing in
London in the 1980s. A quote by Black identifies the conf luence of interests that
would lead to their pioneering experiments in multimedia: “The three main pas-
sions I’ve had since my mid-teens are DJing, synth-building and computing.”25
Both were passionate about the technological revolution happening in music –
particularly the acid house drum machine and synth-based music that was com-
ing out of Chicago and later was associated with rave culture in the UK. In 1987
the pair decided to produce something similar in the UK, and with their first
track Beats + Pieces (1987) created what many consider to be the first bigbeat
track. Mainstream acclaim soon followed with their remix production of Yazz’s
pop single The Only Way Is Up, a hit that became 1988’s longest-running UK
number No. 1 single.
They produced and released dance tracks as Coldcut for the Arista label, but
to gain greater creative control they set up what was their second independent
102 Léon McCarthy and Steve Gibson

record label – Ninja Tune, which has gone on to become one of the world’s best-
known independent music labels. Ninja Tune was instigated as a label not only
to celebrate cutting-edge electronic dance music but also as a way to contribute
to the wider audio-visual culture through software, gaming, interaction, instal-
lations and live audio-visuals.

Coldcut’s Videos
With Black’s prior experience working with computer graphics, he had both an
interest and aptitude for creating computer-based visuals, so along with More,
Miles Visman and Rob Pepperell they formed the Hex collective to create
pop-orientated multimedia. The music video for Coldcut’s 1989 single Christ-
mas Break (see Figure 4.5) was one of Hex’s first productions, and its style was
quite a distance from the video mashup more common to scratch video; loops
of computer-generated 3D animation appear on a highly saturated patterned
background.
Hex’s first stand-alone audio-visual single Global Chaos was released in 1991,
and it had an environmental theme. Global Chaos showed a significant refine-
ment of the style established with Christmas Break, combining higher-resolution
3D landscapes, animated characters and algorithmically generated backgrounds

FIGURE 4.5 Coldcut, still from the music video for their single Christmas Break;
it is an early example of computer-generated animations and patterns
appearing in a music video. Used by permission.
 Scratch Video and Rave 103

intermixed with composited found footage. Visually Hex’s early work was her-
alding the emergence of a new digital aesthetic, one that with refinement went
on to inf luence both computer-generated imagery and psychedelic abstract visu-
als as central styles in contemporary VJ practice.

Multimedia and Software

As well as making music videos, Hex were interested in the interactive potential
of computers and in 1991 developed a computer game called Top Banana which
was later released for the new Commodore CDTV entertainment system. While
somewhat like the most popular computer games of the time (e.g. Super Mario),
what marked it as different was its environmental theme, its use of a female char-
acter lead and the implementation of algorithmically selected background music
throughout. Hex were one of the first artists to grasp the interactive potential
of the newly developed CD-ROM format; in 1992 they merged their audio-
visual series Global Chaos and computer game Top Banana into an interactive
experience for the medium. Other artists saw the potential of the CD-ROM,
and soon a whole new medium was born. CD-ROMs such as Laurie Anderson’s
Puppet Motel (1995) or The Residents Freak Show (1994) explored a new interac-
tive visual narrative that foreshadowed the immersive interactive environments
discussed in Chapter 13.
Hex themselves developed responsive audio-visuals and interactive interfaces,
and this led to gallery commissions for multimedia installations. One such exam-
ple was their project Generator for the Glasgow Gallery of Modern Art in 1994;
visitors to the gallery could use dual control pads to create an audio-visual mix
in real time from sound, video, text and graphic sources.
Black continued to develop interactive software applications even after the
disbanding of the Hex collective. In 1997 in partnership with Cambridge-based
Camart, he released the AV manipulation software VJAMM. While not a com-
mercial success, it was the foundation technology upon which Coldcut could
remix their audio-visual works for the live arena. Notwithstanding the commer-
cial impact, its significance was recognised by the American Museum of the Mov-
ing Image where it is part of their permanent collection. A year later Camart and
Black released a complementary software application called DJAMM as a real-time
audio manipulation tool, one which has morphed into its current version available
for both Android and iOS under the name JAMM Pro. With such a wide range of
projects across media, Black and More established themselves as trailblazers for a
new multimedia age, something Billboard magazine noted in 1997.26
What is notable across these inventions is a group of creative technologists
developing tools for their own needs that they then refined in order to serve
their peers across the multimedia and live audio-visual industry; while some
of these tools were developed for commercial ends, others were offered free of
charge. They helped other audio-visual artists develop, while also inf luencing
other companies who later developed similar software applications.
104 Léon McCarthy and Steve Gibson

Hexstatic
In 1997 the arrival of Stuart Warren Hill and Robin Brunson into the Ninja
Tune fold saw the collective Hex evolve into Hexstatic. In 1998 Hexstatic and
Coldcut collaborated on the Natural Rhythms series for Greenpeace, from which
the breakout audio-visual song Timber emerged (see Figure 4.6). Timber contin-
ued the thread of creating environmentally conscious work, this time with clear
reference to the deforestation of the Amazon rainforest. The piece is perhaps the
perfect manifestation of the audio-visual coupling, a process established with the
scratch video/video mashup, refined by EBN and then executed to perfection by
the collective forces of Coldcut and Hexstatic. By featuring heavily on MTV, it
brought the artists to a mainstream audience and along with it the audio-visual
aesthetic championed by those on its label.

Live Performance
Black and More’s wider interest in multimedia saw them encourage Ninja Tune
artists to explore ways in which audio-visuals could be interactive, both in terms

FIGURE 4.6 Coldcut and Hexstatic performing a remix of their audio-visual work
Timber. Used by permission.
 Scratch Video and Rave 105

of presentation format and live performance. They led the way, with Coldcut
and Hexstatic exploring ways in which technology could augment their own
live shows.
Hit audio-visual songs such as Timber defined the style of their shows, a sound
that mixed hip-hop and bigbeat over a visual track of audio-responsive algorith-
mic abstractions and selected footage. The footage was chosen for its impact and
message, a message often addressing environmental or social themes.
The loop-based BPM-driven nature of their audio-visual productions natu-
rally lent themselves to the live arena. The technical challenge was in finding
ways to access, control and sync visual clips to the rhythm of the musical mix. As
already mentioned, Black was developing interactive systems for the control of
audio and visuals, and much of the installation Generator (with its separate control
panels for the audio and visual streams) emerged from technology designed for
use by Coldcut and Hexstatic on stage.
The result of their efforts was a dazzling live show, an experience that set
the bar for what can be possible when producers, DJs and VJs combine their
creative vision with cutting-edge interactive technologies. Their live produc-
tions inf luenced how the likes of Fatboy Slim, DJ Shadow and the Chemical
Brothers approached the remixing and control of live audio-visuals. The soft-
ware tools they pioneered helped other VJs in the production and control of live
audio-visuals.

Conclusion
This chapter discussed how developments during the 1980s and 1990s con-
tributed to the definition of Live Visual performance practice and its related
aesthetics. In the early 1980s a DIY postmodern punk sensibility pervaded, so
when VHS technology made it possible to plunder the freely available resource
of broadcast TV, artists grasped the opportunity to comment on their contem-
porary society (see Chapter 15 for a further discussion on culture and sampling).
Scratch video emerged from this period, and it went on to inf luence the style
of music video that the masses became familiar with through the popular MTV
channel.
Meanwhile electronic-based dance music was gaining in popularity, but as
an experience the DJ lacked that focal point and visual spectacle that a live band
bring to an audience. Before MTV established its mass appeal and the raw aes-
thetic of the video mashup became acceptable for broadcast, scratch video artists
had limited opportunities to present their work. The visual requirements of the
burgeoning nightclub scene was an opportunity for artists, and so scratch video
artists commonly became artists-in-residence in underground nightclubs.
With the refinement of their styles, some scratch video artists shifted into
music video and commercial direction, first via MTV and then other channels.
The rise of rave in response to acid house provided context for the development
of electronic music styles such as techno and drum and bass. The presence of this
106 Léon McCarthy and Steve Gibson

aesthetic in mainstream media saw some rave artists (e.g. Prodigy) and scratch
video artists (e.g. EBN) make the jump to more mainstream live stage arenas as
part of the set production for musical acts (such as the collaborations between
EBN and U2). The larger budgets involved meant more money to spend on the
development of content and live performance equipment.
By the mid-1990s the potential that VHS and video held for visual experience
and live performance had reached its zenith in the hands of EBN, Hexstatic and
others. It was during the 1990s with the availability of the desktop computer
that computer-based graphic processing became a newly viable option for artists.
Early adopters, such as those related to the Ninja Tune label in the UK, pio-
neered the use of computer-based processes, software applications and interac-
tive interfaces to produce new types of audio-visual experience, both for off line
consumption and live performance.

What Followed
Toward the end of the 20th century club culture was going mainstream; mega-
clubs were becoming the norm and with that the rise of the ‘Superstar DJ.’ There
was never a greater demand for Live Visual performers, who in this context came
to be commonly known as VJs. The loop-based abstract visual aesthetic first asso-
ciated with rave culture and then refined (via algorithmic processes) by the likes of
Matt Black was suited to these nightclubs. However, the production of responsive
audio-visual algorithms was reliant on the desktop computer, and only the mega-
clubs had the budgets to install expensive media servers. Huge festivals such as
Sonar (Barcelona) were also established, and these showcased not only electronic
music but also audio-visual performances and more experimental forms of real-
time visual performance and installation (see Chapter 15 for a further discussion).
This engendered a whole new form of audio-visual performance that became
dominant in events across Europe and North America in the new millennium.
By the start of the 21st century, laptop-based computers were powerful
enough to process computer graphics and algorithmic visuals in real time. Simul-
taneously, VHS was being replaced by the higher-quality digital video protocol.
With a relatively inexpensive video mixer (such as the Edirol V-4) a VJ could
mix and composite computer graphics, DVD video clips and live cameras. What
would have previously required a large, complex and expensive setup was now
accessible and portable. Just like the DJ, the mobility of such a setup gave VJs the
same ability to perform in more varied contexts, the result being that far more
venues could now feature Live Visuals.
In the 1990s audio-visual artists such as EBN and Coldcut were taking stances
on contemporary issues – political in the case of EBN and environmental in
the case of Coldcut. While EBN created the perfect form of video mashup,
Coldcut and Hexstatic established a style that mixed found footage with com-
puter graphics – abstract visuals with realistic content. Simultaneously, groups
associated with the rave scene were also politically active in protest against the
 Scratch Video and Rave 107

crackdown on dance culture. Concurrently (and somewhat in contrast) a focus

on hedonism and psychedelia was also a common feature of rave and techno, and
this helped move the Live Visuals scene into a more abstract and kaleidoscopic
visual landscape that harkened (somewhat) back to the 1960s. Later in the 1990s
more complex narratives were emerging within the practice, as seen in the afore-
mentioned CD-ROM projects of Hex and the Residents. While VJs could try to
accomplish so in the nightclub, their audience was unlikely to have the requisite
focus for a long-form narrative.
As the 20th century came to an end, Live Visuals artists who were interested
in longer narrative forms shifted toward productions that were more cinematic
in nature (as discussed in relation to live cinema in Chapter 12) and were concep-
tualised specifically for a gallery or music performance context.
The work produced within this key period in many ways foreshadowed what
was to follow, both in terms of the plurality of approaches that were at play
within Live Visuals and the tension between criticality, commercialisation and
technology-focused experimental imperatives. There is no doubt that the present
forms of Live Visuals production owe much to the original work of scratch video
artists, rave DJs and visualists, CD-ROM producers and multimedia artists of the
late 1980s and 1990s. In the next chapter, these developments will be discussed
with regard to the period from 2000 to the present.

Notes
1 Simon Reynolds, Energy Flash: A Journey through Rave Music and Dance Culture (London:
Picador, 1998).
2 Emily Gosling, “How Genesis P-Orridge Changed the Course of Electronic Music
Culture Forever,” VICE, February 7, 2017, www.vice.com/en/article/ypjgpy/the-cult-
of-genesis-p-orridge.
3 Andy Lippman, “Scratch and Run,” City Limits Magazine, October 1984, 5–11.
4 Philip Auslander, Liveness: Performance in a Mediatized Culture (London: Routledge,
2008).
5 Andy Lippman, “Scratch and Run,” City Limits Magazine, October 1984, 5–11.
6 Sergei Eisenstein, Film Form: Essays in Film Theory (New York: Harcourt Brace & Com-
pany, 1949).
7 Cabaret Voltaire, Sensoria (7” Mix), Mute Records YouTube page, 2013, www.youtube.
com/watch?v=c2vCpT1H7u0.
8 Blue Monday’s success as a dance track and video also in essence financed much of the
rave scene in Manchester.
9 George Grebner and Larry Gross, “Living with Television: The Violence Profile,” Jour-
nal of Communication, 26(2), 1976, 172–199.
10 George Barber, The Greatest Hits of Scratch Video: Volume One & Two, Lux Online, 1984
and 1985, www.luxonline.org.uk/artists/george_barber/the_greatest_hits_of_scratch_
video.html.
11 Gorilla Tapes, Invisible TV Part 1, 1987, www.youtube.com/watch?v=-hn2ZYR3rUI.
12 Merrill Aldighieri, V.J. Diaries, Howl Arts website, 2017, www.howlarts.org/event/
merrill-aldighieri-v-j-diaries/.
13 Emily Gosling, “How Genesis P-Orridge Changed the Course of Electronic Music
Culture Forever,” VICE, February 7, 2017, www.vice.com/en/article/ypjgpy/the-cult-
of-genesis-p-orridge.
108 Léon McCarthy and Steve Gibson

14 Jeremy Deller, Everyone in the Place: An Incomplete History of Britain 1984–1999, docu-
mentary film: 34m 44s, 2019, www.youtube.com/watch?v=-7p5hFZz3xE (Accessed
January 3, 2022).
15 Matthew Collin, Altered State: The Story of Ecstasy Culture and Acid House (London: Ser-
pent’s Tail, 1997).
16 Erica Weir, “Raves: A Review of the Culture, the Drugs and the Prevention of Harm,”
Canadian Medical Association Journal (CMAJ), 162(13), 2000.
17 Mondo 2000, Official website, www.mondo2000.com/.
18 Omri, “90s Rave Culture & Acid House: The Beginning of the Revolution,” Techno
Station Magazine, October 8, 2016, www.technostation.tv/90s-rave-culture/.
19 Criminal Justice and Public Order Act 1994, UK Public General Acts1 994 c. 33 Part
V, Powers in relation to . . . Section 63, ttps://www.legislation.gov.uk/ukpga/1994/33/
section/63.
20 Holly Willis, New Digital Cinema: Reinventing the Moving Image (New York: Wallflower
Press, 2005), 70.
21 Marilyn A. Gillen, “EBN Expands the Multimedia Envelope,” Billboard, 197(10), March
11, 1995, 60.
22 Holly Willis, New Digital Cinema: Reinventing the Moving Image (New York: Wallflower
Press, 2005), 70.
23 Brett Atwood, “U2’s ZooTV Finds a Home on MTV,” Billboard, 109(15), April 12,
1997, 75.
24 Deborah Russell, “VuJack Heralds New Era of Video Sampling,” Billboard, 107(2), Janu-
ary 14, 1995, 32.
25 Andy Price, “Interview with Matt Black,” MusicTech Magazine, February 13, 2020,
https://musictech.com/features/interviews/matt-black-legacy-coldcut-ninja-tune/
(Accessed October 29, 2021).
26 Brett Atwood, “Coldcut Dishes Out More Multimedia,” Billboard Magazine, January 18,
1997, 66.
5
THE POST-CONCEPTUAL
DIGITAL ERA (2000–PRESENT)
Paul Goodfellow and Steve Gibson

Introduction
This chapter covers the period from 2000 until the present moment. Existing
in a post-conceptual world in which no artistic movement holds any promi-
nence, live visualists and audio-visual artists continued the promiscuous mix-
ing of forms, genres and tools to produce a diverse body of work ranging from
improved club DJ/VJ performances to large-scale live cinema events with a
large ensemble of audio and visual performers. The artistic reality of the early
21st century was one of post-conceptualism. The occasionally aggressive politi-
cal conceptualism of the 1980s and 1990s gave way to a much more f luid and
hybrid situation, in which forms, techniques, technologies and aesthetics were
combined from seemingly irreconcilable sources. Aspects of conceptualism and
postmodernism remained, but with no dominant artistic ideology coming to the
fore, a hybrid practice with many offshoots and (sub)genres became the norm.
This chapter will chart how Live Visuals is situated within this wider, more
pluralistic creative culture, whilst also covering the historical advancements of
the 21st century. Throughout the early 21st century the use of Live Visuals and
real-time graphics became increasingly the norm, not just in performance but
also in games, virtual environments and even forms of advertising. By 2020 Live
Visuals were present in many facets of our culture, oftentimes existing as par-
allel and/or separate movements to those of the audio-visual culture discussed
throughout this book.
Social media, live-streaming and Zoom all extend both the understand-
ing and participation within Live Visuals within wider culture. Underpinning
these developments are dramatic socio-historical changes or intensities which
have shaped our relationships with technology, each other and ourselves. Three
key intensities have shaped the new millennium, and these are discussed in this
DOI: 10.4324/9781003282396-7
110 Paul Goodfellow and Steve Gibson

chapter. Firstly, there are developments within technology and how they have
enmeshed culture within digital systems. Secondly, the understanding that the
concept of ‘nature’ must be extended to include technology and the fragility of
the planet due to the climate crisis. The final intensity is the overwhelming sense
of melancholy which is performed in contemporary art and Live Visual culture as
we come to terms with our entanglement with technological systems.
The real breakthrough for Live Visuals happened in the early 2000s when
computer technology advanced sufficiently to be able to deal with larger video
files in real time and a number of specialized Live Visuals software solutions were
developed, including Arkaos, Isadora, Modul8, TouchDesigner and Resolume.
These applications allowed VJs and live visualists to randomly access a bank
of video clips, to apply effects in real time, and to use digital signal processing
(DSP) to have the visuals respond to aspects of the incoming audio signal. With
the addition of MIDI controllers with sliders, knobs and buttons to control video
parameters (at times in synchronisation with audio parameters), the rise of the
live audio-visual performer became a common feature of both the club and fes-
tival circuit.
In the mid-2000s VJing in clubs became de rigeur, though at times the
algorithmic programmed visuals utilizing DSP to respond to audio data pro-
duced results that could hardly be said to be ‘live.’ VJs and programmers
began to develop their own software that combined these algorithmic ele-
ments but also allowed for more direct live interaction using specialist con-
trollers. Simultaneously the rise of audio-visual performers such as Carsten
Nicolai/alva nota spawned a series of performances in which both the audio
and video were controlled in real time by one performer. Artists began to
produce performances using Live Visuals, live audio and a specialized stage
configuration to immerse both the performance and audience in a live mixed-
media experience.
Parallel to this, experiments with projection mapping became ubiquitous.
This led to a number of projection events on to architectural facades using soft-
ware such as MadMapper. Much of this work is conceptually indebted to media
artist Rafael Lozano-Hemmer’s Relational Architecture series of installations that
use iconic buildings/locales as surfaces for projection and public interaction (see
Chapters 13 and 14 for a further discussion of this). Projection mapping allows
the live visualist to move more freely into the broad public domain. The best
examples, such as the work of the MXZehn or the Klip Collective, engage with
the meaning of the buildings they work with, as exemplified by their What’s He
Building in There? for the Sundance Festival in 2013.
The chapter concludes with a speculation about possible future developments
for Live Visuals as they become truly immersive and four-dimensional due to
increasing access to virtual reality and sensor-based technologies. As the field of
Live Visuals expands, a new collaborative model of audio-visual production is
also proposed that combines the formal and conceptual methods from the past
and the present but also anticipates the future of immersion.
 The Post-Conceptual Digital Era 111

Socio-historical Intensities of the 21st Century

This chapter considers the explosion of Live Visuals which has taken place in the
21st century, which can be understood as the ‘digital era.’ This period of Live
Visual production is significantly different from the preceding decades due to
three significant ‘intensities’ that have shaped culture in the new millennium.
These intensities are discussed here as they describe the environment in which
Live Visuals are produced and consumed in the 21st century. Intensity is a useful
concept, developed by Gilles Deleuze and Félix Guattari in A Thousand Plateaus
(1980), to describe forces that are difficult to locate in a concrete sense but we
feel their inf luence.1 Due to its enigmatic nature, Deleuze and Guattari did not
offer a simple synopsis of ‘intensity,’ but Ian Buchanan summarises it as follows:

Intensities are the agitations of the mind and body [. . .] that move us in
an emotional, spiritual or libidinal sense but we cannot name; they are the
stirrings in our mental equilibrium that come before love and hate, anger
and frustration; they are the sensations we long to sustain when were on
a ‘high’ and cannot wait to escape or extinguish when we’re stuck feeling
‘low.’2

In A Thousand Plateaus, Deleuze and Guattari describe the world as f lows of

matter and energies, including thoughts, which can be categorised or stratified
within distinct ‘strata.’3 These strata, which are observable in the world, are
the geological or material strata, the biological or living strata and the techno-
semiological or the informational strata. The first intensity during the digital era
is the expansion of the techno-semiological or technological, informational and
systemic dimensions of culture and how this has impacted our understanding
and relationship to nature, each other and ourselves in the 21st century.
This informational shift can be seen within Live Visuals as artists moved
from scratch mixing using videotaped imagery in the 1990s to producing and
mixing live digital streams of information in the new millennium. This evolu-
tion within Live Visuals shifted our focus from the surface level of the image
content and its iconic and symbolic function to its indexical relationship with
the wider networks of information systems that produce, sustain and circulate
them within culture. This shift can be understood as part of a wider enfoldment
within technological and information systems in the 21st century, and this inten-
sity is explored in this chapter.
While experiments with real-time digital software did occur in the 1990s (as
mentioned earlier in Chapter 4, the first piece of VJ software, Vujak,4 was created
in 1992 by Emergency Broadcast Network but never released commercially),
technologically the development of faster graphics processing on computers and
the implementation of DSP to analyse incoming audio data and produce visual
results ultimately led to the development of a plethora of real-time visual tools
and VJ software in the late 20th and early 21st centuries. Applications such as
112 Paul Goodfellow and Steve Gibson

Macromedia Director and Max MSP began to deal with live video in the late
1990s, and a number of MIDI-based solutions for triggering and editing visuals
using these tools in real time were developed at around the time of the turn of
the millennium, spawning the form we now know as VJing. This first techno-
logical intensity radically altered the way images could be processed in real time,
not only opening up avenues for audio-visual performance but also allowing
the development of multiplayer games or environments with huge, ever-shifting
graphical and immersive worlds, such as Second Life (see Figure 5.1).
The second intensity is the existential and material crises facing ‘nature.’ As
the technological and informational world has extended into every aspect of
life, the very idea of nature as something romantic which exists independently
from human activity has eroded, and we see that all ‘natural’ and ‘unnatural’
materials are deeply and inextricably entangled.5 However, the impact of human
activity not only disrupted our phenomenological relationship to nature but has
destabilised the Earth’s biological, ecological and climate systems. The COVID-
19 global pandemic caused by the SARS-CoV-2 virus is a stark example of our
physical entanglement with the biological and ecological systems which extend
across the planet. This second intensity undoubtedly has had an effect on the
production and dissemination of Live Visuals and audio-visual culture, not least
because the pandemic effectively shut down the performance world (almost)
globally for at least 18 months.
The final intensity is the emotional and psychological impact of the first two
intensities: our enfoldment within technological systems which control our

FIGURE 5.1 HyacintheLuynes – Own work, CC BY-SA 3.0, Live radio hour in
Second Life 11th birthday with Draxtor Despres and Jo Yardley.
Source: https://en.wikipedia.org/wiki/Second_Life#/media/File:Second_Life_11th_Birthday_Live_
Drax_Files_Radio_Hour.jpg
 The Post-Conceptual Digital Era 113

thoughts and emotions and the climate crisis, which forces us to confront the
unnaturalness of nature and our vulnerable place within it. These understand-
ings have led to a range of melancholic feelings, including nostalgia, saudade,
weirdness and eeriness.6 These feelings are both performed and addressed in the
art of the early 21st century. Without a doubt these feelings are also evident in
many of the products of Live Visuals, virtual reality and media art cultures (see
The Klip Collective’s What’s He Building in There? Figure 5.5, in this chapter,
or Alan Kwan’s Alien Abduction Simulator, Figure 13.8, in Chapter 13 for striking
examples).

Ground Zero
In 1999 we watched the film The Matrix (1999), which told the story of the
human ‘lifeworld’ being merely a computer simulation.7 The lifeworld is a con-
cept developed by Edmund Husserl to describe the given world which can be
experienced together.8 In the film this simulated lifeworld was fed directly into
the body and mind of the individual who passively slept, unaware of their com-
plicity within a vast system of entertainment, control and exploitation.
In 1999 we also waited in trepidation for the impact of the ‘Y2K’ or ‘Millen-
nium bug.’ These were new terms coined to describe errors within the inter-
nal time codes of computer programmes with the potential to cause systems to
cataclysmically fail. The errors related to the representation of dates within pro-
gramme code, which, due to coding and processing efficiencies, or ‘bit conser-
vation,’ were represented by the last two digits of the year. Thus, the year 2000
was represented by 00 and could therefore be misunderstood as the year 1900.9
More fundamentally, it reinforced the idea that the millennium represented a
temporal ground zero or resetting of time. The technical issues posed by this
oversight had been anticipated within the information technology industries for
some time, and most system issues had been resolved before the deadline. How-
ever, some problems were reported on New Year’s Day, such as an alarm sound-
ing at a nuclear power plant in Onagawa, Japan, which reinforced the perception
that a major catastrophe had been averted.10 The fear induced by these potential
glitches, both real and imagined, have both defined and anticipated the concerns
of the new millennium, which can be understood in terms of our relationship to
computer-mediated systems and our increasingly complex experience of reality
and its simulation in information.
In contrast to this feeling of impending dread, the relationship with emerging
computer technology before this date and throughout the 1990s can be defined
in largely positive and progressive terms. This confidence was demonstrated in
the emergence of radically new musical genres, such as drum and bass and the
emergence of VJing as a distinct art form. This optimism was also tracked and
championed in technology magazines such as Wired and the countercultural
magazine Whole Earth Catalog. Published by Stewart Brand between 1968 and
1998, Whole Earth Catalog originally offered a holistic and ecological vision for
114 Paul Goodfellow and Steve Gibson

reconnecting to the Earth. However, in the 1990s, this vision of nature and
society was extended to include emerging computer technologies. In 1994 Brand
extolled the virtues of cybernetic technologies, suggesting they would promote
individual freedom and support collaborations that were beyond the control of
big business and government, declaring, “We are as gods and might as well get
good at it.”11
This newfound power to access and reconstitute information aligned with
François Lyotard’s description of the postmodern society in which ‘grand narra-
tives’ would prove inadequate descriptions of an increasingly hybrid society and
would instead be replaced with competing interest groups, narratives and modes
of discourse.12 At the heart of this fragmentation was the challenge made to
authorship wrought by technological developments as we moved from the pas-
sive consumption of information delivered via newspapers and broadcast media
to an active participation in the retrieval, organisation and dissemination pro-
cesses made possible through hypertexts, the Internet, information systems and
real-time graphics.
This shift in relationship afforded by information technologies between the
text, the image and the ‘end user’ had been anticipated decades earlier in the phi-
losophy and literary criticism of Roland Barthes, Jacques Derrida and Umberto

FIGURE 5.2 ͠͡ ͬͬ ͠ ͬͬ ͠ ͬͬ Menkman, Killer Pillers on nine screens. Entter VJing at Mapping
Rosa
in the Zoo, 2007. An example of relatively early VJing which employed
found footage in a layered real-time context.
Source: Attribution-NonCommercial 2.0 Generic (CC BY-NC 2.0). www.f lickr.com/photos/
r00s/493577591
 The Post-Conceptual Digital Era 115

Eco, with each imagining the liberation of the text from interpretations imposed
by the author. This open and networked perspective aligned with ‘hypertext,’ a
concept developed by the information science pioneer Theodor H. Nelson in the
1960s. Nelson described hypertext as “nonsequential writing (sic) text that branches
and allows choices to the reader, best read at an interactive screen. As popularly
conceived, this is a series of text chunks connected by links which offer the
reader different pathways.”13 Nelson also used the term hypermedia, which was
the multimedia expansion of hypertext to include images and sound.
Alongside the philosophical (Barthes, Derrida and Eco) and technological
(Nelson) liberation of the image and text, artists also played with strategies of
reconfigurations in the production of new works. This can be traced back to the
avant-garde and Marcel Duchamp’s employment of chance and the Dada and
Surrealist technique of the cut-up (art movements mentioned in Chapter 7). The
cut-up technique first emerged in Dadaism in the 1920s when artists created
poetry with random words cut from newspapers. This was developed further
in the 1960s by the experimental writer and visual artist William S. Burroughs,
who, inf luenced by the surrealist artist Brion Gysin, employed cut-up techniques
to disrupt language and express the hallucinatory, disorienting perceptions felt
whilst under the inf luence of alcohol and drugs.14 This was also key to the devel-
opment of the work of the Fluxus artists, described previously in Chapter 3.
Cut-up and experimental production techniques were also applied to music
and can be traced back to composers including John Cage, Karlheinz Stockhau-
sen and Steve Reich, who employed chance, indeterminacy and repetition in
the creation of new works. Sampling and cut-up techniques were also evident
in Jamaican Dub from the 1960s with the producer and musician Lee “Scratch”
Perry, for example, using pre-recorded samples to produce live mixes during
DJ sets. Sampling was also used by experimental artists such as Brian Eno and
Holger Czukay in the 1970s and 1980s, before being employed more widely
within hip-hop (Public Enemy, Beastie Boys) in the 1980s and 1990s. This in
turn inf luenced the visual culture of scratch video in the 1990s (as discussed in
Chapter 4), as well as Live Visuals and VJ culture more broadly in the 21st cen-
tury. The selection, edit and reuse of samples in music set the template for the VJ
artist in the late 1990s to work alongside the DJ to build compelling multimedia
experiences based on the synchronisation of video loops with the music. In the
new millennium the VJ became the ultimate remixer of visual culture, cutting
up found footage to accompany the new forms of music that came out of the late
20th and early 21st centuries (see Figure 5.2).

Systems Enfoldment
During the 1990s, digital tools such as personal computers, hypertext and the
Internet, as well as new forms of audio-visual software such as Macromedia
Director and Flash, made the promises of postmodernism – deconstruction,
decentralisation and personal autonomy – a reality. In essence, the 1990s had
116 Paul Goodfellow and Steve Gibson

promised a new era of personal freedom, which symbolically began with the fall
of the Berlin Wall on November 9, 1989. Francis Fukuyama, in his infamous
1992 book The End of History, stated:

What we may be witnessing is not just the end of the Cold War, or the
passing of a particular period of post-war history, but the end of history
as such: that is, the end point of mankind’s ideological evolution and the
universalization of Western liberal democracy as the final form of human
government.15

Fukuyama articulated the widespread, but misguided, belief that politics, eco-
nomics and technology were facilitating a democratic and stable future. This
positive milieu was exhibited across the political spectrum, demonstrated in
Brand’s earlier statement “We are as gods.”16 However, as much as the new tools
empowered us to collect, archive and remix information, we too were being
absorbed within the new technological systems, and this enfoldment would
become the defining characteristic of 21st-century culture.
One of the first books of the new millennium, Katherine Hayles’s How We
Became Posthuman: Virtual Bodies in Cybernetics, Literature, and Informatics (1999),
discussed the impact of technological changes on the very conception of human-
ity,17 in particular, how we as both species and individual psyches are manipulated
to conform to the requirements and design of information technology, rather than
the technology conforming to the desires and requirements of human need.18
Hayles develops this argument by mapping the development of systems think-
ing in terms of three periods or ‘waves’ of cybernetics. The first wave appeared
during the 1940s, and a significant development, Ludwig von Bertalanffy’s Gen-
eral System Theory, described living creatures as systems which were kept in bal-
ance through the exchange of energy and material.19 Second-wave cybernetics
is concerned with ref lexivity, described as the ‘ref lexive turn.’ The ref lexiv-
ity of all systems is acknowledged, meaning that the observer of a system will
inevitably disrupt the system under observation. Thus, the ecologist will both
operate within and impact upon the ecosystem she studies, and the psychologist
will become part of the patient’s environment and inf luence their thoughts and
feelings.
The first two waves directly inf luenced the systems artists of the late 1960s dis-
cussed in Chapter 7. However, cybernetics evolved further, and Hayles describes
the third wave in terms of ‘emergence’ and ‘virtuality.’ Hayles locates this shift
to the 1990s and the Fourth Conference of Artificial Life in 1994, where the
biologist Thomas S. Ray made two proposals.20 Firstly, he put forward a plan to
protect the biodiversity in the Costa Rican rainforests. Secondly, he proposed
that an artificial life computer program he developed should be ‘released’ on the
Internet to ‘breed’ and evolve. As Hayles summarises, “The first aimed to extend
biological diversity for protein-based life-forms; the second sought the same for
silicon-based life-forms.”21
 The Post-Conceptual Digital Era 117

Both the protein-based and silicon-based systems point to ‘liveness’ and even
sentience being emergent properties of the system. Hayles suggested that our
sense of self, or human consciousness, is an emergent property of the feedback
systems operating in the body and the mind and there is no difference, in prin-
ciple, between a sentient human and an intelligent machine. This radical, ‘emer-
gent materialist’ and posthuman thought is both the defining characteristic of
the current cybernetic thinking and goes some way to explaining both the art
and the milieu of the early 21st century.
Thus, rewatching The Matrix or walking through Jason Rhoades’s art instal-
lation The Creation Myth (1998) in the early 21st century, we can feel that we
are both enmeshed within and emerge from systems (see Figure 7.3 in Chapter
7). We have started to understand that on a personal, social and ecological level
we are increasingly engaged with systems which simulate and disconnect us
from the physical subjects of these fields: the self, the community and the Earth.
This shift in focus from the material subject, grounded in space and time, to its
description in data has profoundly affected our mental wellbeing, political dis-
course and environmental understanding.
This destabilising shift is most keenly felt in the systems which structure our
personal and social interactions, such as social media which distort meaning and
communication and disrupt both the communal ‘lifeworld’ and sense of self. The
design of communication via single-user mobile phones and the fragmentation of
communication within system-mediated groups or ‘filter bubbles’ means that the
dominant phenomenological experience is not one of shared experience, but of
isolation. Likewise, the production, filtering and circulation of idealised images
of the self across social media distribute self-perception across the network and
make it contingent on the approval of algorithms and other isolated participants
operating within the system. These systems demonstrate the cybernetic qualities
of feedback, ref lexivity and emergency, and most radically, they demonstrate
virtuality and simulation. As Erkki Kurenniemi observed: “We should no longer
think in terms of technology shaping self-perception, but instead, of technology
simulating the self, and then replacing the self with its simulation.”22

Expanded Melancholia
Thus the 21st century has demonstrated that we live within systems: climate,
late capital and technology, which exceed our understanding and ability to con-
trol them. It has also demonstrated that we as individuals can be understood
as systems, and not only can these systems be simulated, we have willingly, if
blindly, participated within our own simulation. We are no longer the centre
of our lifeworld or even the centre of ourselves anymore, and the dominant
feelings of the 21st century can be described as an ‘expanded melancholia.’ The
term ‘expanded’ is drawn from its application within art, where it is employed
to denote conceptual and material expansion beyond the original framework.
These include Expanded Cinema by Gene Youngblood (1970),23 which described
118 Paul Goodfellow and Steve Gibson

the expansion of cinema into video and media art, and Expanded Sculpture by
Rosalind Krauss (1979),24 which described the expansion of sculpture into new
art forms. Melancholy can be understood in such expanded terms to encompass
a broad assemblage of feelings, including nostalgia, saudade, eeriness and the
weird.
Nostalgia has been one of the dominant feelings in the 21st century, with
contemporary culture heavily focussed on the aesthetics and cultural materials
of the late 20th century. Nostalgia is evident within electronic music and the
subgenre of vaporwave, which emerged in the early 2010s and can be felt in the
music of Oneohtrix Point Never and the aesthetics of the media artist Jon Raf-
man. Vaporwave directly references and samples critically ignored music such as
incidental ‘elevator’ music from the 1980s and 1990s. Such music employed elec-
tronic instrumentation such as synthetic saxophone samples, which experienced
in the 21st century, sound both naïve and forlorn. Vaporwave music is often
coupled with graphics from the 1980s and 1990s, drawn from early computer
games, cyberpunk and techno artwork and early web design. Both the music and
artwork exhibit an optimism towards a positive technological future that never
arrived, at least not in the way we expected, and this understanding casts them
as profoundly melancholic.
Nostalgia has also been a central aspect of contemporary art in the 21st cen-
tury. In part, this has been a continuation of the postmodern project, discussed
in Chapter 7, whereby art appropriates earlier ideas, forms and artworks and
remixes them to say something about our relationship to the past and memory
and make sense of the present. Such appropriation has a long history in art which
stretches back to Dadaism and the reuse of found materials. However, the art of
the 21st century has been defined by an expanded melancholia with much work
profoundly nostalgic in terms of ideas, materials and processes. Fredrick Jameson
described cinema’s obsession with the past as ‘nostalgia mode,’ stating that we
have “become incapable of achieving aesthetic representations of our own cur-
rent experience.”25 This accusation can be levelled at contemporary Art in the
early 21st century.
This nostalgia mode has revealed itself in three significant ways in con-
temporary art, audio-visual culture, media art and Live Visuals. Firstly, some
artists have sought to retrieve the ideas, strategies and aesthetics of conceptual
art which emerged in the late 1960s. An early example is Damien Hirst’s Lov-
ing in a World of Desire (1995), a colourful reworking of Hans Haacke’s original
environmental sculpture Floating Sphere (1964). Hirst replaces Haacke’s pure
white sphere with a multicoloured plastic beach ball, making the work simul-
taneously nostalgic and ironic, but ultimately, melancholic as it expresses sau-
dade for the lost simplicity of early conceptual art. Secondly, other artists have
focused on the collection, organisation and representation of found materials.
Hal Foster has described this as the ‘archival impulse,’ which can be under-
stood as both a postmodern strategy and a reaction to the informational excess
of the Internet. 26
 The Post-Conceptual Digital Era 119

The artists of the archival impulse are generally from the Generation X demo-
graphic group who went through art school in the 1980s and 1990s and therefore
distinctly remember a time before digital technology and the Internet. In con-
trast, artists from the Generation Y demographic group have largely grown up
in parallel to many of the developments of mobile communication, the Internet
and social media, and their experience of technology and communication is one
of constant f lux. This can be seen in the final form of nostalgia mode art: post-
Internet art, a category first employed by Gene McHugh to describe art of the
early 21st century, which employs the aesthetics and motifs of early computer
graphics and the Internet. Such work exhibits a mix of nostalgia for the recent
technological past and underlying insecurity towards the technological future.27
Such work exhibits saudade for a time before we were enfolded within tech-
nological and information systems, when the Internet and digital technologies
were something you accessed only when needed. William Gibson, the writer
who anticipated our immersion within cybernetic systems in his 1984 novel
Neuromancer, observed this shift, stating in 2010:

Cyberspace, not so long ago, was a specific elsewhere, one we visited peri-
odically, peering into it from the familiar physical world. Now cyberspace
has everted. Turned itself inside out. Colonized the physical.28

Saudade is a Portuguese term, which suggests a longing for something that is both
lost and irretrievable. António Braz Teixeira summarised saudade as “the desire
of the loved thing or the loved creature, turned painful by the absence of it.”29
Thus early 21st-century art exhibits saudade concerning the past and an open
and optimistic future that was promised by technology but never materialised.
The Generation X artist Hito Steyerl lamented the power shift between users
and technological systems when she stated: “The 1990s were about decoding
and understanding these relations but now it’s more about how to be immersed
without drowning.”30
The realisation that we are in the matrix of complex systems has instilled
other, more ambiguous feelings, which, although harder to pinpoint, are central
to the psyche of contemporary culture. Mark Fisher has argued that the prevail-
ing feelings of the 21st century can be understood in terms of ‘weirdness’ and
‘eeriness.’31 He defines the weird as a “presence of that which does not belong,”32 and
the starkest example of this is the exponential proliferation of COVID-19 and
the ensuing humanitarian and existential crisis, which exceeded apprehension at
both an individual psyche and governmental level.
In contrast to the weird, Fisher describes eeriness as the “failure of absence”
and the “failure of presence.”33 He describes the alien presence of something
which shouldn’t be there as “failure of absence,” and we can think of the images
of the huge temporary hospitals and vaccination centres of the pandemic as
examples of eerie presence, whilst the “failure of presence” can be felt in the
empty cities during the pandemic societal lockdowns. Although these two
120 Paul Goodfellow and Steve Gibson

FIGURE 5.3 Universal Everything, Infinity, 2021. Used by permission.

Source: www.universaleverything.com/projects/infinity

understandings of eerie haunted the physical world during the pandemic, they
have more widely permeated the human condition in the new millennium as we
have become simultaneously extended and dematerialised by the cybernetic and
technological apparatus of contemporary culture.
An example of embodied eeriness in contemporary Live Visual art is Infinity
(2021), by the art and design group Universal Everything, led by Matt Pyke (see
Figure 5.3). Universal Everything describes Infinity as “a never-ending video
artwork, an endless parade of unique personalities born from code.”34 This work
is inspired by an earlier project for MTV developed by Pike and Paul Simpson in
which a furry character walked across the screen and its fur morphed in length,
colour and texture. Motion capture data of human walking drove the movement,
and this was looped to create the gait and motion of the character, which was
recognisably human but turned uncanny by the furry appearance. Infinity took
this idea further by applying the human motion capture data to a never-ending
parade of characters that have been procedurally generated and rendered in real
time. Each character has a different virtual body, and the simulated physics of
each digital body interacts with the motion data to create new forms of move-
ment. Some characters sprint across the screen, whilst others lurch or struggle
under the weight of their heavy fur or geode-encrusted forms. Infinity was live
streamed for four hours each day on Universal Everything’s Instagram page.
This project is a powerful example of Live Visual art as situated in the early
21st century and demonstrates the characteristics of the third wave of cyber-
netics, as it is by nature both ‘virtual’ and exhibits unanticipated or ‘emergent’
characteristics. It is, however, not a return to the representational and figurative
 The Post-Conceptual Digital Era 121

nature of premodern art – at least not from a human perspective. Within art
history, art moved from figuration and representation towards abstraction and
conceptualism. Infinity suggests the same narrative is now unfolding within post-
human art, and we are witnessing the first stages of the simulacrum as it evolves
live on our post-social media.

Post-conceptualism
Technologies have increasingly decentred the experience of being present in
a physical body as we live virtually in social media bubbles, whilst at the same
time the climate crisis and COVID-19 have forced us to acknowledge our cor-
poral reality and the importance of physical connection and affective experience.
This is the context in which new forms of post-conceptual art, Live Visuals and
audio-visual performances have emerged within the 21st century which express
our current state.
While the history of post-conceptualism in art is well-beyond the scope of
this chapter, for the purposes of our discussion, it has been described by Peter
Osborne as follows: “Post-conceptual art is not the name for a particular type of
art so much as the historical-ontological condition for the production of Con-
temporary Art in general.”35 Osborne’s strict definition of post-conceptualism as
applied within contemporary art is explored further in Chapter 7, whilst the dis-
cussion here relates to the application of the term from a media art perspective.
Joseph Nechvatal, for example, goes into some specifics, describing post-concep-
tualism as related to the reintegration of various previously assumed opposites, as
well as the integration of the digital with the analogue:

The keystone of the post-conceptualness of post-conceptual art is that vir-

tual producing rule-based computer technology has become a significant
means for making and understanding contemporary art and that this brings
us to a place where one finds the merging of the computed (the virtual)
with the uncomputed corporeal (the actual).36

If conceptualism was focussed on the ‘concept,’ as opposed to the ‘form,’ post-

conceptualism is much more hybrid, f luid and unafraid of the technological. In
that way it sits in parallel to the ideas of Barthes, Nelson and Hayles discussed
earlier. At the same time post-conceptualism is promiscuous and hard to pin
down as a single movement, but rather is characterised by its overall relation to
the technological and social conditions of the present.
Many of the artists generally discussed in relation to post-conceptualism are
digital and/or media artists who have moved beyond the limits of conceptual-
ism and are not afraid to reintegrate form, meaning, concept, technology and
materiality. This is evident in the work of such diverse artists such as Cory Arch-
angel (discussed in Chapter 7), Rafael Lozano-Hemmer (discussed in Chapters
13 and 14) and the Shezad Dawood (discussed in Chapter 13). While many live
122 Paul Goodfellow and Steve Gibson

visualists are likely unaware of post-conceptualism as a notion, there are obvious

resonances with post-conceptualism as played out in media art and the unasham-
edly varied work of live visualists and audio-visual artists. This can be seen in
both the wilful use of multiple materials from the digital and analogue realm and
the radical mix of styles and techniques from montage, to the cut-up, to total
abstraction.
The residue of post-conceptualism, as well as previously discussed notions
of nostalgia, the weird and the eerie, are abundantly evident in the work of live
visualists, from club VJs, to artists working with projection mapping, to VR art-
ists. As the visual remix is often key to artists working in real time, an engage-
ment with nostalgia is almost a given (though this engagement can be anywhere
on the scale from fully naïve to hyper-critical). The repurposing of visual imag-
ery is often buried in the work of visualists and live visuals performers, occasion-
ally forming part of the content of the work.
An example of this can be seen in the piece Virtual VJ by Steve Gibson and
Stefan Arisona (see Figure 5.4), in which the artist and audience interact with
an audio-visual environment by moving in 3D space. In this case many of the
images/videos used are abstracted versions of dystopian sci-fi imagery, often
buried to the point of unintelligibility but musically matched with heavily pro-
cessed audio samples from the same sci-fi sources.37
In addition, our relation to technology is often represented by bodily interac-
tion with live audio-visual material, oftentimes from an ambivalent or, at best,
neutral philosophical position, as evident in the work of Atau Tanaka’s Sensors,
Sonics, Sights in which the audio-visual materials are entirely generated and con-
trolled by physical controllers (see Figure 10.5 in Chapter 10). This is also a key

FIGURE 5.4 Steve Gibson and Stefan Arisona, Virtual VJ, 2015. Steve Gibson performs
Virtual VJ at the G-VERL Launch, University of Hertfordshire. An
image from Alien appears composited and buried with a number of other
shape and light effects.
Source: Photo by Stefan Arisona.
 The Post-Conceptual Digital Era 123

precept of Virtual VJ as well as Steve Gibson’s Opto-Phono-Kinesia (see Figure 13.9

in Chapter 13).

Performing the Digital

The material of audio-visual culture in the 21st century is inarguably uncon-
strained by previous limits on style and genre (though obviously individual art-
ists set their own limits). Beginning in loose terms in the early 2000s, VJs began
to be a fixture in music clubs as soon as new software such as ArKaos, Resol-
ume and TouchDesigner became more widely available and computer process-
ing speeds were capable of rendering, compositing and effecting visual material
in real time. Most initial VJ experiments in the 2000s used a wide variety of
sources, oftentimes primarily based on found material cued in synch with elec-
tronic and dance music. Certainly the term VJ was used as far back as the 1980s
(see Chapter 3 for a discussion of Merrill Aldighieri), and arguably groups like
Emergency Broadcast Network in the 1990s were proto-VJs; however, genuine
VJing as it is known today was only made possible with the software mentioned
earlier, which allowed the use of multiple streams of video simultaneously, real-
time special effects and a use of DSP to process audio input and apply visual
results. Today VJing is a vast enterprise with huge networks of VJs, such as VJ
Union,38 representing the collective input of thousands of club VJ and audio-
visual artists (the history and materials of VJing are covered in further detail in
Chapter 12).
Beyond club VJing, Live Visual events became extremely common in the
early 21st century. These included VideA Festival from 2000 to 2005, PixelAche
in Finland from 2003 and most importantly the MUTEK Festival in Montreal39
which began in 2000 as a primarily experimental electronic music event, but
has now branched out to arguably the most important event for Live Visuals and
audio-visual performance. Companies such as GarageCube (makers of Modul8
and later MadMapper) also began co-organising specific events for VJing such
as the Mapping Festival40 in Switzerland, starting in 2005 and running in some
form until the present. Both Mutek and the Mapping Festival became the site for
not only VJ performance but also increasingly complex forms of audio-visuals,
often involving complex projection mapping scenarios (see Amon Tobin’s ISAM
as an example of extremely complex projection mapping and audio-visual
matching – Figure 11.3 in Chapter 11).
Similarly, a number of companies specialising in Live Visuals, light installa-
tion, projection mapping and audio-visual installation began springing up in the
early to mid-2000s and later, particularly in London (United Visual Artists,41
Light Surgeons,42 Squidsoup43), but also further afield such as Newcastle (Novak
Collective44), Zurich (Scheinwerfer), Berlin (MXZehn45), Toronto (Derivative46),
Barcelona (Hamill Industries)47 and Philadelphia (Klip Collective48). While
many of these groups began as live visualists, a cursory look at their websites
shows the increasing complexity of their work, often within the public sphere
124 Paul Goodfellow and Steve Gibson

and using multiple media such as large-screen multi-projection, complex light-

ing setups, live music and even sculptural elements. The Live Visuals events of
the 2010s and 2020s have become increasingly complex public spectacles that
seek to immerse audiences into total artworks (not dissimilar, at least in audi-
ence experience, to the 19th-century panoramas and stereoscopes discussed in
Chapter 13). This loops us back around to a discussion of technology, nature and
weirdness.
While there are numerous examples of the intersection of live visuals, projec-
tion mapping, nature, weirdness, eeriness and our relation to technology, in the
interest of space, none perhaps covers all of those so closely and precisely as Klip
Collective’s What’s He Building in There? created for the 2013 Sundance Festival
(see Figure 5.5). Based on Tom Wait’s equally weird song of a similar name,49
the project encompasses many of the concerns of the post-conceptualist 21st
century, but it also serves as an excellent example of the use of Live Visuals in
the digital age.
The piece has been extensively documented by the Reelhouse.org.50 In the
video, a venue (the Yard) has been meticulously projection-mapped to reveal
various elements supposed ‘inside’ the large building. The building is a single-
storey, barn-like, wooden structure, and the event is being shown at night.
There appear to be real (human) participants interacting with the projection-
mapped venue, but this may be slight-of-hand. Strange objects appear, sound
effects groan and creak and a man appears inside, beside a conveyor belt which
is conveying indistinct orange objects. Strange undefinable machinery appears,
snippets of bass appear and disappear along with processed voices and the entire
experience seems like a parable for the weirdness of the 21st century. The look
is deliberately nostalgic (maybe nostalgic for the future), but equally impressive

FIGURE 5.5 Klip Collective, What’s He Building in There?, 2013 Sundance Festival.
Klip transformed the entire front of The Yard venue into a 3D projection-
mapped parable.
Source: Photographer: Kevin Ritchie. Courtesy of Klip Collective. Used by permission.
 The Post-Conceptual Digital Era 125

from a technological point of view (the projection mapping is very precise, the
visual imagery is seamlessly integrated into the building and the soundtrack is
meticulously produced). Klip lead artist Ricardo Rivera outlines the themes of
the project as follows:

There are a few themes that are within the piece. The most blatant is man
vs. machine. Organic vs. Mechanical. We are all processed everyday and
spit out to just do it again the next day. The piece also explores the physi-
cality of the building I by playing off of what’s inside vs. outside. It’s kind
of a peak into my psyche. The machine being work and the other weirdo
things are the distractions.51

If Klip represents the eerie and weird (but quasi-real, or at least surreal) intersect-
ing with the digital, then a series of performers (primarily but not exclusively)
from Germany represent a much more rational and formal approach to the per-
formed image. Artist groups such as MXZehn represent a reinvigoration of the
abstract and the formal with the post-conceptual visual world. A key marker of
much post-conceptual art (not just in media art and audio-visual performance,
but key to both) is a reappraisal of aspects of formalism:

Perhaps due to its technical and technological nature, digital media art has
been (and remains) more generally concerned with the processes, models
and structures that for much of the past 40 years were considered passé
by the contemporary art world. This is particularly evident in the work
of audiovisual performing artists (e.g. The Light Surgeons . . .), as well
as interactive artists who rely on narrative structures for their work (e.g.
Donna Leishman).52

While several artists are associated with abstract/formal live audio-visuals,

including those outside of Germany such as Amon Tobin (UK, Canada), Ryoji
Ikeda ( Japan) and Max Cooper (Belfast), amongst many others, perhaps the most
important abstract/formal live audio-visual artist to emerge from this milieu is
Carsten Nicolai (aka alva noto). As an artist Nicolai uses strictly geometrical,
non-figurative and abstract forms, imagery and objects to create quasi-scientific/
quasi-fantastical installations and performances. Works such as reflektor distortion
explore notions of our perceptions of the real:

The installation reflektor distortion – conceived as a rotating, water-filled

basin – is inspired by the shape of a parabolic mirror that ‘rotates’ water
via centrifugal force. The work consists of the three main components
mirror, reflection and distortion. Both curve and distortion of the water sur-
face is affected by speed and integrated resistors that generate a perma-
nently new and re-organizing mirror ref lection. The water surface will be
supplementary distorted via speaker by resonating low sound frequencies.
126 Paul Goodfellow and Steve Gibson

The function of the mirror is hereby eminent: The mirror surface is the
medium that reveals reality as distorted ref lection. Rising the question of
the observed and the real image the installation plays with the artist’s thesis
that we all have a permanent distorted perception of reality.53

Other projects such as unidisplay (see Figure 5.6) use large immersive projec-
tions manipulated by Derivative’s TouchDesigner software (see Chapter 19 for
a discussion of this with programmer Markus Heckmann) in order to “interfere
with the viewers’ perception, through optical illusion, jitter, f licker, after-image,
movement, complementary colour effect.”54 The perception of the work is
undoubtedly affected by its gargantuan scale, as well as its presentation in a mas-
sive open space. The notion of scale is a concern throughout much of the work
of live visualists in the 21st century, particularly for those involved with public
projection. In some cases, this can simply be used for spectacular effect, such as
Drive Production’s enormous projection mapping on to Millbank Tower on the
Thames in London for Deadmau5’s performance as part of the Nokia Lumia

FIGURE 5.6 Carsten Nicolai, unicolor, 2014, DLP-projectors, DMX-LED lights,

projection screen, mirrors, computer, sound, bench with loudspeakers,
dimensions variable. Exhibition view, as part of the exhibition “City
and Nature,” Sapporo International Art Festival, Sapporo Art Museum,
Japan 2014. Photography: Julija Goyd. Courtesy Galerie EIGEN+ART
Leipzig/Berlin and Pace Gallery. © VG Bild-Kunst, Bonn 2021. Used
by permission.
Source: https://derivative.ca/community-post/carsten-nicolais-unidisplay-extended-private-viewing/
60776
 The Post-Conceptual Digital Era 127

phone launch.55 In the case of Nicholai, there is a subtler, more philosophical

angle at play which represents the quasi-scientific, formal foil to the surreal, but
meaningful work of other groups such as the aforementioned Klip Collective.
One interesting aspect of Nicolai’s work is how it spills over to his per-
formances by his alter ego, alva noto. He explicitly makes a clear connection
between his work as an installation artist and audio-visual performer. In addi-
tion, as discussed in Chapter 19, he pursues a very strict method of matching
his sound materials to visual results, with a 1–1 correspondences between the
mediums. This harkens back to the work of Norman McLaren such as Synchrony
(as discussed in Chapter 2).
It should be stressed that abstract and formal work in the 21st-century Live
Visuals world, while arguably dominant in many areas (VJing, digital installa-
tion, projection mapping), is balanced by other work that is alternatively quasi-
figurative, broadly realist and in some cases immersed in kitsch and populism. In
addition to the quasi-realist/surrealist work of Klip Collective, other groups such
as The Light Surgeons deal with realistic imagery (often in a documentary man-
ner), albeit mixed with geometric and quasi-abstract compositions (see Chap-
ter 17 for an interview with Chris Allen, the director of The Light Surgeons).
Others, such as London-based Polar Fantasy,56 use deliberately lo-fi and kitsch
elements such as lo-res 3D animations and old game footage in an over-the-top
series of works for different venues and music artists. This work ref lects (some-
what ironically) on past male-dominated audio-visual culture, using rave and
game culture as touchstones in order to both have fun with and poke fun at the
absurdity of some Live Visuals culture.
Thus, Live Visuals of the early 21st century exhibit a broad range of work
encompassing the abstract and formal through to the conceptual and political,
and this ref lects its rich heritage which can be traced through developments in
art over the past 100 years.

Conclusions
Within the trajectory of art history, art moved from a premodern era which was
representational, to the modern era, which was non-representational: from figu-
ration and representation towards abstraction and conceptualism. As a branch
of media art, contemporary Live Visuals emerged from art in the late 1960s
when the experiments in materials and abstraction from the formalist paint-
ers converged with the technological and conceptual experimentations of the
avant-garde. The convergence created the two self-sustaining but overlapping
ecosystems of contemporary art and media art, and this is discussed in Chapter 7.
As a form of media art, Live Visuals continue to exhibit the characteristics of this
heterogeneous mix of formalism and conceptualism.
Some works discussed in this chapter and the preceding chapters on the his-
tory of Live Visuals are motivated by the aesthetic and formal possibilities of the
medium, whilst other works are primarily motivated by the communication of
128 Paul Goodfellow and Steve Gibson

conceptual, social and political ideas, and the most successful work manages to
synthesise the aesthetic and conceptual together. Underpinning this broad spec-
trum of work is the necessary engagement with the ever-evolving technological
substrate which supports the investigation and use of Live Visuals. From early
cinema through to scratch video, rave and live cinema of the late 20th century
and on to the present 21st century and the digital era, the field of Live Visuals
has been shaped by technology, with Live Visuals exhibiting a dialectical nature,
as their aesthetic and conceptual content is defined by the potentialities and con-
straints of the systems, processes and technologies employed in their production.
This chapter has considered the technological developments of the 21st cen-
tury and, in particular, the shift to the digital production of Live Visuals. This
move effectively absorbed all prior developments of Live Visual production and
created digital equivalents to many analogue modes of production and per-
formance, such as scratch video. An advantage of the new digital methods has
been the ease with which the video material, digitised as data, can be stored,
transported, mixed and performed. A potential disadvantage has been the psy-
chological and technological break between the Live Visual performer and the
production of the live images, as the images are increasingly mediated by pre-
defined image processing tools and algorithms that act as a perceptual and cre-
ative ‘black box.’ As the digital works are reproducible and remixable by multiple
artists, the performed works do not exhibit that original essence (the notion of
‘aura’ and Live Visuals is discussed further in Chapter 7). These two potential
disadvantages have, however, been both understood and addressed by Live Visual
artists. Firstly, through the production of custom digital tools, using software
such as TouchDesigner. Secondly, through the rehabilitation of old equipment
to shift the focus away from the technological substrate and onto the formal and
conceptual content of the work.
Thus, it can be argued that the new millennium, defined as a period of Live
Visuals, can be described not so much by the technological developments but by
the fact that Live Visual artists are not constrained by technology anymore. Any-
thing, in a digital image production sense, is now possible, and any limits an art-
ist may experience are more likely to have a cultural, conceptual or psychological
basis rather than a technological basis. One constraint experienced by both the
Live Visual artist and the Live Visual artwork is the cultural distinction between
contemporary art and media art which locates such work outside of the institu-
tions and discourses of contemporary art. The historical and theoretical basis for
the divide between contemporary art and media art is addressed in Chapter 7
with a view to initiating a more inclusive dialogue between these two essentially
complementary fields. The second constraint exhibited within some contempo-
rary Live Visual artwork is that the conceptual content is overshadowed by the
work’s ‘spectacular’ nature – a condition also discussed in Chapter 7. The power
of the new tools and processes for the production and dissemination of work,
such as massive light-emitting diode (LED) walls or projection-mapped spaces,
translates even a conceptually driven work into an experience of visual teeming
 The Post-Conceptual Digital Era 129

excess for the audience who experience the Live Visuals primarily as sensation
and affect.
The final constraint or ‘intensity’ which has shaped the concerns of the Live
Visual artist in the early 21st century is the psychological conditions, or the
changes to our ‘lifeworld’ as our lives become increasingly mediated by digital
technology. This chapter has discussed how we have become enfolded within
technological systems in the new millennium and how this has impacted our
sense of self, how we engage with others and how we engage with nature. The
Earth has seen dramatic changes due to the climate crisis and the loss of habitats
and biodiversity, and how this has been monitored, mapped and communicated
has increased our sense of alienation from nature. Likewise, social media has
exponentially increased the transfer of information but reduced shared meaning
as individual users are segregated within bubbles, and even language has become
unstable, contested and stratified.
These intensities have had a profound effect on both the individual lifeworld
and culture as a whole, and several terms have been employed in this chapter to
describe the overarching or expanded melancholia we feel in these first decades
of the 21st century. These feelings include saudade for a lost relationship with
nature, and this would include the yearning we feel for the pre-digital age when
there was a more direct indexical relationship between the photographic image
and the world it represented. We have also experienced weirdness and the teem-
ing excess of information that pervades our waking lives, or the teeming excess
of COVID-19 and our inability to apprehend its invisible reach. We have also
experienced eeriness in the disembodied nature of communication in the age of
social media and Zoom calls.
We are also experiencing something more profoundly eerie as we become
decentred within our own experience of ourselves. In the 21st century, we have
increasingly transferred and distributed ourselves online, and this simulation of
the self increasingly dominates how we see ourselves and others. Jean Baudrillard
has suggested that a perceptual problem arises when the simulation of the subject
becomes so complex that there is a break between the ‘sign’ and the thing being
described, stating that “simulation envelops the whole edifice of representation
itself as a simulacrum.”57 He described this shift from equivalent sign to simula-
cra taking place in the following stages:

it is the ref lection of a profound reality;

it masks and denatures a profound reality;
it masks the absence of a profound reality;
it has no relation to any reality whatsoever: it is its own pure simulacrum.58

We therefore feel eeriness when we see the simulation of ourselves in the mel-
ancholic art of the early 21st century in computer games or ref lected back at us
from our social media streams. We feel eeriness, in the two senses described by
Fisher, as a “failure of absence” and a “failure of presence.”59
130 Paul Goodfellow and Steve Gibson

[T]here is an alien presence which compels speculation, as we look at our

idealised simulation as a form of apparition or haunting. These algorithmi-
cally enhanced images are haunted; not in reference to the supernatural
but the virtual, as they present a picture of the self which has never materi-
ally existed.60

Our simulation on the Internet also exhibits a “failure of presence,” as the

“socially networked self ” exists without a body or material presence. Likewise,
digitally based art, and this would include Live Visuals, also demonstrate a “fail-
ure of presence,” as they live only as data, pixels and code.
Live Visuals in the early 21st century have demonstrated that there are now
no practical limits to the production and remixing of digital images. Photo-
graphic, filmic and data-based information can be endlessly reconfigured to
create compelling images which can be disseminated via screens, projection-
mapped spaces and translated into lighting systems to fully immerse the audi-
ence into Live Visual experiences. This is not, however, the endpoint in the
development of Live Visuals, as the next stage of evolution will see Live Visu-
als transcend the conceptual boundaries of the f lat screen to create fully
immersive, four-dimensional Live Visual experiences. The possibility of full
immersion within digital environments through virtual reality (VR) is starting to
appear as the technology and graphical processing power are becoming afford-
able options for the media artist. Likewise, the technical knowledge and formal
understanding which underpin these new multidimensional aesthetic spaces are
being investigated and understood by artists. One can speculate that the spatio-
temporal concerns of past movements such as futurism and cubism will now be
returned to as space-time will finally be visualised with live, four-dimensional
visuals.
The post-conceptual condition that Live Visuals inhabit (by design or acci-
dent) suggests a new collaborative model of audio-visual production combin-
ing formal and conceptual methods from the past and the present. We can see
this mixture in works discussed, such as Klip’s What’s He Building in There? The
new promiscuousness of post-conceptualism can be seen as a possible source of
strength for Live Visuals, though as we have noted throughout, the tendency
towards the spectacle for its own sake is a constant potential pitfall for Live Visuals
culture, which we have seen played out with somewhat depressing regularity over
the past 20 years. On the other hand, the 21st century has also seen the reintegra-
tion of formal strategies in the digital artwork, while still generally resisting the
modernism impulse to turn every movement into a formalist manifesto factory.
This is evident in a diverse body of work from the Relational Architecture series of
Rafael Lozano-Hemmer to the uni-verse series of works from Carsten Nicolai. If
Live Visuals are going to have continued significance, both as an art movement
and as part of wider media art culture, then they will f lourish best if notions of
concept and form can be reunited in a way that is neither too rigid nor too loose.
 The Post-Conceptual Digital Era 131

Notes
1 Gilles Deleuze and Félix Guattari, A Thousand Plateaus: Capitalism and Schizophrenia,
New edition (London and New York: Continuum International Publishing Group,
2004).
2 Ian Buchanan, Assemblage Theory and Method: An Introduction and Guide (London: Blooms-
bury Academic, 2020), 37.
3 Gilles Deleuze and Félix Guattari, A Thousand Plateaus: Capitalism and Schizophrenia,
New edition (London and New York: Continuum International Publishing Group,
2004), 46–84.
4 See Brian Kane Official website, “Vujak” page, https://briankane.net/tag/vujak/.
5 Erich Horl, General Ecology: The New Ecological Paradigm, edited by James Edward Burton
(London: Bloomsbury Academic, 2017).
6 Paul Goodfellow, “Eerie Systems and Saudade for a Lost Nature,” Arts, 8(4), 2019,
https://doi.org/10.3390/arts8040124.
7 Lana Wachowski and Lilly Wachowski, The Matrix (Burbank, CA: Warner Bros, 1999).
8 Chad Engelland, Phenomenology, The MIT Press Essential Knowledge Series (Cam-
bridge, MA: MIT Press, 2020).
9 “Year 2000 Problem,” Wikipedia, https://en.wikipedia.org/w/index.php?title=Year_2000_
problem&oldid=1057148743 (Accessed November 25, 2021).
10 “BBC News | World | Asia-Pacific | Japan Nuclear Plants Malfunction,” http://news.
bbc.co.uk/1/hi/world/asia-pacific/585950.stm (Accessed November 27, 2021).
11 Howard Rheingold (ed.), Millennium Whole Earth Catalogue (San Francisco: HarperCol-
lins, Australia, 1995), 1.
12 Jean-François Lyotard, The Postmodern Condition: A Report on Knowledge, Theory and
History of Literature (Minneapolis: University of Minnesota Press, 1984).
13 George P. Landow (ed.), Hypertext: The Convergence of Contemporary Critical Theory and
Technology (Baltimore: The Johns Hopkins University Press, 1991), 4.
14 Barry Miles, William S. Burroughs: A Life (London: Orion Publishing Group Limited,
2015).
15 Francis Fukuyama, The End of History and the Last Man (New York: The Free Press,
1992), 1.
16 Howard Rheingold (ed.), Millennium Whole Earth Catalogue (San Francisco: HarperCol-
lins, Australia, 1995), 1.
17 Katherine N. Hayles, How We Became Posthuman: Virtual Bodies in Cybernetics, Literature,
and Informatics, 74th edition (Chicago, IL: University of Chicago Press, 1999).
18 Katherine N. Hayles, How We Became Posthuman: Virtual Bodies in Cybernetics, Literature,
and Informatics, 74th edition (Chicago, IL: University of Chicago Press, 1999).
19 Ludwig Von Bertalanffy, Wolfgang Hofkirchner, and David Rousseau, General System
Theory: Foundations, Development, Applications, Revised edition (New York: George Bra-
ziller, 2015).
20 Katherine N. Hayles, How We Became Posthuman: Virtual Bodies in Cybernetics, Literature,
and Informatics, 74th edition (Chicago, IL: University of Chicago Press, 1999), 223–224.
21 Katherine N. Hayles, How We Became Posthuman: Virtual Bodies in Cybernetics, Literature,
and Informatics, 74th edition (Chicago, IL: University of Chicago Press, 1999), 224.
22 Shumon Basar, Douglas Coupland, and Hans Ulrich Obrist, The Extreme Self (Cologne:
Verlag der Buchhandlung Walther Konig, 2021), 58.
23 Gene Youngblood, Expanded Cinema (New York, NY: E. P. Dutton, 1970).
24 Rosalind Krauss, “Sculpture in the Expanded Field,” October, 8, 1979, 31–44, https://
doi.org/10.2307/778224.
25 Fredric Jameson, The Cultural Turn: Selected Writings on the Postmodern, 1983–1998
(Brooklyn: Verso, 2009), 9.
26 Hal Foster, Bad New Days: Art, Criticism, Emergency (Brooklyn: Verso Books, 2015),
31–60.
132 Paul Goodfellow and Steve Gibson

27 Jörg Heiser, “Analyze This,” Frieze, https://frieze.com/article/analyze?language=de

(Accessed August 6, 2019).
28 William Gibson, “Opinion | Google’s Earth,” The New York Times, August 31, 2010, sec.
Opinion, www.nytimes.com/2010/09/01/opinion/01gibson.html.
29 Félix Neto and Etienne Mullet, “A Prototype Analysis of the Portuguese Concept of
Saudade,” Journal of Cross-Cultural Psychology, 45(4), 2014, 661, https://doi.org/10.1177/
0022022113518370.
30 Jörg Heiser, “Analyze This,” Frieze, https://frieze.com/article/analyze?language=de
(Accessed August 6, 2019).
31 M. Fisher, The Weird and the Eerie (London: Watkins Media Limited, 2016).
32 M. Fisher, The Weird and the Eerie (London: Watkins Media Limited, 2016), 61.
33 M. Fisher, The Weird and the Eerie (London: Watkins Media Limited, 2016), 61.
34 Universal Everything, Infinity Website, 2021, www.universaleverything.com/projects/
infinity.
35 Peter Osborne, Anywhere or Not at All: Philosophy of Contemporary Art (London: Verso
Books, 2013), 3 & 51.
36 Joseph Nechvatal, “The Viractuality of Post-Conceptual Art,” 2013, https://joseph
nechvatal.wordpress.com/2013/08/17/the-viractuality-of-post-conceptual-art/#_ftn7.
37 See Steve Gibson and Stefan Arisona, Virtual VJ Solo, Steve Gibson Vimeo page, https://
vimeo.com/manage/videos/228642288.
38 VJ Union Global Facebook page, www.facebook.com/groups/510530552353856.
39 MUTEK Festival Official website, https://mutek.org/.
40 Mapping Festival 2051 Official website, https://2051.mappingfestival.com/.
41 United Visuals Artists Official website, www.uva.co.uk/.
42 Light Surgeons Official website, www.lightsurgeons.com/.
43 Squidsoup Official website, www.squidsoup.org/.
44 Novak Collective Official website, https://novak.uk/.
45 MXZehn Official website, https://mxav.net/.
46 Derivative Official website, https://derivative.ca/.
47 Hamill Indutries Official website, https://hamillindustries.com/.
48 Klip Collective Official website, www.klip.tv/.
49 Tom Waits, What’s He Building, Tom Waits YouTube page, 2020, www.youtube.com/
watch?v=OeaRVeV8DdA.
50 Klip Collective, What’s Building in There? Documentation, Reelouse Official website,
2013, www.reelhouse.org/klipcollective/whatshebuildinginthere.
51 ElizaGalesInterviews, “Interview with Artist Ricardo Rivera,” Reelhouse Official web-
site, 2013, www.reelhouse.org/klipcollective/whatshebuildinginthere/update/51758a8
21554ab080000002f.
52 Steve Gibson, “Being Formal without Being a Formalist,” Leonardo Just Accepted publi-
cation January 30, 2021, 2–3, https://direct.mit.edu/leon/article/doi/10.1162/leon_a_
02056/98082/Being-Formal-Without-Being-a-Formalist.
53 Carsten Nicolai, Reflektor Distortion, 2016, www.carstennicolai.de/?c=works&w=reflektor_
distortion.
54 Carsten Nicolai, Unidisplay, 2012, www.carstennicolai.de/?c=works&w=unidisplay.
55 Drive Productions, Nokia Lumia Live Ft. Deadmau5, 2012, www.youtube.com/watch?v=
BAhmyJFgtVA.
56 Polar Fantasy Instagram account: www.instagram.com/polarfantasy/.
57 Jean Baudrillard, Simulacra and Simulation, translated by Sheila Glaser (Ann Arbor: Uni-
versity of Michigan Press, 1994), 6.
58 Jean Baudrillard, Simulacra and Simulation, translated by Sheila Glaser (Ann Arbor: Uni-
versity of Michigan Press, 1994), 6.
59 Mark Fisher, The Weird and the Eerie (London: Watkins Media Limited, 2016), 61.
60 Paul Goodfellow, “Eerie Systems and Saudade for a Lost Nature,”Arts, 8(4), 2019, https://
doi.org/10.3390/arts8040124.
PART II

The Theory of Live Visuals

6
CROSS-MODAL THEORIES OF
SOUND AND IMAGE
Joseph Hyde

Introduction
This chapter will examine theoretical frameworks that have developed through
the history of audio-visual forms and practices. These frameworks are of rel-
evance to Live Visuals but are usually not exclusive to this area of work, since
the issues at play between sound and image are generally the same regardless of
whether they are produced in real time or not. For that reason, this discussion
will not make a distinction between the two.
Some of these frameworks have long histories – this chapter will take a his-
torical view where this is useful, but will focus on theory that is relevant and
in use by contemporary artists. In particular, it will focus on key developments
in the 20th century. This period is important to this discourse for a number of
reasons. Firstly, and most obviously, this is because it is in the 20th century that
audio-visual practice, as in moving images with synchronised sound, became
not only possible but dominant as a cultural and artistic practice. Secondly, in
the 20th century both audio and visual artistic practices underwent fundamen-
tal changes which facilitated new common ground between them. One might
make the assumption that the latter was caused by the former, but the order of
historical causality indicates otherwise. In reality we can see a complex interplay
between these elements which this chapter will attempt to unpack.

Colour
The idea of colour music is the earliest concept we will discuss here. The his-
tory of colour organs is covered in Chapters 1 and 2 – the ideas behind it in fact
date back at least to Aristotle (who wrote in De Anima: “In the pleasing nature
of their harmony colours can be related like musical sounds and be mutually
DOI: 10.4324/9781003282396-9
136 Joseph Hyde

proportional”1), can be seen in the writings of Leonardo da Vinci and were con-
siderably developed by Isaac Newton.
Colour music is most often predicated on the idea that colour and musical
pitch can be theoretically linked, since both can be equated with frequency. It
is important to state from the outset that whilst this idea has proved attractive
to a number of artists, musicians, instrument builders and theoreticians, it has
also been problematised. John Whitney wrote: “each Century since Leonardo a
vision, grand and obscure as its myth, compelled one or two inventors to struggle
with the pathetic inadequacies of the color organ.”2 Fred Collopy, who has writ-
ten extensively on the topic, produced a chart of ‘300 years of color scales’ which
demonstrates more how the various systems that have evolved differ from each
other than what they have in common.3
Newton’s theories in this regard are interesting and somewhat mysterious,
deeply embedded in our culture, but much misunderstood. It is useful to note
that these theories effectively seeded the development of colour organs, but
through a lineage which is not just indirect, but oppositional. The first such
instrument to be commonly discussed is Louis Bertrand Castel’s Clavecin pour les
yeux, first documented around 1730 (as discussed in Chapter 1). This was clearly
inf luenced by Newton’s ideas, first outlined in a paper entitled “New Theory
About Colours” presented to the Royal Society in 1672, and then further devel-
oped in Opticks.4 Castel was, however, a vocal critic of Newton’s theories and
approach – his criticism was largely based on nationalism and philosophical dif-
ferences, the latter centring on a belief in analogic and proto-phenomenological
as opposed to empirical scientific methodology.
This has perhaps affected the subsequent history of colour music and led to
some of its shortcomings. With this in mind, it can be interesting to return to

FIGURE 6.1 Newton’s circle of colours published in Opticks (1704).

Source: Creative Commons Attribution 4.0 International, www.researchgate.net/figure/Newtons-
colour-circle-published-in-Opticks-1704_fig3_273324129
 Cross-Modal Theories of Sound and Image 137

Newton’s original writings, which in some ways avoid some of the inconsisten-
cies of subsequent interpretations.
Newton’s theories as manifested (or misinterpreted) in the subsequent history
of colour music can be summarised in Figure 6.1, taken from Opticks.5 Newton is
often interpreted as equating the spectrum of colours (from red to violet) with a
C major scale, C to B. However, as can clearly be seen here, Newton in fact made
this analogy with a Dorian mode scale, D to C. More interestingly, musical notes
were not in fact equated with specific colours, but with boundaries between
colour bands – “from D to E be all degrees of red, at E the mean colour between
red and orange, from E to F all degrees of orange, at F the mean between orange
and yellow, from F to G all degrees of yellow, and so on.”6 This represents a more
subtle and nuanced approach than that used by most subsequent proponents of
colour music. Interestingly, while Newton uses the model of a musical scale to
divide his colour space, he never speculates that colour might in any way equate
to an actual musical scale. It is clear that his colour wheel, arranged across a musi-
cal octave, represents the entire spectrum of visible colour. In taking this analogy
literally, proponents of colour music have fallen into a trap in that one musical
octave does not represent the entirety of the human hearing range, which is at
least eight octaves. Colour musicians have never quite known what to do with
this disparity – Alexander Rimington (see Chapter 2) arranged the colours red
to violet across a single musical octave C to B, with higher octaves represented
through increased brightness, but this mechanism does not stand up to much
scrutiny.
If one avoids this over-literal interpretation of Newton’s ideas, we find some
theoretical basis to the analogy that still stands. Principally, viewing the entire
colour space as an octave, does offer some parallels with a musical octave. The
most interesting of these is the perceptual ‘circularity’ of both frequency spaces.
As one reaches the top of a musical octave, one arrives at the starting point again,
and this implies certain directionalities on which many musical structures are
built. Newton’s diagram is reminiscent of a modern-day colour wheel (as found
in image or video editing software applications), where one can see that the
transition from violet to red (in Newton’s model centring on the note D) is as
seamless as that between any of the other colours, just as in musical pitch space.
Interestingly, the full range of perceptual colour space involves something close
to a doubling in frequency (from around 400 to approaching 800 Thz), just as a
single musical octave does.
Rimington discusses both phenomena in his writings, published between
1895 and 1912.7 At first sight, Rimington’s solution to the ‘octave problem’
seems compatible with this colour wheel model – with reference again to a con-
temporary colour wheel, one might visualise successive pitch octaves circling
around the wheel but spiralling in towards its white centre. But this does not
bear scrutiny on a perceptual or physical level – in reality, the sonic equivalent
of this progression would not be ‘higher pitched,’ or even ‘brighter,’ but ‘noisier.’
Generally, the biggest weakness of Rimington’s model, and most of those at play
138 Joseph Hyde

in colour music, is that it does not acknowledge the spectral qualities of musical
timbre. A musical note will usually contain frequencies other than the funda-
mental in the form of harmonics (and these will often evolve over time). It is
hard to envisage a model for colour that exactly follows this principle, although
Aristotle’s ideas on colour and pitch centred on ‘harmonic’ (as in low-order ratio)
relationships between certain colours, and related ideas can be found in the theo-
ries of 20th century colour theorists such as Josef Albers. In a broad sense, devel-
opments in both fine art and music in the 20th century saw a subtler approach to
frequency which made more complex models possible, which will be discussed
in the next section.

Tone
As discussed in Chapter 2, the term ‘visual music’ first appeared in the early
20th century. It was first applied to abstract painting, and more accurately,
cubist and post-expressionist painting. Visual music is generally associated, cer-
tainly in connection with that historical period, with a project to bring to paint-
ing some of the qualities seen and admired in music. This can be seen as part
of a broader movement, rebuilding the foundations of art and music, breaking
apart the largely European hegemonies of the 19th century, admitting to an
extent the inf luence of non-western cultures whilst building new models in the
New World and laying the foundations of modernism. In this profound shake-
up, ideas from the world of visual arts inf luenced music as much as vice versa,
and both were inf luenced by new ideas around science, philosophy, politics and
spirituality.
Kandinsky is perhaps the artist most associated with visual music in its ear-
liest incarnation, and his work is covered in depth elsewhere in this volume.
There are a few key elements in his theoretical framework that will be useful to
our discussion. Kandinsky is often associated with synesthesia, or cross-sensory
perception. This phenomenon seems likely to have been part of the human con-
dition for as long as our species has existed, but it attracted particular attention
during the 19th and 20th centuries. Kandinsky’s is a typical case, and whilst
there is some doubt as to whether he actually experienced synesthesia as a neu-
rological condition, it was an idea that fascinated him and informed much of
his writing.
Here are two descriptions of colour from Concerning the Spiritual in Art:

Warm red, intensified by a suitable yellow, is orange. This blend brings

red almost to the point of spreading out towards the spectator. But the ele-
ment of red is always sufficiently strong to keep the colour from f lippancy.
Orange is like a man, convinced of his own powers. Its note is that of the
angelus or of an old violin.
Violet is therefore both in the physical and spiritual sense a cooled red.
It is consequently rather sad and ailing. It is worn by old women, and in
 Cross-Modal Theories of Sound and Image 139

China as a sign of mourning. In music it is an English horn, or the deep

notes of wood instruments (e.g. a bassoon).8

A number of factors can be identified here: the range of Kandinsky’s colour

associations is quite broad and by no means limited to colour/sound analogies.
Kandinsky incorporates associations between colours (a key theme in this text)
and associations between colour and forms – even, in the first instance here,
between colour and motion. He also associates colour with emotions and other
mental and spiritual states and acknowledges pre-existing cultural associations.
Crucially for our discussion, he tends not to associate colour with pitch, but
rather with instrumentation, and therefore by extension timbre.
An important factor here is the link between Kandinsky and many other of
the artists and composers discussed here with theosophy. It is beyond the scope
of this chapter to discuss the theosophical movement and the beliefs that went
with it, but key components of this belief system are of particular relevance
here. One is the incorporation of many elements of non-western philosophy
and spirituality – this can often be viewed alongside an interest in elements of
art and music from these cultures. An early example of this in music can be seen
in the work of Claude Debussy. His interest manifested itself in many qualities
in the music itself, for instance, in rethinking the roles of melody, harmony,
rhythm and timbre. Most particularly, it perhaps resulted in a revolution in terms
of musical form and a turn away from models based on tension and release and
towards a more non-teleological framework.
One key idea, or set of ideas, centres around vibration being at the root of
all things, including – crucially – ourselves and our consciousness and sensory
apparatus. This idea has complex roots that can be traced back to Hinduism. It
is also, of course, part of various contemporary belief systems, many of which
may be viewed with some suspicion. One might characterise these beliefs as ‘new
age,’ and that is rather precisely correct, in that the New Age movement, along
with other systems such as those of Rudolph Steiner, are directly descended from
theosophy. In the 1960s and 1970s these ideas were at the forefront of avant-
garde culture and connected to the psychedelic movement. Interestingly, in the
‘psychedelic textbook’ of Expanded Cinema, Gene Youngblood states that “syn-
aesthetic and psychedelic mean approximately the same thing.”9 Later, Karlheinz
Stockhausen writes:

There exists in India a secret knowledge based on the study of sounds and
the difference of vibratory modality according to the planes of conscious-
ness. . . . The mantra or great poetry, great music, the sacred word, comes
from the overmind. This is the source of all creative and spiritual activities. . . .
When the consciousness is transparent the sound becomes clearly audible,
and it is a seeing sound, a sound-image or the sound-colour or a sound-idea,
which links indissolubly in the same luminous body the audition to the
vision and the thought. All is full, contained in a single vibration.10
140 Joseph Hyde

According to Kandinsky: “colour is the keyboard, the eyes are the hammers, the
soul is the piano with many strings. The artist is the hand which plays, touching
one key or another, to cause vibrations in the soul.”11 We see that synesthesia is
not employed in a literal sense here, but rather in a metaphorical one, as part of a
broader philosophy and belief system.
Kandinsky and the composer Arnold Schoenberg were close associates – they
were both part of the Blaue Reiter movement, corresponded regularly and wrote
extensively on each other’s work (see Chapter 2). Their writings show that they
felt they shared a lot in terms of the ideas behind their work. Schoenberg was
also interested in theosophy, though perhaps to a lesser extent than Kandinsky. A
stronger commonality in their ideas can be found in a shared interest in the work
of Sigmund Freud, fuelling the idea of Expressionism in both of their work. This
was perhaps less about the expression of subconscious emotion and more about
the freedom to do so – freedom of expression. Both sought to explore such free-
dom by breaking the constraints of their artform. In Kandinsky, this involved
breaking the constraints of representation; for Schoenberg, challenging tonality.
Schoenberg’s pre-war Expressionism clearly demonstrates this aim. In contrast
to the rigid systematisation represented by serialism in his later work, here he
seems to be exploring the opposite – a kind of stream of consciousness akin to the
‘automatic writing’ employed by Freud. He wrote to Kandinsky that his music
was “without architecture, without structure. Only an ever-changing, unbroken
succession of colours, rhythms and mood.”12 In the process, he believed that his
music “was drawing close to the principles of contemporary painting.”13
Of particular interest for this discussion is the idea of Klang farbenmelodie. The
term was derived from Klang farbe (sound-colour), which already had consider-
able currency at the time and represented an inf luential concept. This term,
dating back to the early 19th century, was popularised by Hermann von Helm-
holtz in his book On the Sensations of Tone as a Psychological Basis for the Theory of
Music, published in 1863.14 Helmholtz uses the word pairing Musikalische Klang-
farbe, referring to the spectral content of musical tones, which he observes are
often as important for human perception as the fundamental. His work did for
sound what Newton’s did for light a century and a half earlier and was based on
fascinating experimental instruments such as the electromechanical Helmholtz
Sound Synthesizer (Figure 6.2).
The idea that Musikalische Klang farbe (close, but not quite identical in mean-
ing to ‘timbre’) was as important as pitch led composers to blur the boundaries
between the functions of harmony and timbre. Interestingly, the composers most
associated with this evolution in musical language were often engaged in multi-
media practice (Wagner and his Gesamtkunstwerk) or were exploring visual paral-
lels between audio and visual farbe. In the title page of the score of his Nocturnes
(1897–1899), inspired by paintings by James Abbott McNeill Whistler, Debussy
offers this (ironically rather monochromatic) description: “Nuages renders the
immutable aspect of the sky and the slow, solemn motion of the clouds, fading
away in grey tones lightly tinged with white.”15
 Cross-Modal Theories of Sound and Image 141

FIGURE 6.2 Helmholtz Sound Synthesizer, the Collection of Historical Scientific

Instruments, Harvard University. Used by permission.
Source: http://waywiser.rc.fas.harvard.edu/view/objects/asitem/items$0040:14588

In his Klang farbenmelodie, Schoenberg seems to be taking these ideas a step fur-
ther. In contrast to the later works of Webern, he uses the sequence represented
by the Klang farbenmelodie not in addition to a pitch sequence (or series), but rather
as an alternative. This represents a prescient type of spectral music and dem-
onstrated an understanding of some of the principles discovered by Helmholtz
(Schoenberg was indirectly aware of these through the writings of psychologist
Carl Stumpf ). In particular, Schoenberg exploited Helmholtz’s discovery that
the divisions between notes as distinct entities can be blurred if their distinc-
tive noisy attacks are obscured. This is explored in particular in his Five Pieces
for Orchestra (1909), and in particular in the third movement – Farben – in which
fairly static harmonies are articulated through seamless changes in orchestration.
Schoenberg summarises this approach as follows:

The distinction between tone color and pitch, as it is usually expressed, I

cannot accept without reservations. I think the tone becomes perceptible
by virtue of tone color, of which one dimension is pitch. Tone color is,
thus, the main topic, pitch a subdivision. Pitch is nothing else but tone
color measured in one direction.16

Another synesthetic composer is Olivier Messaien, who extends the idea of

klang farbenmelodie into his Mode de valeurs et d’intensités, often cited as an imme-
diate precursor to integral serialism (though Messaien was unhappy with the
idea). Messaien was a natural synesthete, and his synesthesia, often referred to in
his writings, is described in subtle and complex terms. Again, there is nothing
as simple as a pitch/colour correspondence at play. Like Schoenberg, Messiaen
142 Joseph Hyde

likens colours to tonalities and modes, particularly his ‘modes of limited trans-
position.’ Note that these modes were seen by Messiaen more as harmonic than
melodic devices – he wrote: “People have often referred to my modes of lim-
ited transposition as scales. They are not scales, but harmonic colours.”17 These
‘harmonic colours’ were often described by Messaien in evocative terms which
include elements of form and texture rather than just purely colour-based imag-
ery, for example: “transparent sulphur yellow with mauve ref lections and little
patches of Prussian blue and brown purplish-blue.”18

Vibration
If we follow this path and take patterns of vibration to be the key element of
audio-visual interaction, we find a rich history of work in the arts, science and
technology exploring this. What is interesting about this type of interaction is
that it can be demonstrably direct, with sound producing image (or vice versa)
in a readily comprehensible cause-and-effect chain which does not rely on spec-
ulative theory, arbitrary mappings or elaborate philosophies. This model has
become familiar through the waveform displays (graphing amplitude against
time, i.e. motion) we often use to interact with sound in a technological envi-
ronment, but it has a long history.
Ernst Chladni’s Entdeckungen über die Theorie des Klanges, written in 1787,19 is
the first thorough exploration of acoustics. It contains the famous ‘sound figures’
(Figure 6.3) which were produced by setting in vibration thin metal plates on
which sand had been sprinkled. The vibrating movement of the plates shifts the
sand into the distinctive patterns which show not just the harmonic content of
these vibrations but also how they travel across the surface of the plates. This phe-
nomenon was explored more deeply but much later by Hans Jenny in his book
Cymatics, The Structure and Dynamics of Waves and Vibrations, published in 1967.20
Related ideas were explored by physiologist Charles Wheatstone and his
invention the kaleidophone (1827), inspired by David Brewster’s kaleidoscope
(1817). This device has mirrored beads attached to the top of metal rods, mounted
perpendicular to a wooden base (Figure 6.4). With a light source such as a candle
mounted close to the mirrored beads and the rods made to vibrate, patterns of
light similar to, but more complex than, Chladni’s figures were produced. These
ideas were taken further by mathematician and physicist Jules Lissajous, who used
the stereoscope kaleidophone and later his own experimental setup to explore
the interaction of two perpendicular vibrations. His experiment, documented in
1857, consists of two small mirrors mounted on tuning forks and is astonishingly
similar to the galvanometer module found in a contemporary laser projector.
The mirror galvanometer itself was patented by Lord Kelvin in 1858, though
this was developed from research published by Hermann von Helmholtz in 1849.
Neither version was intended to transcribe sound, but rather electromagnetic
vibration. It was used to produce the patterns now known as Lissajous figures
(see Figure 6.5), which have become a mainstay of audio-visual practice.21
 Cross-Modal Theories of Sound and Image 143

FIGURE 6.3 Chladni Sound Figures, Share-Alike, https://echo.mpiwg-berlin.mpg.de/

ECHOdocuView?url=/permanent/library/5M6VYMSC/pageimg&start=
91&viewMode=auto&pn=94&mode=imagepath
Source: https://publicdomainreview.org/collection/chladni-figures-1787

Édouard-Léon Scott de Martinville’s phonautograph, patented in 1857, is often

seen as one of the first audio recording technologies, predating Edison’s phono-
graph by two decades. There is much in common between the two devices – both
employ a membrane as the interface between acoustic and mechanical vibration
(modelled on a similar mechanism in the human eardrum), and both use a stylus
144 Joseph Hyde

FIGURE 6.4 Wheatstone’s kaleidophone (1827).

Source: Creative Commons, https://en.wikipedia.org/wiki/Kaleidophone#/media/File:Kaleidophone-
IMG_7007-gradient.jpg

to transcribe this vibration onto a surface. In Scott de Martinville’s case, this was
literal transcription (a phonautogram) – the stylus scratched a piece of smoke-black-
ened glass to produce a trace that is visibly the antecedent of today’s waveform
displays (Figure 6.6). Relevant to our discussion is that Scott de Martinville’s
aim was not to reproduce sound, and indeed his device had no mechanism to
do so. Rather, the final aim here was simply to write or visualise sound to further
understand it, and the phonautograph and similar devices which followed it (one
of which actually employed preserved parts of a human ear) was the analysis of
hearing, acoustics, speech and to a lesser extent musical pitch.
In 1880, Alexander Graham Bell and his assistant, Charles Sumner Tainter,
invented the photophone. This was developed more or less simultaneously with
 Cross-Modal Theories of Sound and Image 145

FIGURE 6.5 Lissajous Curves, Creative Commons, www.historyofinformation.com/

detail.php?id=3263.
Source: Alternate version, a photo is Creative Commons sourced at https://en.wikipedia.org/wiki/
Kaleidophone#/media/File:Kaleidophone-IMG_7007-gradient.jpg

the telephone, but transmitted an audio signal (usually a voice) by modulating

the movement of a light signal, again using a vibrating mirror. Although the tele-
phone, and its electromagnetic model, is the more famous of Bell’s inventions,
he regarded the photophone as the greater. This opinion has perhaps been borne
out subsequently, since the technique involved has much in common with the
fibre-optic model of contemporary telecommunications. Bell also seems to have
been interested in the cross-modal and the audio-visual nature of his invention:
“I have heard articulate speech produced by sunlight! I have heard a ray of the
sun laugh and cough and sing!. . . I have been able to hear a shadow, and I have
even perceived by ear the passage of the cloud across the sun’s disk!”22
Singer and philanthropist Margaret Watts Hughes invented a device she called
the eidophone, documented in an article in the Century in 1891.23 The device
had much in common with the transmitter of Bell’s photophone and consisted of
a membrane stretched over a horn connected to a mouthpiece (Watts produced
several variations of differing sizes and forms). Watts initially placed small seeds
146 Joseph Hyde

FIGURE 6.6 Phonautogram, First Sounds project. Used by permission.

Source: www.firstsounds.org/research/articles/scott-discography.pdf

on this membrane and observed the movement of these seeds to measure the
power of her voice. She subsequently made an interesting discovery:

On one occasion as I sang I noticed that the seeds which I had placed
on the India rubber membrane, on becoming quiescent, instead of scat-
tering promiscuously in all directions and falling over the edge of the
receiver onto the table, as was customary when a rather loud note was
sung, resolved themselves into a perfect geometrical figure.24

This phenomenon clearly resembles Chladni’s experiments, but actually has

more in common with Hans Jenny’s much later work with Cymatics. Watts went
on to develop her techniques more thoroughly, replacing seeds and powders with
pigments and pastes of various consistencies (akin to the various liquids, etc.,
used by Jenny) and ‘printing’ the results onto glass plates. At the same time, she
evolved her vocal technique specifically in light of the forms produced by the
eidophone. The specifics of this are not well documented, but from the results
one can make an assumption that she was perhaps exploring some kind of over-
tone singing. In any case, the results are stunning (Figure 6.7) and a harbinger of
the psychedelic vocabulary of the 1960s liquid light show (see Chapter 3), both in
their use of coloured liquids and glass plates to produce images and in the nature
of the images themselves.
The cross-modality of sound and light through motion became a far less eso-
teric concern through cinema and the introduction of sound film, or ‘talkies,’
 Cross-Modal Theories of Sound and Image 147

FIGURE 6.7 Margaret Watts Hughes (date unknown). Impression Figure. “Octave and
5th interval Bb.” Pigment on glass. Photograph by Rob Mullender-
Ross, Courtesy of Cyfarthfa Castle Museum and Art Gallery and Rob
Mullender-Ross.
Source: https://publicdomainreview.org/essay/picturing-a-voice-margaret-watts-hughes-and-the-
eidophone

in the 1920s. Various technologies were at play in ‘sound on film’ at this stage
in its development. The most famous of the early talkies, 1927’s The Jazz Singer,
utilised Warner Brothers’ Vitaphone sound-on-disc system, in which 16-inch
phonographic records were synchronised with the film projector. However,
from the 1930s, optical methods of sound recording and reproduction became
prevalent and would remain so until the 1970s. These methods were a direct
descendant of many of those described earlier – the recording element of most
optical sound systems bears much in common with Bell’s photophone, involving
an acoustically agitated vibrating mirror directing light to expose a trace on cel-
luloid film. This recalls Scott de Martinville’s phonautograph. The reproduction
of sound relied on more recent electronic technology, in particular, the intro-
duction of the thallium oxysulfide cell, a light-sensitive vacuum tube invented
by American physicist Theodore Case in 1917.
In some ways, particularly where variable area techniques are employed, an
optical film soundtrack clearly resembles a contemporary waveform display.
It is to an extent ‘human comprehensible,’ allowing at least some of the fea-
tures of a sound to be ‘read’ with the eye. As such, it represents the fulfilment
of the dream postulated by the phonautograph and has exerted a fascination
148 Joseph Hyde

over creative minds – inventors, technologists, artists and musicians – since

its introduction. The translation of sound into a visible trace and back again
offered the tantalising possibility of not only ‘reading’ but of ‘writing’ sound,
and therefore opened up possibilities for the very first entirely synthetic, or
synthesised, sound.
One of the first known uses of synthesised optical sound, circulated as a some-
what sensationalised story in the international press of the time and documented
by Thomas Y. Levin in his writings on Rudolf Pfenninger, involved British
sound engineer E. A. Humphries replacing a short segment of problematic dia-
logue with new words by drawing them by hand on the optical soundtrack, this
drawing based on extensive analysis of vocal recordings. There is some ambigu-
ity around this story, even as to what the words concerned were (one version has
it as ‘all of a tremble’, which would have a certain poetry), but if it is true, it is
certainly remarkable as an example of speech synthesis from 1931.
This process of analysis by eye and synthesis through inscription had an early
proponent in the artist László Moholy-Nagy. Maholy-Nagy had proposed an
analogous process with gramophone records:

A scientific examination of the tiny inscriptions in the grooves of the pho-

nograph in order to learn exactly what graphic forms corresponded to
which acoustic phenomena. Through magnification . . . one could dis-
cover the general formal logic that governed the relation of the acoustic
to the graphematic, master it, and then be able to produce marks that,
once reduced to the appropriate size and inscribed onto the record surface,
would literally be acoustic writing.25

Since this idea was perhaps somewhat impractical (although it was explored,
certainly in a theoretical sense, by contemporary composers such as Paul Hin-
demith and Georges Antheil), Moholy-Nagy rapidly transferred his concerns
to the new optical film technology, challenging creative minds to put this
technology to similar use and rapidly finding this challenge met by filmmaker
Rudolf Pfenninger. Maholy-Nagy wrote: “today, thanks to the excellent
work of Rudolf Pfenninger, these ideas have been successfully applied to the
medium of sound film. In Pfenninger’s sound-script, the theoretical prerequi-
sites and the practical processes achieved perfection.” 26 Pfenninger called his
optical sound technique tönende handschrift (sounding handwriting) and used it
to produce soundtracks for his own animations, starting with Pitsch und Patsch
in 1930. His technique was remarkably fully f ledged but incredibly laborious
(and markedly similar to that used by Norman McLaren decades later). Levin
describes it thus:

He sat down with an oscilloscope and studied the visual patterns produced
by specific sounds until he was able . . . to isolate a unique graphic signa-
ture for each tone. Using the newly available optical film soundtrack to test
 Cross-Modal Theories of Sound and Image 149

his experimental results, he would painstakingly draw the desired curve

onto a strip of paper which he then photographed in order to integrate it
into the optical sound track.27

Better known at the time were Oskar Fischinger’s Ornament Sound Experiments of
1932 (described in Chapter 2), which captured the public interest through exten-
sive popular press coverage. These perhaps represent an evolution with respect to
Pfenninger’s work, not necessarily in terms of the work itself or its execution, but
more in terms of the conceptual framework behind it. Fischinger did not under-
take careful analysis of existing sounds in the manner of Pfenninger. Rather, he
produced designs, or ‘ornaments,’ that were entirely fanciful and abstract. Fis-
chinger was certainly not without musical or acoustic knowledge – early in his
career he trained as an organ builder. There is little attempt to emulate ‘real’
acoustic sound waves in his ornaments, but it seems likely this was intentional. In
a well-known picture taken in his Berlin studio in 1932, Fischinger is pictured
holding long rolls covered in triangular forms. In fact, this picture was staged
purely for publicity purposes and supposedly to obfuscate his techniques from any
competitors. Whilst similar triangular forms (or waves) do play a part in the actual
film sequences, there are many forms, such as (typically Fischinger-esque) con-
centric circles, which bear little resemblance to an amplitude/time waveform (see
Chapter 2). It is still interesting to hear what these ornaments sound like today.
The experiments are a remarkably bold exercise in synthesis. Notably, there is no
attempt to explore the traditional musical framework of melody, harmony and
rhythm – rather, these seem entirely to be experiments in pattern and timbre.
Also of interest are some of the broader ideas and beliefs that Fischinger mani-
fests in this work, which to some extent can be related to those of Kandinsky
and earlier proponents of visual music. Fischinger was interested in theosophy
and Buddhism, and he seems to share with some of his predecessors the idea that
sound is an inherently spiritual medium. These ideas can be argued to have taken
on a life that transcends Fischinger’s own work through a little-known associa-
tion with John Cage. In 1937, 25-year old Cage spent a few days with Fischinger
on the set of his MGM film An Optical Poem. As William Moritz writes:

During the long pauses while each new setup was being arranged, Oskar
told John about his Ornament Ton experiments, and his Buddhist-inspired
belief that all things have a sound, even if we do not always listen or hear
it, just as a stone has an inherent movement even if it is still. Cage credited
Oskar with offering him the revelation that changed his whole perception
of music and sound.28

Some of the most interesting developments in optical sound in the late 1920s to
1930s happened in the Soviet Union. Much of this history was lost for decades,
but has been recently rediscovered through the remarkable scholarship of Andrey
Smirnov, resulting in the Sound in Z exhibition at the Palais de Tokyo in Paris
150 Joseph Hyde

in 2008 (curated by Jeremy Deller and Matt Price) and the publication of a book
of the same name in 2013. Sound in Z documents the history of “Experiments in
Sound and Electronic Music in early 20th Century Russia”29 and includes details
of many extraordinary optical sound techniques, instruments and artworks.
One of the most remarkable is the Variophone, developed by Evgeny Sholpo
in the 1930s and 1940s. This instrument improves on early optical sound experi-
ments in almost every way and represents nothing less than a fully-f ledged (up
to 12 voice) polyphonic synthesizer. It made the revolutionary step of produc-
ing tones by means of rotating disks (Figure 6.8) – since each disk represented a
different tone, it meant that timbre could be separated from pitch and simulta-
neously sidestepped a problem inherent with most optical soundtrack synthesis
methods, which are marred by the constant ‘hum’ produced as a factor of the
actual frame rate of the film.
Sholpo’s instrument was incredibly expressive, allowing for glissandi, vibrato
and rubato. Recordings of classical music using the Variophone such as Wagner’s
Flight of the Valkyries and Liszt’s 6th Rhapsody foretell the work of Wendy Carlos
and Isao Tomita decades later. The idea of synthesis based on optical technologies
and spinning discs was revisited in various synthesizers and even prototypical
samplers (such as the Optigan and the Orchetron) in the 1960s and 1970s. The

FIGURE 6.8 Disks from Evgeny Sholpo’s Variophone. Courtesy of Andrey Smirnov.
Used by permission.
Source: www.researchgate.net/figure/Variophone-disks-with-cut-wave-shapes-Version-1-1932-
Courtesy-of-Andrey-Smirnov_fig7_326414520
 Cross-Modal Theories of Sound and Image 151

ANS synthesizer (the name was derived from the initials of synesthetic composer
Alexander Nikolayevich Scriabin), developed by Boris Yankovsky and Evgeny
Murzin, was a direct descendant of the Variophone. It was first conceived of in
1939, but the first version of the instrument was not completed until 1957. Again,
this was a remarkably revelatory instrument, built on a spectral model of com-
position and a complex implementation of additive synthesis. In contrast to the
Variophone, discs (optically) encoded hundreds of individual waveforms. Several
such discs were employed, allowing the simultaneous sounding (in real time!) of
576 (version 1) or 720 (version 2) sine tones covering the full frequency range. Of
particular note is the interface, which essentially allows the user to draw a sono-
gram (a plot of frequency spectra vs. time) that will be reproduced using the sine
tone bank – a model almost identical to that employed by Xenakis’s UPIC system
two decades later. The ANS was used by composers such as Alfred Schnittke and
Sofia Gubajdulina, but its very distinctive sound is perhaps most associated with
soundtracks to the early films of Andrei Tarkovsky.

Space
If one extrapolates further from the ideas outlined earlier around vibration in
both audio and visual domains, one might consider theoretical frameworks based
on motion and its constituent parts, time and space. In considering the latter, it
can be seen that the representation of sound and music using two-dimensional
space is not new. There is a centuries-old tradition of doing just that in the form
of the musical score. In many such scores the relationship between sound and
space is imprecise – there is a relationship between the horizontal dimension and
time and between the vertical dimension and pitch, but both are usually not pro-
portionately consistent and can be seen as symbolic as much as representational.
In the 20th century, a number of composers, early examples being Earle Brown
and John Cage, started to explore notation systems in which the balance shifted
somewhat towards the latter, and two-dimensional space is used in a proportional
way to represent musical parameters. This shift is often accompanied by an exten-
sion in the range of visual representation, associated with the term ‘graphic score.’
Proportionality usually centres around time, in various systems usually known
as ‘proportional notation.’ The driving force in the evolution of proportional
notation was recorded media, where there is often a direct relationship between
physical dimensions and time. This might be closely associated with magnetic
audio tape, where there is a simple relationship between the length of a piece of
tape and the duration of time it represents. However, since early graphic and/or
proportional notations pre-date the introduction of tape, it seems worthwhile to
look for the origins of this idea elsewhere, perhaps in visual arts or media.
There is a short but interesting period in visual art history where the desire
to incorporate time into a visual and spatial medium pre-dated it becoming
commonplace reality in the form of film. This can be seen in many of the artists
connected with the Bauhaus movement. Several artists did manage to make early
152 Joseph Hyde

time-based works in the form of kinetic art, a prime example being Maholy-
Nagy. Paul Klee is an artist whose role in the history of visual music is at least as
important as that of Kandinsky, as discussed in Chapter 2. Although Kandinsky
was equally concerned with colour and form, the latter forming the core of his
later work Point and Line to Plane (which was a Bauhaus publication), Klee made
form central to his theoretical framework, in particular his well-known ideas
around line. These ideas, expressed succinctly in The Pedagogical Sketchbook, have
been tremendously inf luential on several generations of composers, most notably
Boulez and Birtwistle.
Of particular note here are a series of works produced by Hans Richter and
Viking Eggeling. Like Klee, Richter and Eggling were interested in exploring
musical forms and ideas such as counterpoint. Richter wrote: “The principle of
counterpoint is not limited to music. For us, it was more than a technical device;
it was a philosophic way of dealing with the experience of growth.”30 Eggeling
found a guiding principle in Generalbass der Malerei (Figured Bass in Painting).
Over a short period from 1919 to 1921 they produced (separately and together)
a series of long ‘scroll paintings,’ well-known examples being Eggeling’s Hori-
zontal-Vertikal-Messe I-IIII (1919–1921) (Figure 6.9) and Richter’s Fugue (1920).
These paintings function as abstract graphic scores, and through evolution of
forms when read horizontally or vertically, have a clear implication of move-
ment. One interesting aspect of this implicit movement is that, as Richter notes,
it requires active participation on the part of the viewer.31 It is interesting to note
that although both artists went on to make films exploring similar ideas, this was
not (as might appear) their original intention. Richter writes:

We had gotten more than we asked for: the necessity to release this accu-
mulated “energy” into actual movement! Never during our collaboration
had we dreamt of that. But there it was. And movement implied film!

FIGURE 6.9 Viking Eggeling’s Hauptpartie des Horizontal-vertikalorchesters (1921). Public

Domain.
Source: https://commons.wikimedia.org/wiki/File:Viking_Eggeling_Hauptpartie_des_Horizontal-
vertikalorchesters.jpg
 Cross-Modal Theories of Sound and Image 153

Few people have ever come to this medium so unexpectedly and with
so much inner resistance. We knew no more about cameras and film than
what we had seen in shop windows.32

There was also no direct process of translation from Richter and Eggeling’s scroll
paintings to their early films, even where the films bear the same title. Rather,
the films were made from scratch in a process that both artists found difficult.
However, the lineage of the scroll paintings can perhaps be seen in many more
recent ‘hand-painted’ films such as those of Norman McLaren and Len Lye (see
Chapter 2 for an-depth discussion of McLaren and Lye). Another proponent of
this way of working is Stan Brakhage, who studied with Richter many years later
in the United States.
Another intriguing antecedent to the time-based graphical score can be found
in the work of Oskar Fischinger. Although many of his films are hand-painted
(or drawn), this was achieved on a frame-by-frame basis. However, in many
(possibly all) cases, he produced proportional graphic scores of the music he was
animating to, which also included indications regarding forms and movement
in the animation. Since the first such scores that survive date from the early
1930s, they actually represent very early examples of proportional notation. It is
arguable that these scores had a direct inf luence on 20th century music through
John Cage. This might seem somewhat fanciful given the short duration of their
association. However, in his paper on Fischinger and Cage, Richard H. Brown
makes a convincing case. He compares the graphic score of Franz Liszt’s Hungar-
ian Rhapsody produced as part of the process of making An Optical Poem (which
Cage worked with Fischinger on in 1937) with that of Cage’s Quartet for Percus-
sion, finding remarkable similarities between the two, both in terms of superfi-
cial appearance and deeper function. Although the official date of Cage’s quartet
is 1935, Brown also makes a strong argument that this date is incorrect and Cage
actually wrote the piece in 1937.33
A reasonably complex model for a spatial model of audio-visual relationships
was developed by composer and musicologist Joseph Schillinger. Schillinger’s
ideas, first outlined in The Schillinger Model of Musical Composition 34 in 1941, were
very inf luential at the time, particularly in terms of informing implementations
of parametric control seen in integral serialism and early electronic music. In a
later work, The Mathematical Basis of the Arts (1948), he postulates a parallel tax-
onomy for music and ‘visual kinetic composition.’ In connection with this, he
proposes novel relationships between time and space:

Schillinger attempts to relate cinematic time and space proportionally,

such that, for example, one incremental move on the screen (which is
divided into a grid 24 x 24) might be equivalent to one second tem-
porally (sound film being projected at the rate of twenty-four frames
per second). In this way, the two ‘rhythms’, spatial and temporal, are
coordinated. 35
154 Joseph Hyde

Animator Mary Ellen Bute is strongly associated with Schillinger and his ideas.
She produced graphics for Shillinger and Lewis Jacobs’ film Synchronisation (never
completed) and went on to make systematic use of Schillinger’s ideas (prior to
their publication in the previously mentioned texts) in her own abstract films
over the 1930s and 1940s. Bute’s work makes some interesting connections in
the worlds of music and film making. Early on she worked with Leon Theremin,
although their collaboration was cut short by his departure from the United
States. Later she worked with Norman McLaren, and she also worked with engi-
neer Ralph K. Potter of Bell Labs, who built her a specialised oscilloscope used
in her Abstronics films from 1934 onwards. These films were very much ahead
of their time and greatly inf luenced the many excursions into ‘oscillographics’
that followed.
John Whitney, computer (motion) graphics pioneer, already introduced in
Chapter 3, took ideas around motion and music considerably further. These ideas
are detailed in some depth in his book Digital Harmony: On the Complementarity of
Music and Visual Art,36 but can be seen to be employed in his work decades ear-
lier. The five Film Exercises, produced with his brother James between 1943 and
1945, offer an extraordinary prescient vision of tightly controlled audio-visual
parameterisation. The films are built on a model inf luenced heavily by serialism
and a series of ‘permutations’ such as transposition, inversion and retrogression,
which are applied both to the sound material and the forms that make up the
visual component of the works. Since these processes are applied to all aspects
of (entirely synthetic) sound and abstract visuals, these pieces are astonishing
precursors of total or integral serialism. The soundtracks are some of the most
advanced examples of early synthesis, although they were produced by optical
rather than electronic means. The Whitney brothers devised an instrument that
would draw synthetic waveforms onto film by means of a series of pendulums.
The sonic result seems very much to prefigure the early electronic music of
Karlheinz Stockhausen, not just sonically but also in terms of ideas. Particularly
remarkable is the fact that the Whitneys realised they had at their disposal a
continuum of pitch that could incorporate infinite microtonal gradations right
down to – and beyond – the point that frequency became rhythm, again prefig-
uring Stockhausen.
Later in his career, Whitney perhaps reveals himself to have slightly more
conservative ideas about music, certainly in comparison to the ultra-modernist
aesthetic at play in the Film Exercises. The theories he developed, however, are
no less remarkable. He developed a series of complex principles for ‘harmonic
motion’ which could only be realised with his ‘analog computer’ (in reality, a
bespoke mechanical device built with an intricate system of cams designed for
anti-aircraft gun direction, as discussed in Chapter 3) he built from military
surplus in the late 1950s and then later using digital computers. Using these
tools, he built upon the idea of permutation to develop a system of ‘differential
motion.’ The systems he used allowed the precise control of hundreds of graphi-
cal elements and would tend to apply the same process to these elements but
 Cross-Modal Theories of Sound and Image 155

by different (or differential) degrees. This principle, which lies at the heart of
many subsequent computer animation techniques, gave him something he saw as
very close to the functioning of musical harmony in a metaphorical rather than
a literal sense. In particular, he found tension and release in the way graphical
elements could be made to diverge and converge into particular configurations
using this technique. This principle can be seen and indeed (as Whitney would
have it) felt in most of his well-known films, perhaps the prime example being
Arabesque.
While Whitney progressed from analogue to digital techniques (which will
be further discussed in the last section of this chapter), many artists have sub-
sequently gone the other way and found in analogue technology an attractively
direct way to integrate sound and image. This can be seen in particular in the
continued use of oscilloscopes and related technologies in creative audio-visual
work. This distinct area of practice is a direct descendant of the theories of Lis-
sajous and others and the implementations of Mary Ellen Bute. In the second half
of the 20th century it was often known as ‘oscillographics.’ In the 21st century,
Derek Holzer has coined the term vector synthesis. The vibrancy of this practice
can be seen in the body of work presented across two Vector Hack events, pro-
duced by Holzer and others in 2018 and 2020. Although this work is very diverse
in terms of aesthetic and technique, most of it has one thing in common. Gener-
ally, audio signals are directly used to drive a beam of electrons or light. Some
works will abstract the relation between audio and visual, but in many cases the
soundtrack is exactly the audio that produces the image.
Although this might at first appear to be solely an exercise in ‘media archaeol-
ogy,’ it is very much informed by contemporary developments. Whilst the means
of display is usually analogue, the range of such displays has been extended. As
well as various kinds of oscilloscopes, proponents of this way of working also
use hacked cathode ray tube (CRT) TVs and game consoles and laser projectors
(there is nothing new about using lasers in this way, but they have become much
more affordable, and therefore widespread, in recent years). At the same time,
the range of signals used to drive the display has also expanded. This expan-
sion has taken place over the analogue domain, exploiting the broad range of
circuits available in systems based on Eurorack and other modular formats. It
is also embraced the digital, with bespoke software and modular programming
environments such as Pure Data and Max/MSP making a huge variety of new
forms available. The author of this chapter has been exploring Lissajous patterns
based on polar rather than cartesian coordinates.37 This is difficult to achieve
using analogue circuits exclusively (the code ‘instrument’ outlined at the core of
Whitney’s Digital Harmony is based on a similar model, but not designed to run
in real time or to produce audio). Other artists, such as Jerobeam Fenderson and
Hansi Raber, use software to derive complex three-dimensional forms.
‘Video synthesis’ is a term with rather imprecise meaning, which is used in vari-
ous ways. However, in its ‘classic’ form, as employed in systems such as the Sandin
Image Processor or the Hearn Videolab and using analogue circuits and CRT
156 Joseph Hyde

displays, it can be seen as a specialised case of vector synthesis. In this case, def lec-
tion of the electron beam is not arbitrary, but is based on the ‘given model’ of a
raster scan. The raster scan relies on the electron beam scanning across the CRT at
50 Hz (PAL; 60 Hz for NTSC systems). This means that frequencies below 50 Hz
will produce colour/luminance fields, frequencies between 50 Hz and around
25/30 kHz will produce vertical bands/lines (the scan frequency multiplied by the
number of pixels in each line) and frequencies above this will produce horizontal
bands/lines (with the frequency further multiplied by the number of lines). By
deriving signals with awareness of this model and across colour channels, a huge
variety of visual forms can be produced using audio signals. There are some issues
to overcome – of course, a significant proportion of the frequency range that might
be used lies beyond the range of human hearing, and the operational voltage range
of audio and video systems is generally not the same. However, it is still relatively
easy to establish a direct relationship between sound and image in this way. This is
ref lected by the development of specialist analogue video modules in the Eurorack
format by companies such as LZX Systems.
A number of artists have explored a middle ground between vector and raster
systems by essentially deriving a raster system ‘by hand’ using vector techniques.
This opens up all sorts of possibilities for unusual scanning patterns and pseudo-
3D distortions of video images. This is possible using analogue technology, for
example, in the Rutt/Etra system built in 1974, but it has become easier and far
more f lexible with the integration of digital technology.
One should note that it is also possible to derive audio from video signals.
This can be achieved very simply using the ‘composite cramming’ technique
described by Nic Collins.38 As with raster manipulation, the results are very
much limited by the raster scan model. The sounds produced will always have
as their primary feature the frequency of the raster scan (50 Hz for PAL or 60
Hz for NTSC) and will contain frequency content outside of the range both of
human hearing and of most audio equipment. However, despite these limitations
this method can be, and has been, put to worthwhile use.

Time
Continuing our examination of motion as common ground in models of audio-
visual interaction, we come to the other component of motion – time. While
spatial information may require a certain amount of ‘translation’ between sound
and image as outlined earlier, this is not the case with the temporal. There are
physiological and psychological differences to the way we perceive time in con-
nection to visual and aural phenomena, which are beyond the scope of this dis-
cussion. Nonetheless, in a straightforward physical sense, time can be seen as
operating in the same way across both domains. This has been recognised in
many of the theoretical frameworks outlined earlier: Schillinger’s taxonomy
includes time as the “general component” in the elements of both “visual kinetic
composition” and music.39
 Cross-Modal Theories of Sound and Image 157

Various theories have evolved around the use of time in both media. In film,
Eisenstein’s theories of montage are well-known and discussed elsewhere in this
volume. In sound, theoretical frameworks are more developed (perhaps because
time is intrinsic to sound – i.e. sound cannot exist without it). Theories of musi-
cal time can be found dating back centuries, and indeed several of these have
been applied to the visual arts. However, in the advent of audio-visual media,
time has become absolute across both domains (both exist on the same timeline).
For this reason, this discussion will be limited to the media age, and we will
begin it with Pierre Schaeffer.
Although Schaeffer is primarily associated with sound, and though some
of his ideas (for example, around the acousmatic) might at first sight seem to be
antithetical to multimedia practice, he was in fact very much interested in the
audio-visual (what he called ‘arts-relais’) and applied some of his ideas to this
domain:

The arts-relais contribute images and sounds which would be as form-

less as the world itself if we did not strive to make them mean something
and relate our ideas to them. Encountering the concrete starting from the
abstract, this is the great invention of language; encountering thought
starting from things, this is the invention of radio and cinema.40

Schaeffer also took some of the inspiration for his ideas from the visual domain.
He was strongly inf luenced by the writings of filmmaker Jean Epstein. He also
based many of his theories on the early phenomenology of Edmund Husserl.
Although phenomenology is, of course, not exclusive to visual phenomena, in
Husserl’s writing it is usually couched in visual terms.
Perhaps because of the origin of these ideas, it is relatively simple to apply
them to the audio-visual domain, and several theoretical frameworks have been
built in this way. Diego Garro has written a number of texts proposing a music
concrète–based model for audio-visual practice (based as much on the theories
of more recent electroacoustic theoreticians such as Denis Smalley as those of
Schaeffer). This is predicated on the idea of the objet audiovisuelle.41 Garro postu-
lates that this can be seen as the basic building block in audio-visual composi-
tion, in the same way as the objet sonore is in musique concrète (or electroacoustic
or acousmatic music). He also proposes that the ways in which the objet may be
transformed are comparable across both media. Garro’s theories are primarily
built on the assumption of concrete materials (i.e. those derived from the real
world). This puts them in contrast with theories around visual music, most of
which are tied up with abstraction (Brian Evans’s definition of visual music
explicitly equates it with abstract animation42).
Schaeffer’s theories were brought to bear on the world of cinema by Michel
Chion in a series of texts – most famously Audio-Vision. Chion’s theories are quite
specific to the world of cinematic sound design and are often tied up with nar-
rative concerns. However, some of his ideas concerning time are more relevant
158 Joseph Hyde

to the world of audio-visual practice. In particular, his idea of synchresis can be

seen as a guiding principle in bringing sound and image together. Chion defines
synchresis as follows:

Synchresis (a word I have forged by combining synchronism and synthe-

sis) is the spontaneous and irresistible weld produced between a particular
auditory phenomenon and visual phenomenon when they occur at the
same time. This join results independently of any rational logic. . . . For a
single body and a single face on the screen, thanks to synchresis, there are
dozens of allowable voices – just as, for a shot of a hammer, any one of a
hundred sounds will do.43

In the context of abstract audio-visual work, this might seem dangerously close to
‘mickey mousing’ (a term used to denote overly simplistic audio-visual relation-
ships). However, the power of such audio-visual fusion cannot be denied. Recent
writing by Quebecois audio-visual artists Jean Piché and Myriam Boucher has
expanded the definition of synchresis, particularly in connection with what they
term vidéomusic. They expand the definition of synchresis thus:

synchresis happens when an auditory event and a visual event:

• occur at the same time;
• occur in a short time interval so that they are perceived as belonging
to the same object;
• are directly associated in a physically recognisable object;
• progress morphologically at the same perceptual speed with coherence
shared directional dynamics;
• or unfold at a predictable periodicity or as a single or repeated pattern.44

They identify ten different categories of synchresis, which can operate over a
range of timescales from a single instant (as is generally the case with Chion’s
original model) to extended periods of time. This rather precise taxonomy for
a specific category of audio-visual relationships is reminiscent of frameworks
developed for sound by the likes of Schaeffer, Smalley and Roads, and is a prom-
ising direction for future development.

Data
Since the 1980s, the rise of personal computing and other domestic digital tech-
nologies has seen significant changes in our relationships with media, and indeed
between those media themselves. We might see this as part of a broader process
of ‘media convergence.’ This term is notoriously slippery to define and has been
through a number of shifts in meaning over the last four decades. It has been
applied to media, technology (and, of course, both of these together), economics
 Cross-Modal Theories of Sound and Image 159

and – increasingly – culture, with the latter perspective being popularised by

Henry Jenkins in his 2006 book Convergence Culture.45 For the purposes of this
discussion we will consider an early definition, from Ithiel de Sola Pool’s Tech-
nologies of Freedom, published in 1983. In it, de Sola Pool describes media conver-
gence as “blurring the lines between media. . . . A single physical means – be it
wires, cables or airwaves – may carry services that in the past were provided in
separated ways.”46
Even this early definition is broader in scope than this discussion requires, but
what we might take from it is the idea that in the digital era, sound and image
use a ‘single physical means.’ Where de Sola Pool talks of cables or airwaves, we
might add computers or networks. Across all of these, the core ‘means’ is identi-
cal: the ones and zeroes of binary data.
At first sight, it might seem that this convergence of media means that many
of the issues and arguments outlined previously have been rendered irrelevant.
One does not need to find correspondences between audio and video if they are
essentially the same thing. In a PC or laptop, or even a tablet or phone, we have a
device in which both media can be authored, stored, distributed and consumed,
and these processes are often similar or identical. This has wide-ranging impli-
cations for audio-visual practice – for the first time, widely available tools allow
audio and video material to be manipulated in similar ways, sometimes in the
same application. This can be seen as part of a more general process of democ-
ratisation, where such tools are becoming more accessible, both by becoming
cheaper and easier to use.
Whilst a computer or application might treat a stream of data the same way
whether it represents sound or video, for it to be humanly comprehensible as
either, it will need to be converted, and the process of conversion will be entirely
different in each case. The relationship between the data and what is seen or
heard is almost entirely opaque: what Kim Cascone called the “impenetrable veil
of digital media.”47 Moholy-Nagy’s idea of learning how to read and interpret
sound and music from its physical appearance in the grooves of a record was
perhaps fanciful, but certainly no one would imagine that this would be pos-
sible with a CD, and the optical disc represents a relatively early digital storage
medium – in more recently introduced formats we have become used to a para-
digm where the physical ‘containers’ for our audio and/or video have become
entirely ‘black boxed’ (physical formats and local computing) or indeed invisible
or virtual (the internet or cloud).
This does not mean direct manipulation of this data has become impossible,
but it has become a radically different proposition. This can be sited in the realm
of the post-digital or glitch, with a new set of aesthetics and techniques around it.
These are usually concerned with misusing, hacking or breaking digital technol-
ogies. As Caleb Kelly has argued, such techniques are not new and can be placed
in a continuum of “cracked media”48 predating the digital. However, with digital
media, they have become a matter of necessity. Where earlier modes of audio-
visual encoding were often ‘human comprehensible’ (a good example being the
160 Joseph Hyde

celluloid optical soundtrack), digital media in its raw binary form is not. A pur-
poseful intervention akin to Pfenninger’s tönende handschrift is not possible, and
any intervention is likely to produce unpredictable results. An edit made to the
binary data of an audio and/or visual stream might have no perceptible effect, or
it might result in an error and cause playback to cease. On the other hand, more
interestingly it may produce a glitch – an artefact which tends to have certain
characteristics, whilst its precise form will be hard to predict (a useful term to
apply here might be Greg Hainge’s “meta noise”).49 It will often offer an extreme
contrast to the surrounding material, whether that is in terms of dynamics, pitch,
colour or form. In some cases, it may alter the behaviour of playback of this
material – an example of this would be the practice of ‘data moshing,’ where data
discontinuities are used to interfere with the workings of video data compression
algorithms to typically psychedelic effect.
Therefore we can see that while the ‘wounded’ CDs of Yasanao Tone, for
example, may seem like a contemporary response to the challenge proposed by
Moholy-Nagy, the outcome of Tone’s CD abuse (largely consisting of pieces
of perforated tape being applied to the underside of the disks) is not at all the
empirical process originally called for, but something much more unpredict-
able and chaotic, very much in keeping with Tone’s Fluxus ethos. In the liner
notes for Solo for Wounded CD (1997), Tone describes this as “a maze where
ambush was everywhere, and that made the performance situation all the more
interesting.”50
Tone explores a similar ‘maze’ in a series of works based on traditional Chi-
nese poetry, such as Musica Iconologos (1993). In this work, images of Chinese
calligraphy were scanned and the images converted to sound files. Perhaps
unsurprisingly, the result is similarly chaotic and noisy, highlighting again a
process which is essentially not comprehensible by humans. Many other art-
ists have explored similar territory across audio and visual domains. Perhaps
the best known is Ryoji Ikeda, who has evolved an aesthetic which highlights
the extremes of digital media across the audio and visual, exploring boundary
conditions of loud/quiet, bright/dark, highest/lowest, 1 and 0. His work is often
predicated on a particular kind of post-digital sensory overload, with an aim to
evoke the sublime – as he has described it:

If it’s beautiful, you can handle it; the sublime, you cannot. If you stand
in some great whited-out landscape in Lapland, the Sahara or the Alps,
you feel something like fear. You’re trying to draw information from the
world, but it’s something that your brain cannot handle.51

Often, Ikeda achieves this effect through cross-modal ‘translations’ akin to

Tone’s, an example being Test Pattern, which is based around barcodes being read
as audio and vice versa. This represents a specific instance of a broader concern
in Ikeda’s work, where data is explored for its very “data-ness.”52 This is some-
what at odds with data visualisation or sonification: there the aim is often to aid
 Cross-Modal Theories of Sound and Image 161

understanding of data, where Ikeda’s aim seems to be to accept, even accentuate,

its incomprehensibility.
Via data, digital technologies remove the necessity from any audio-visual
relationships and allow such relationships to be arrived at in an arbitrary fashion
through a process of mapping, which has become a major part of audio-visual
practice. Many kinds of mapping can be found between audio and visual data
and also with other kinds of data. A special case arises in the shape of data
derived from human movement – which has opened up vast areas of territory in
human-computer interaction (HCI) and New Instruments for Musical Expres-
sion (NIME). This territory is beyond the scope of this chapter but is discussed
at length in the rest of Part II of this volume.
Other types of data used in a digital environment do not need to have any
existence outside of the system. Data structures can be built entirely for the
purposes of controlling events in the form of automation. This is a powerful
paradigm for audio-visual work, because it means that temporal (or other) struc-
tures can be built that are specific neither to the audio nor the visual but can be
manifested through both. Furthermore, whilst the data involved in an audio-
visual performance, installation, instrument or composition will be made up of
binary ones and zeros, so will the code that forms a major part of the means of
production. This opens up many new possibilities for artistic practice. Code can
be written by audio-visual artists themselves, and many tools have been devel-
oped to facilitate this, from graphical programming environments such as Pure
Data and Max/MSP, to languages developed to facilitate live coding, where an
instrument itself can be built as part of a performance. This paradigm is con-
stantly shifting as developments in areas such as machine learning and artificial
intelligence are included, further blurring the boundaries between sound and
image, the tools used to make them and the creators using those tools. All of
this is discussed over the next few chapters, but we end this one with another
quote from Cascone: “The medium is no longer the message; rather . . . tools
themselves have become the message.”53 This leads us directly to the next chap-
ter, which revolves around a discussion of audio-visuals, Live Visuals and media
art in relation to both present-day theories of technology and culture and the
practices of post-conceptualism and contemporary art.

Notes
1 David Kershaw, Tape Music with Absolute Animated Film: Prehistory and Development (PhD
Thesis, University of York, 1982).
2 John Whitney, Digital Harmony: On the Complementarity of Music and Visual Art (King-
sport: Kingsport Press, 1980), 41.
3 Fred Collopy, “Playing (With) Colour,” Glimpse: The Art and Science of Seeing, 2(3),
2009, https://rhythmiclight.com/1998/articles/Playing(With)Color.pdf.
4 Isaac Newton, Opticks: Or, a Treatise of the Reflections, Refractions, Inflexions and Colours of
Light (London: Sam. Smith, and Benj. Walford, 1704).
5 Isaac Newton, Opticks: Or, a Treatise of the Reflections, Refractions, Inflexions and Colours of
Light (London: Sam. Smith, and Benj. Walford, 1704), 155.
162 Joseph Hyde

6 Isaac Newton, Opticks: Or, a Treatise of the Reflections, Refractions, Inflexions and Colours of
Light (London: Sam. Smith, and Benj. Walford, 1704), 154.
7 Adrian Bernard Klein, Colour Music, the Art of Light (London: Lockwood, 1930).
8 Wassily Kandinsky, Concerning the Spiritual in Art (New York: Dover, 1977), 41.
9 Gene Youngblood, Expanded Cinema, 50th Anniversary Edition (New York: Fordham
University Press, 2020), 81.
10 Karlheinz Stockhausen, Towards a Cosmic Music, translated by T. Nevill (Shaftesbury:
Element Books, 1989).
11 Kenneth Lindsay and Peter Vergo (eds.), Kandinsky: Complete Writings on Art (Cam-
bridge: Da Capo Press, 1994).
12 Malcolm MacDonald, Schoenberg (Oxford: Oxford University Press, 2008), 9.
13 Malcolm MacDonald, Schoenberg (Oxford: Oxford University Press, 2008), 9.
14 Hermann L.F. Helmholz, On the Sensations of Tone as a Psychological Basis for the Theory of
Music, translated by Alexander J. Ellis, 3rd edition (Cambridge: Cambridge University
Press, 2009).
15 Leon Vallas, Claude Debussy His Life and Works (London: Dover Publications, 1978).
16 Arnold Schoenberg, Theory of Harmony, translated by Roy E. Carter (Berkley: Belmont
Music Publishers, 1978).
17 Joseph Edward Harris, Musique Colorée: Synesthetic Correspondence in the Works of Olivier
Messiaen (PhD thesis, University of Iowa, 2004), 49.
18 Jonathan W. Bernard, “Messaien’s Synaesthesia: The Correspondence between Colour
and Sound Structure in His Music,” in Music Perception: An Interdisciplinary Journal, vol. 4,
no. 1 (Berkeley: University of California Press, 1986), 44.
19 Ernst Chladni, Entdeckungen über die Theorie des Klanges (Leipzig: Weidmanns Erben und
Reich, 1787).
20 Hans Jenny, Cymatics: The Structure and Dynamics of Waves and Vibrations (Basel: Basilius,
1967).
21 Joost Rekveld, Symmetry and Harmonics, 1998, www.joostrekveld.net/?p=252.
22 Robert V. Bruce, Bell: Alexander Graham Bell and the Conquest of Solitude (Boston: Little
Brown & Co., 1973).
23 Margaret Watts Hughes, “Visible Sound,” in The Century Illustrated, vol. 41, no. 1 (New
York: Scribner, 1891).
24 Margaret Watts Hughes, The Eidophone Voice Figures: Geometrical and Natural Forms Pro-
duced by Vibrations of the Human Voice (London: Christian Herald Co. Ltd, 1904), 2.
25 Thomas Y. Levin, “Tones from Out of Nowhere: Rudolph Pfenninger and the Archae-
ology of Synthetic Sound,” in Grey Room, vol. 12 (Cambridge, MA: MIT Press, 2003).
26 Kristina Passuth, Moholy-Nagy (Dresden: Verlag der Kunst, 1982), 322.
27 Thomas Y. Levin, “Tones from Out of Nowhere: Rudolph Pfenninger and the Archae-
ology of Synthetic Sound,” in Grey Room, vol. 12 (Cambridge, MA: MIT Press, 2003).
28 William Moritz, Optical Poetry: The Life and Work of Oskar Fischinger (Eastleigh: John
Libby Publishing, 2004), 78.
29 Andrey Smirnov, Sound in Z: Experiments in Sound and Electronic Music in Early 20th
Century Russia (London: Sound and Music / Koenig Books, 2013).
30 Hans Richter, “Easel – Scroll – Film,” in Magazine of Art, vol. 45 (New York: American
Federation of Arts, 1952), 78–86.
31 Hans Richter, “Easel – Scroll – Film,” in Magazine of Art, vol. 45 (New York: American
Federation of Arts, 1952), 78–86.
32 Hans Richter, “Easel – Scroll – Film,” in Magazine of Art, vol. 45 (New York: American
Federation of Arts, 1952), 78–86.
33 Richard H. Brown, “The Spirit Inside Each Object: John Cage, Oskar Fischinger, and
‘the Future of Music’,” Journal of the Society for American Music, 6, 2012, 83–113.
34 Joseph Schillinger, The Schillinger Model of Musical Composition: New Edition (Cambridge:
Da Capo Press, 1977).
35 Joseph Schillinger, The Mathematical Basis of the Arts (New York: Philosophical Library,
1948).
 Cross-Modal Theories of Sound and Image 163

36 John Whitney, Digital Harmony: On the Complementarity of Music and Visual Art (King-
sport: Kingsport Press, 1980).
37 Joseph Hyde, “Hybrid Analogue/Digital Audiovisual Performance,” in Jason Berna-
gozzi, ed., The Signal Culture Cookbook, vol. 2 (Owego: Signal Culture, 2014).
38 Nicolas Collins, Handmade Electronic Music: The Art of Hardware Hacking, 3rd edition
(London: Routledge, 2020).
39 David Kershaw, Tape Music with Absolute Animated Film: Prehistory and Development (PhD
thesis, University of York, 1982), 205.
40 Sophie Brunet (ed.), Pierre Schaeffer: De la musique concrete a la musique même (Paris: La
Revue Musicale, 1938–1977).
41 Diego Garro, “A Glow on Pythagoras’ Curtain: A Composer’s Perspective on Electro-
acoustic Music with Video,” in EMS International Conference Series (Montréal, 2005).
42 Brian Evans, “Foundations of a Visual Music,” in Computer Music Journal, vol. 29, no. 4,
Visual Music (Cambridge, MA: MIT Press, 2005), 11–24.
43 Michel Chion, Audio-Vision: Sound on Screen, translated by Claudia Gorbman (New
York: Columbia University Press, 1990).
44 Myriam Boucher and Jean Piché, “Sound/Image Relations in Videomusic: A Typologi-
cal Proposition,” in Andrew Knight-Hill, ed., Sound and Image: Aesthetics and Practices
(London: Focal Press, 2020).
45 Henry Jenkins, Convergence Culture: Where Old and New Media Collide (New York: New
York University Press, 2006).
46 Ithiel de Sola Pool, Technologies of Freedom (Cambridge, MA: Belknap Press, 1983).
47 Kim Cascone, “The Aesthetics of Failure: ‘Post-Digital’ Tendencies in Contemporary
Computer Music,” in Computer Music Journal, vol. 24, no. 4 (Cambridge, MA: MIT
Press, 2000).
48 Caleb Kelly, Cracked Media: The Sound of Malfunction (Cambridge, MA: MIT Press,
2009).
49 Greg Hainge, “Of Glitch and Men: The Place of the Human in the Successful Integra-
tion of Failure and Noise in the Digital Realm,” in Communication Theory, vol. 17, no. 1
(Hoboken: Wiley, 2007).
50 Yasanao Tone, Solo For Wounded CD liner notes (New York: Tzadik, 1997).
51 Lynne Heller, “The Intrinsic Irony of the Future Sublime,” in Canadian Review of Ameri-
can Studies, vol. 50, no. 3 (Toronto: University of Toronto Press, 2020).
52 Ryoji Ikeda, Dataplex [2001–05] liner notes (Chemnitz: Raster Noton, 2005).
53 Kim Cascone, “The Aesthetics of Failure: ‘Post-Digital’ Tendencies in Contemporary
Computer Music,” in Computer Music Journal, vol. 24, no. 4 (Cambridge, MA: MIT
Press, 2000).
7
LIVE VISUALS IN THEORY
AND ART
Paul Goodfellow

Introduction
This chapter considers Live Visuals in relation to art history and theory. The
chapter begins with a history of contemporary art which suggests two discern-
ible but overlapping narratives within art history over the past 100 years. These
are formalism and avant-garde, and the proposal is made that Live Visuals exhibit
characteristics of both fields, and particular attention is paid to two periods of
artistic development. The first period is the early 20th century when the avant-
garde developed in parallel to the formalism of painting and the traditional nar-
rative of art history. The second period is the 1960s, when the ideas and methods
of the avant-garde were absorbed within mainstream art, laying the foundations
for both contemporary art and media art.
Live Visuals can be understood as a form of media art, and the chapter con-
tinues with a consideration of the bifurcation of contemporary art and media
art and the failure of these two fields to address both their shared history and
fundamental differences meaningfully. The chapter considers the primary failure
of contemporary art as the lack of deep engagement with contemporary tech-
nological culture. At the same time, the primary failure of media art is a lack
of engagement with the discourses and philosophies framing contemporary art.
The separation of technology-infused media art from conceptually driven
contemporary art has created a space to reimagine Live Visuals from a contempo-
rary or ‘post-conceptual’ perspective. The characteristics of the post-conceptual
art object are introduced, and Live Visuals are measured against these qualities.
In particular, post-conceptual art is considered as a means of transporting signs
within culture and the ability of Live Visuals to operate as both index and icon.
The chapter concludes with a consideration of the liveness of Live Visuals and
their ability to channel the conditions of contemporary culture through the pro-
duction of shared visceral or affective experiences.
DOI: 10.4324/9781003282396-10
 Live Visuals in Theory and Art 165

Bricolage
The approach employed is a form of bricolage – the weaving together of narratives
from art history and ideas from art theory to offer a new perspective on Live
Visuals. Claude Lévi-Strauss developed the concept of bricolage to describe the
process of combining available tools and ideas and repurposing them for a new
problem. He suggested that these

elements are collected and retained on the principle that ‘they may always
come in handy.’ Such elements are specialised up to a point, sufficiently
for the ‘bricoleur’ not to need the equipment and knowledge of all trades
and professions, but not enough for each of them to have one definite and
determinate use.1

Bricolage neatly describes this consideration of art history and art theory, as these
are vast, heterogeneous and contested fields of knowledge. Consequently, this
chapter is a bricolage of art history and ideas combined to offer new insight into
the role of Live Visuals within contemporary culture and, specifically, its poten-
tial relationship to contemporary art.

History: Framing Live Visuals Within Art History

The Histories of Art: Formalism, the Avant-Garde and the 1960s

The history of art is complex but essentially describes the role visual objects
have played within culture. These objects were at first visual with aesthetic and
narrative roles, but through the course of the 20th century became mediated by
language, and the conceptual dimension of the work began to dominate both
their morphology and role within culture. This shift ref lected the expansion of
visual culture, and the production of images during this period as art became one
amongst many forms of aesthetic communication. Developments in photogra-
phy, cinema, computer games, virtual reality and even social media have created
new image forms, which may exceed art objects in purely visual and aesthetic
terms. This expansion of visuality has simultaneously narrowed and refined art’s
aesthetic role and expanded its dialogical role within cultural discourse. Under-
standing these shifts in the historical and ontological status of art will allow us to
appreciate better the unique qualities of the Live Visual image within contem-
porary culture.
This evolution can be mapped over the past 150 years as culture transformed
through the modernist and postmodernist epochs. Although the modernist proj-
ect of evolving society through scientific and technological advancement is still
the dominant driver of change within contemporary culture, the ‘modernist’
era as defined in cultural (philosophical and art historical) terms operating from
the late 19th century through to the 1960s, though arguably modernism held
sway until the 1980s in some areas, such as contemporary music. This period
166 Paul Goodfellow

witnessed the development of technologies essential to Live Visuals, such as

photography, cinema, television and computing. More broadly, infrastructure
transformed society as telecommunications, rail, f light and container shipping
increased global connectivity and distributed food and consumer goods. Criti-
cally speaking, this era was replaced, or at least decentred, by postmodernism
in the 1970s and 1980s when the simple narratives of progress through science,
technology and capitalism started to be questioned. It is at this pivotal point,
where modernism meets postmodernism, that ‘media art’ forms were born and
defined in opposition to, or at least outside of, the institutional support for con-
temporary art, which emerged during this period.
Live Visuals in a contemporary sense can be traced to the media-based art
emerging in the 1960s, and it is instructive to understand its shared lineage with
contemporary art. To understand the convergence and divergence of media art
with contemporary art, one must consider the art of the 20th century as having
two dominant narratives: formalism/post-formalism and the avant-garde. The
first narrative is the trajectory of formalism, particularly within painting, and
the second is the narrative of the non-painterly avant-garde. In the first narrative,
art moved from a pre-modern era which was mimetic or representational, to the
modern era, which was non-mimetic or non-representational. Clement Green-
berg drove this narrative through his essays Towards a Newer Laocoon (1940)2 and
Modernist Painting (1960)3 in which he historicised the move away from mime-
sis. Arthur Danto summarised Greenberg’s position as a form of Kantian self-
criticism in the sense that abstract painting became increasingly concerned with
the materials and process of a painting, as opposed to a dialogue with a subject
external to the canvas.4 Greenberg argued that there was a single line drawn
from Giotto, through Delacroix, the Impressionists, Post-Impressionists and the
Abstract Expressionists, which left little room, according to Sandford Schwartz,
for the ideas of the avant-garde.5
This narrow reading of art and art history was increasingly contested, as it
did not account for a broad range of work, ideas and technologies that were
embraced by the avant-garde decades before their wider acceptance within
mainstream art of the 1960s. Additionally, the critic Hal Foster noted that the
exclusion of surrealism also meant that three important discourses of moder-
nity: “psychoanalysis, cultural Marxism, and ethnology,” were being side-
lined by the narrow focus on painting.6 There were, of course, developments
within painting, such as cubism, which were revolutionary in painting terms
and have been defined as avant-garde. However, for this discussion, the term
is restricted to the progressive movements and developments within art that do
not fall within the lineage of painting but extend and challenge the boundar-
ies of art in conceptual and technological terms. This stricter definition of the
avant-garde is, according to Peter Bürger, inherently political, and he cites
Renato Poggioli, who described modernist forward-facing work as exhibiting
the “strategy of negation” as many forms of avant-garde art sought to disrupt
the status quo.7
 Live Visuals in Theory and Art 167

The second narrative – the avant-garde – illustrates both the profound inf lu-
ence radical artists of the 1920s had on the art of the 1960s but also suggests rea-
sons why media-based art has been side-lined within institutional art discourse
due to the avant-garde’s focus on technologies and its explicit anticipation of the
future. Henri de Saint-Simon first employed the term avant-garde in 1845 to
describe progressive thinking and action, stating: “the power of the artists is in
fact most immediate and most rapid: when we wish to spread ideas among men,”
and he continued that this was “exercising a positive power over society, a true
priestly function, and of marching forcefully in the van [vanguard] of all the
intellectual faculties.”8
The avant-garde more fully emerged in the early 20th century as a collection
of co-evolutionary movements – futurism, dadaism and surrealism, all of which
were a response on some level to the psychological and political traumas of World
War I. Although futurism, led by Filippo Tommaso Marinetti, emerged in Italy
prior to the war, its inf luence extended across Europe after 1918 and informed the
Vorticism movement in Britain. Both movements were interested in the concepts
of time and progress and the depiction of movement.9 The Italian futurists were
intensely patriotic and were sympathetic to fascism which has, to some degree,
diminished their role within art narratives. However, their contribution to the
avant-garde must be acknowledged. In particular, futurism sought to conceptualise
the future and understood the power of technologies to affect change through mass
media and social engagement. Claire Bishop notes futurism’s use of the ‘serate’ or
‘evening party’ as a precursor to the art happenings of the 1960s as they mixed the
reading of political and artistic manifestos with music, poetry and painting.10
Dada originally developed in Zürich as a loose collective of artists before
spreading to other European cities and New York. Dada was the antecedent
of conceptual art, defining itself as a form of ‘anti-art’ which moved beyond
aesthetic concerns to hold “bourgeois culture” to account for WWI.11 Hugo
Ball, one of the founders of dada, wanted “to draw attention, across the barriers
of war and nationalism, to the few independent spirits who live for other ide-
als” through chaotic live art events.12 These early ‘multimedia’ events, which
would include the performance of contradictory manifestos, poetry in multiple
languages, chanting and music created on non-traditional instruments such as
typewriters and pots, can be seen as the forerunner to the happenings and Fluxus
events of the 1960s and multimedia art installations of the 1990s, as discussed in
Chapters 3 and 13, respectively.
It can be argued that the most important figure emanating from the avant-
garde was Marcel Duchamp, as he was the first artist to highlight the conceptual
dimension of an artwork explicitly. In 1917 he famously presented a commer-
cially produced urinal as an art object Fountain (1917), which simultaneously
foregrounded the conceptual dimension of the artwork and the aesthetic quali-
ties of the object.13 The work also marked the use of found objects as ‘ready-
mades,’ and such reappropriation became central to the pop art of the 1960s and
VJ and remix culture of Live Visuals.
168 Paul Goodfellow

Duchamp also employed methods of chance in the production of work, which

anticipated the strategies of Fluxus, minimalism and systems artists and the use
of randomness and chance in the production and editing of Live Visual perfor-
mances. His most famous chance work, 3 Stoppages etalon (1913–1914), or 3 Stan-
dard Stoppages, was created by dropping three 1-metre threads onto a canvas and
fixing how they landed with varnish. Duchamp felt this employment of chance
was liberating, stating that it “was a way to escape from those traditional methods
of expression long associated with art.”14 However, it was Duchamp’s The Bride
Stripped Bare by Her Bachelors, Even (The Large Glass) (1923) which established the
complex nature of the conceptual artwork and combines ideas, processes and
artefacts into a single work (see Figure 7.1).
The Dadaist strategies for art production and societal disruption morphed
into the surrealist movement, which emerged in France around artists including
Andre Breton, Max Ernst, Salvador Dalí, Luis Buñuel and Man Ray. However,
a broad and heterogeneous movement, a central tenet of surrealism, was the
concept of ‘automatism’ articulated in Breton’s Surrealist Manifesto. Automatism
was a strategy employed by surrealist artists to produce art and writing in direct
and intuitive ways, which bypassed the conscious decision-making of the artist.15
Automatic drawing, for example, sought to express the subconscious through the
largely unmediated movement of the hand and is analogous to the free expres-
sion of a jazz musician who feels and deviates from the structure of a jazz stan-
dard. The inf luence of surrealist automatism can be seen within the work of Live
Visualists as they produce and mix imagery in real time. Like automatic drawing,
Live Visuals are the product of both technical and creative knowledge mixed
with an intuitive engagement with the materials through performance. The act’s
very liveness reasserts the artist’s agency within works that would otherwise
appear technologically determined.
As with the futurists and dada, surrealism was interested in reworking words
to free them from their normative meaning. The futurist Marinetti, for example,
called for the “destruction of the syntax” and to free words from their “syntactic
chains.”16 Surrealism took this wordplay further and, drawing from the psy-
choanalytic writing of Sigmund Freud, imagined the subconscious connections
made between words. This use of words, simultaneously concrete materials and
reconfigurable combinations, became the basis of conceptual art in the 1960s.
Boris Groys has suggested that this complex play of free words, which allows for
infinite repurposing and interpretation, anticipates if not the model of Google,
then its morphology, suggesting that “Google - with its metalinguistic, opera-
tional, and manipulative approach to language - establishes itself [. . .] in the tra-
dition of twentieth-century avant-garde art.”17 Likewise, this can be applied to
the play of imagery within Live Visuals as the artist or VJ curates heterogeneous
images. The audience contextualises and connects the f low of disparate images
based on their personal understanding and experience.
Underpinning the wordplay, deambulation, automatism and their use of
experimental photography was a deep interest in the unconscious mind and psy-
choanalysis. Drawing again from Freud, the image, particularly the photograph,
 Live Visuals in Theory and Art 169

FIGURE 7.1 Marcel Duchamp, The Bride Stripped Bare by Her Bachelors, Even (The
Large Glass), 1923. Philadelphia Museum of Art. Photo by Richard
Winchell, via Flickr. Used by permission.
Source: www.artsy.net/news/artsy-editorial-works-pablo-picasso-marcel-duchamp-public-domain

was seen as a dream or wish and the art object a materialisation of this “conf licted
desire.”18 Of particular interest within surrealism was the photographic image
and its quality of “mirroring” the subject. Through various visual manipulations,
such as double exposures, the juxtaposition of positive and negative prints and
170 Paul Goodfellow

montage suggested “the world [is] redoubled as sign.”19 Drawing from Freud’s
essay The Uncanny (1919),20 the image of the ghost was a powerful image to sug-
gest the uncanny double of the living and the photograph as an uncanny record
of a once live event. Moreover, as noted in Chapter 5, Live Visuals exhibit the
uncanniness of the contemporary systems-based culture as they reveal the ghosts
and traces of the algorithm. Finally, surrealism also fundamentally understood
the power of participation and the importance of embodied experience, two
essential ingredients of Live Visuals. This is evidenced in André Breton’s perfor-
mative and participatory walks around Paris, or deambulation, which in turn led
to Guy Debord’s psychogeographic walks in the 1950s and the conceptual walks
of Richard Long, such as A Line Made by Walking (1967) and performative studio
walks of Bruce Nauman, such as Walking with Contrapposto (1968).
However, the avant-garde was not restricted to the West, and several nota-
ble developments in Russia should be considered. Firstly, Kazimir Malevich’s
suprematism imagined an art form in which pure feeling or perception domi-
nated a work. In his essay Suprematism (1927), Malevich stated, “I understand the
supremacy of pure feeling in creative art” and continued that art should arrive at
“nonobjective representation.”21 This overwhelming sensory experience is dem-
onstrated in the light installations of James Turrell and immersive Live Visual
experiences, an example being the collaboration between the musician J. Space-
man of the band Spiritualized, the director Jonathan Glazer and One of Us for
the Coachella festival in which they produced a dark “cathedral-like space” with
“isolated pools of light” being triggered by Spiritualized’s Ladies and Gentlemen,
We Are Floating in Space (2011).22
A critical development in the Russian avant-garde was Soviet montage the-
ory, an inf luential film movement that proposed five different types of montage
techniques: metric, rhythmic, tonal, overtonal and intellectual.23 Sergei Eisen-
stein’s Battleship Potemkin (1925) is a famous example, with the Odessa Steps scene
incorporating multiple forms of montage to create an experience that remains
both exciting and affective. The visual experimentation reached new technical
and conceptual heights with Dziga Vertov’s Man with a Movie Camera (1929),
in which he incorporated a range of visual techniques which remain central to
Live Visual performance. These include slow-motion, multiple exposures, split-
screen and reversed footage.24 Conceptually the work was also radical, anticipat-
ing the self-ref lexivity of contemporary art and postmodernism. Vertov’s work
was underpinned by his concept of ‘Kino-Glaz,’ an idea that anticipated post-
humanism and proposed that one’s sublimation with machines would lead to
new forms of perception not directly accessible through the human eye.25
Finally, the Russian avant-garde theatre of Proletkult (Proletarian Cultural-
Educational Organisation) encouraged collective authorship and experience
through live theatrical events.26 The founder of Proletkult, Aleksandr Bogda-
nov, understood the power of art to activate an audience, suggesting that “art
can organise feelings in exactly the same way as ideological propaganda [organ-
ises] thought; feelings determine will with no less force than ideas.”27 Proletkult
 Live Visuals in Theory and Art 171

demonstrated the power of art to move a mass audience through emotion, and
this has been deployed within cinema and television to create narratives with
mass appeal and within live music and club events to direct the participants’
emotions.
It is beyond the scope of this chapter to explore why the progressive ideas
and processes of the avant-garde took several decades to enter mainstream insti-
tutional art in the 1960s. One can, however, speculate that the conservative
market forces of modern art and the cultural traumas of World War II signifi-
cantly delayed the absorption of avant-garde within mainstream institutional
art. However, one can observe that the two narratives of art – formalism and
the avant-garde – met in the late 1960s to produce a wide range of conceptually
driven art forms, which would, in turn, lead to contemporary art, media art and
Live Visuals. These new strands of art can be summarised as minimalism, per-
formance art, pop art, conceptual art, kinetic art and systems art. Underpinning
this explosion of movements were the radical ideas of the avant-garde which
had been carried through from surrealism, the situationist international and
the writing of Guy Debord in the 1950s, before taking hold in broader culture
through student movements protesting for democracy and the countercultural
revolution – an amalgam of anti-authoritarianism and civil rights protests.28
It is essential to consider the genesis of these new art forms as they demon-
strate that media art and, specifically Live Visuals, fundamentally share the same
DNA as contemporary art. Firstly, from the formalist perspective, one can see
how suprematism and constructivism leads to abstract expressionism, which in
turn leads to minimalism, and all of these movements can be understood in
terms of ‘action’ and ‘process.’29 This procedural and formalist aesthetic can be
seen in contemporary Live Visuals, which exhibit both photographic and paint-
erly attributes. Whereas traditional film and cinema have a strictly photographic
foundation, the digital image and Live Visuals have a more complex underpin-
ning whereby the pixel values are manipulated, mixed and layered in a painterly
fashion.30 Another aspect of the 1960s art scene that has inf luenced Live Visuals
is performance art, which in turn can trace its roots to the futurist serate and the
Dadaist cabaret.
The overriding contribution of art in the 1960s was the understanding that
art was primarily a conceptual activity supported through the production and
circulation of aesthetic objects. Groys suggests that the emergence of minimal-
ism and conceptual art in the 1960s can be understood “as the completion of the
revolutionary process that the classical avant-garde initiated at the beginning
of the twentieth century.”31 Groys argues that the institutional acceptance and
deployment of the radical ideas of the earlier avant-garde had the effect of simul-
taneously fulfilling the conceptual remit of the avant-garde whilst neutralising
the radicalism and, in particular, the avant-garde’s engagement with the future.
If conceptual art was the logical conclusion of the avant-garde, two specific
conceptual art forms, pop art and systems art, emerged in the 1960s, directly
shaping media art and Live Visuals. Firstly, pop art emerged as a response to and
172 Paul Goodfellow

appropriation of mass-produced images, and we can think of Andy Warhol’s

appropriation of Campbell’s soup cans, first used in 1961 and the packaging of
Brillo soap pads (1964) as key examples of work which simultaneously employed
and critiqued advertising and commercial graphic design.32 Pop art can be seen
as evolving from dada’s use of the readymade and the more politically motivated
détournement of the situationist international which emerged from surrealism in
the 1950s. Détournement redeployed, or détourned, familiar imagery to reveal the
workings of capitalism and mass media.33 However, Warhol’s shrewd relation-
ship with the art market contrasts with situationist international’s overtly Marx-
ist critique of culture. This political ambiguity and playful relationship with the
mechanisms of capital anticipated postmodernism, and Warhol’s inf luence can
be seen directly within the work and strategies of Jeff Koons and Damien Hirst,
both of whom toy with the market and the readymade.
In contrast, the spirit of détournement can be seen in the more explicitly politi-
cal work of Jenny Holzer and Barbara Kruger and the ‘culture jamming’ move-
ment.34 This range of appropriation is also present within Live Visuals, as at one
end, some works uncritically contribute to advertising and commercial enter-
tainment events, and at the other, there are works that challenge ideas through
the processes of liveness and détournement. We can think of the live documen-
taries of The Light Surgeons as live détournement as they challenge the audience
through the juxtaposition and reconfiguration of imagery.
Secondly, systems art emerged in the 1960s as a form of conceptual art that
dealt with cybernetics: information and information systems. As discussed in
Chapter 5, Katherine Hayles described the three developmental periods or waves
of cybernetics that emerged since 1945.35 The first wave was focused on the sta-
bility of the system through homeostasis and the processes of feedback. Second-
wave cybernetics, which emerged in the late 1950s, focused on the ref lexive
nature of systems whereby participants would alter the system under observa-
tion. As discussed in Chapter 5, during the early 1960s, several artists started to
explore the first two waves of cybernetics, and in particular the ecological sys-
tems operating in nature and the cybernetic systems operating within technology
to establish systems art, which focused on the communication of information.
The principal text on cybernetics, Shannon and Weaver’s Mathematical The-
ory of Communication (1949), defined ‘communication’ through the fundamental
concepts of the message, signal and noise.36 They described ‘information’ in a
strictly technical sense of signal clarity, thus separating the medium (signal) from
the message (meaning), which became the defining principle of conceptual art
in the late 1960s. It also laid the foundation for media art to focus not on the
circulated messages but also the technological systems of representation and the
spectacle of data presentation.
Inf luential systems artists include Les Lévine, Hans Haacke, Nam June Paik
and Roy Ascott, who articulated both the biological and technological foun-
dations of his work. The artist, writer and curator Jack Burnham mapped out
systems art in his inf luential Artforum essays: Systems Esthetics (sic) (1968) and
 Live Visuals in Theory and Art 173

Real-time Systems (1969). In Systems Esthetics (sic), for example, Burnham stressed
both the relational nature of art and its conceptual debt owed to Duchamp, stat-
ing “that art does not reside in material entities, but in relations between people
and between people and the components of their environment. This accounts for
the radicality of Duchamp and his enduring inf luence.”37
Other inf luential systemic developments in this period included the publica-
tion of Buckminster Fuller’s Operating Manual for Spaceship Earth (1969)38 that
described the Earth in terms of planetary systems. Fuller’s descriptive mix of
ecological and technological thinking anticipated the contemporary understand-
ing of nature found in the writing of Timothy Morton.39 and Erich Hörl40 as one
of ecological-technological entanglement. However, as outlined in Chapter 3, it
was the publication of Gene Youngblood’s Expanded Cinema41 that most directly
laid the foundation for video art and media art and marked the beginnings of
its bifurcation from contemporary art. As with any writing which anticipates
the future, both the content and tone of the text locate it at a specific time in
the past. However, Youngblood correctly anticipated the shift from the cin-
ematic broadcast era to a more dialogical “era of image-exchange between man
and man.”42 Youngblood also imagined the shift to networked communication
as an enfoldment within technology, and this is something contemporary art
has largely resisted and media art has embraced. The original multimedia artist
Nam June Paik bridged this latent divide between contemporary art and media
art. His work is historically significant as it performs the “media and message”
dichotomy which defines the divide between media art, which focuses on the
media, and contemporary art, which focuses on the message.

The Failures of Contemporary Art and Media Art

The two parallel art narratives of formalism and the avant-garde co-evolved dur-
ing the 20th century before comingling in the late 1960s to create new hybrid
art forms which were simultaneously formalist, conceptual and technological.
However, by the early 1990s, two distinct branches of art, both of which shared
this 1960s, formalist and avant-garde DNA, could be seen operating within
culture. Firstly, the institutional art of the late 1960s had grown exponentially
through new museums, art fairs and biennial exhibitions and became known as
contemporary art and post-conceptual art. The second branch of art, media art,
continued exploring the potentials of the emerging media and computer tech-
nologies, and this field grew its own institutional and support networks, includ-
ing the digital art festivals Transmediale (Berlin), Ars Electronica (Linz) and ISEA
International (formerly Inter-Society for the Electronic Arts). In Contemporary
Art and New Media Digital Divide or Hybrid Discourse? Edward A. Shanken offers
a thorough analysis of the divisions between contemporary art and (new) media
art. He notes that both fields have supported the development of artists, curators,
theorists and pedagogies, but their “discourses have become increasingly diver-
gent.”43 Shanken argues that (new) media art is to some degree excluded from the
174 Paul Goodfellow

contemporary art institutions and debates, and this section considers the reasons
for this dialectical failure and its implications for the status of Live Visuals. On
one side, media art has failed to fully engage with the concerns of contemporary
art, and contemporary art has failed to fully engage with the digital realities of
21st-century culture.44
The failure of contemporary art to address media art and digital art can be
understood in terms of four structural f laws within the contemporary art model.
These four weaknesses or blind spots can be summarised as the post-medium
condition and the turn away from technologies, the failure of art schools to
integrate technologies, the shift away from aesthetics and finally, the failure to
address the future.
Contemporary art, and in particular conceptually driven art, emphasises the
idea over the object. This development in the late 1960s but rooted in the
avant-garde marked a turn away from the new technological substrates emerg-
ing during this era. This vigorous conceptuality was articulated in Sol LeWitt’s
Artforum article Paragraphs on Conceptual Art (1967), in which he stated that “in
conceptual art the idea or concept is the most important aspect of the work” and
“the idea becomes a machine that makes the art.”45 LeWitt and this famous state-
ment have been historically aligned with the rise of serialism and systems art
and are therefore a direct antecedent of media art. However, the statement is not
an advocation of technology-augmented art production, but a de-technological
statement suggesting the idea is the algorithm. He makes this explicit when
he criticised the new emerging art forms stating, “New materials are one of
the great aff lictions of contemporary art. Some artists confuse new materials
with new ideas. There is nothing worse than seeing art that wallows in gaudy
baubles.”46 He continued, “The danger is, I think, in making the physicality of
the materials so important that it becomes the idea of the work (another kind
of expressionism).”47
The central concern is that new materials and technologies would eclipse the
conceptual component of the artwork and reduce the art experience to one of
shallow spectacle. This focus on the messages being circulated away from the
technological substrate which transported the message can then be understood
as an inversion of Marshall McLuhan’s famous phrase “the medium is the mes-
sage.” In Understanding Media: The Extensions of Man (1964), McLuhan proposed
that the medium of communication dominates the message, and conceptual
art can therefore be understood as a challenge to the hegemony of information
technology.48
The idea that the “message is the message” was central to the work of ‘Art &
Language,’ a conceptual art collective established in 1967. They were critical
of the f low of art discourse through traditional studio practice and the gallery
system and instead circulated their ideas through journals and text-based works.
This distillation of conceptual art to the text chimed with the move from mod-
ernism to postmodernism articulated in the writing of Roland Barthes, Jacques
Derrida and Umberto Eco, each expressing ways in which the text or artwork
 Live Visuals in Theory and Art 175

should be understood as an “open work” that can have a life independent from
the original artist or author.49
This focus on the conceptual, open artwork shifted attention away from spe-
cific mediums: painting, sculpture and photography and on to the underlying
idea. Rosalind Krauss described this as the “post-medium condition” in her text
A Voyage on the North Sea: Art in the Age of the Post-Medium Condition (1973), in
which she charts conceptual art’s turn towards poststructuralist philosophy.50 In
this inf luential text, Krauss suggests the antecedents of media art such as Young-
blood’s Expanded Cinema, video art and hybrid media art were, in contrast to
conceptual art (at least at the level of technology), subservient to mass media and
capital, and the inference is that they should be treated with a degree of critical
suspicion.
This paradigmatic shift within art impacted education, and art schools moved
away from specialist degrees in sculpture, painting and photography to offer
degrees in ‘fine art’ and ‘contemporary art.’ In these new conceptual degrees,
all material supports, including digital technologies, were of equal value. How-
ever, as technically and financially demanding digital tools were not taught,
students tended to revert to the more traditional accessible materials of modern
art. Thus, there was a technological gap within art education, and it was only
with the advent of mass consumption of digital tools in the 21st century, such as
the image manipulation of Photoshop, the algorithmic filtering of Instagram or
the mimetic distribution of TikTok, that digitally mediated images are becoming
central to the practice of young artists. However, it can be argued that many of
these contemporary artists are approaching the digital sphere from a subjugated
consumer position, as their access to imagery and data is mediated by technology
and social media companies. This subjugated state was discussed in Chapter 5
concerning the technological “nostalgia mode” of post-conceptual art.
This is not to suggest that contemporary art has not engaged in our contem-
porary condition; instead, it has approached this in a largely non-technological
way to consider ways in which we mediate and are mediated by information and
information technologies. This can be seen in the 1980s in the work of Hol-
zer and Kruger, who employed visual and conceptual strategies of advertising
to critique the operations of capitalism. Barry Smart, referencing Hal Foster,
described such work as a “postmodernism of resistance.”51 Such work was both
underpinned and understood through post-structural theory, which described
society in terms of organisational structure. These theories included Bruno
Latour’s ‘actor-network theory,’52 which described society in terms of networks,
and Gilles Deleuze and Félix Guattari’s ‘assemblage theory,’53 which emphasised
the dynamic and relational nature of society. By the 1990s, a range of artists
inf luenced by the relational thinking of Nicolas Bourriaud and his text Relational
Aesthetics54 (discussed in Chapter 13) began to address the pervading information
and technological systems more directly through works of social engagement.
Thus, contemporary art engaged with many concepts, such as interaction, par-
ticipation, programming and networks, which would have been familiar to the
176 Paul Goodfellow

earlier generation of systems or Fluxus artists of the 1960s and have remained
central to the media artists of the 21st century.55 However, as they are performing
these concerns in analogue and analogously, they do not, as Bishop suggests, “really
confront the question of what it means to think, see, and filter affect through the
digital”56 If contemporary art’s response to our digital, networked and information
age is, as Bishop suggests, to return to “the analog, the archival, the obsolete and
predigital modes of communication,”57 how can it express the aesthetic and affec-
tive complexities of our contemporary digital- and systems-mediated condition?
If the primary failure of contemporary art is the lack of engagement with con-
temporary technology as either a tool or medium, then media art’s primary failure
is its lack of engagement with the discourses and institutions of contemporary art.
Of course, such an engagement is dialectical and requires contemporary art to be
open to new forms of media. This situation begs the question as to whether media
art will, in the long run, continue to operate as the technological avant-garde or
be absorbed into mainstream institutional art. If the tools of media art become
absorbed within the toolkit of the contemporary artist, the distinction between
the two fields will inevitably fade. Likewise, as media art engages with the con-
ceptual ideas of contemporary art, which largely operate independently of specific
technological substrates, and we can think of the relational aesthetics and relational
architecture, discussed in Chapter 13, then the distinctions will be further eroded.
If contemporary art can be defined as predominantly conceptual and the
channelling of ideas, then media art can be defined as predominantly techno-
logical and the channelling of data. Whereas contemporary art can be under-
stood as dealing with the expansion of the photographic image, media art can be
understood as dealing with the expansion of information systems. Thus, contem-
porary art can be characterised as looking backwards to measure contemporary
experience and images against history, whereas media art can be characterised
as looking forward and anticipating the future. On one level, the progressive
nature of media art is a strength, making it the natural heir to the first wave of
the avant-garde. However, the need to always be focused on the next technologi-
cal breakthrough makes it vulnerable to novelty, technological materialism 58 and
uncritical ahistorism. These vulnerabilities are explored through Guy Debord’s
concept of the ‘spectacle’ and Walter Benjamin’s concept of ‘aura.’
The danger of focussing on novelty and surface appearance was anticipated
in LeWitt’s comments earlier when he suggested that the uncritical employ-
ment of new materials can produce “gaudy baubles.” This warning aligns with
Debord’s concept of the ‘spectacle,’ which he articulated in the Society of the
Spectacle (1968).59 Debord’s spectacle refers to the alienation felt within capitalist
culture as it diverts us from an authentic lived experience. Debord suggested that
“everything that was directly lived has receded into representation.”60
Debord presciently anticipated the 21st century in which the circulation of
images on the Internet and social media increasingly decentres the need or, more
importantly, the desire for direct first-hand experience. Debord’s arguments
were refined and extended by Jean Baudrillard in Simulacra and Simulation (1983),
 Live Visuals in Theory and Art 177

in which he observes that “we live in a world where there is more and more
information, and less and less meaning.”61 Baudrillard argued that the informa-
tional content of a message contained in a text, photograph or film is exhausted
through its over-circulation within society. This can be seen in the reduction
of a news event, for example, whereby the original commentary on the climate
crisis or late capitalism is diminished and distorted into an ironic meme.62
These criticisms can be applied to the content of media art as it contributes
to the over-circulation of images and has the same effect of using imagery and
ideas as mere surface and spectacle. However, one can counter this argument
by suggesting firstly that many media artists are fully aware of the spectacular
and recursive nature of visual culture. We can think of the work of The Light
Surgeons (and their project SuperEverything* as discussed in Chapters 12 and 17),
for example, whose work critiques or holds a mirror up to society to challenge
the psychologically destabilising nature of (social) media culture. One could also
argue that the very liveness of Live Visuals counteracts the passivity and disem-
bodied nature of social media images, as Live Visuals are temporal, spatial and
fundamentally a corporal experience.
The vitality of Live Visuals and media art leads to a persistent criticism of digitally
mediated art – its lack of ‘aura.’ This concept was developed by Walter Benjamin
across several essays in which he argues that reproducible art forms such as photog-
raphy and film lack the aura evident in earlier non-reproducible art, such as paint-
ing and sculpture. In the most famous essay, The Work of Art in the Age of Mechanical
Reproduction (1968),63 Benjamin defines the aura of traditional handmade art objects
as possessing a unique essence or presence which cannot be carried through into
mechanically reproduced copies, stating that “even the most perfect reproduction
of a work of art is lacking in one element: its presence in time and space, its unique
existence at the place where it happens to be.”64 He argues that the original aura-
infused artwork has its path through history traced in its surface, and this linearity
cannot be claimed for copies, as they are not the original physical object.
This challenge to media art can be considered from two perspectives: liveness
and technological materiality. Firstly, Benjamin made the distinction between
performances recorded in film and live performances on stage, and this can be
applied to Live Visuals, as the presence of the performer and audience generates
the aura.65 This holds true even for Live Visual performances that are generated
from algorithms, as the event is still contingent on the dialogical relationship
between the audience and algorithm or audience and machine. The ‘auratic’
nature or unique quality of digitally mediated events is dramatically illustrated
by the ‘listening parties’ of Kanye West. During these events, West plays pre-
recorded music, but creates a shared live experience with the audience. In Benja-
min’s terms, the pre-recorded music and visuals in themselves are ‘non-auratic,’
but they can be used in live ‘auratic’ events. This raises interesting questions
regarding the definition of the Live Visual as an artwork. Is the performance the
artwork, or the underlying essence, code or technologies the artwork? If the Live
Visuals are generated by an algorithm, does the original code exhibit aura and
178 Paul Goodfellow

FIGURE 7.2 Cory Arcangel, I Shot Andy Warhol, 2002. Handmade hacked Hogan’s Alley
cartridge, Nintendo Entertainment System video game system and light
gun. (Installation view: The New York/Liverpool Project, Liverpool
Biennial Independents, Liverpool, UK, September, 2004-November,
2004. Photo: Michael Connor. Used by permission.
Source: https://coryarcangel.com/things-i-made/2002-002-i-shot-andy-warhol
 Live Visuals in Theory and Art 179

is this aura perceptible to the audience? This search for the essence or origins of
a digital work may explain, in part, the current fascination with non-fungible
tokens (NFTs) within art, as they demonstrate the potential to locate a digital
work in space, time and the marketplace.
Secondly, media art and, in particular, Live Visuals can be considered in rela-
tion to the technologies which support their production and performance. In
Reinventing the Medium (1999),66 Krauss develops an idea originally articulated by
Benjamin that a technology in service to capitalism cannot be fully experienced
aesthetically, and only when it is side-lined by new, more efficient technology can
we consider older technologies as both objects and vehicles for art experiences. In
particular, Krauss discusses the use of outmoded slide projectors in the work of
James Coleman to suggest that such works allow the ‘auratic’ qualities of both the
technological support (the projector) and the processes of dissemination (projec-
tion) to be revealed. Krauss suggests that the work, freed from its functional role
as technology, can be employed by Coleman in a work focussed on a “paradoxical
collision between stillness and movement that the static slide provokes right at the
interstice of its changes.”67 A similar argument could be made for any number of
media artists who employ recently defunct or do-it-yourself (DIY) technologies
to highlight the aesthetic and affective experience of the digital condition. Cory
Arcangel, for example, has hacked old computer games to elicit their aesthetic
content. This can be seen in Arcangel’s work I Shot Andy Warhol (2002), in which
he hacks a Nintendo video game to reveal both the nostalgia for the medium and
nostalgia for the art historical reference of Andy Warhol (see Figure 7.2).
Thus, there are cultural, technological and philosophical reasons why media
art and Live Visuals, in particular, are being excluded from contemporary art
discourse, and this excluded state was the central focus of a contentious and
widely discussed article by Claire Bishop in Artforum: Digital Divide: Contempo-
rary Art And New Media (2012).68 Bishop’s article and the ensuing criticisms of
her arguments, which would include Shanken as noted earlier,69 help map out
the failure of contemporary art and media art to communicate both their areas
of commonality and their unique differences. However, there are some distinct
characteristics of contemporary art against which we must measure Live Visuals
if they are to be considered as art.

The Characteristics of Post-Conceptual Art: Conceptuality,

Aesthetics and Distribution
As Terry Smith has noted, the term ‘contemporary’ has replaced the earlier terms
of ‘modern’ and ‘postmodern’ to describe art of the 1990s onward, but its mean-
ing is simultaneously “obvious and opaque.” 70 However, on one level the term
contemporary art can be understood as referring to “the institutionalized net-
work through which the art of today presents itself to itself and to its interested
audiences all over the world.” 71 This would include the museums, contemporary
180 Paul Goodfellow

galleries and international art fairs and biennials, such as Biennale Arte (Venice)
and Documenta (Kassel), through which a complex and sophisticated cultural dis-
course has developed.
At the centre of this increasingly globalised activity is the contemporary art
object, which can be understood as post-conceptual in constitution. This term
is unpacked in great detail by Peter Osborne in Anywhere or Not at All: The
Philosophy of Contemporary Art (2013)72 and The Postconceptual Condition (2018).73
Osborne suggests that all contemporary art should be understood as ‘postcon-
ceptual,’ as it is by nature conceptual. Additionally, Osborne suggests that the
post-conceptual artwork is both ‘aesthetic’ and ‘radically distributed,’ and these
qualities are now brief ly discussed with reference to Live Visuals.74
Firstly, Osborne reiterates that contemporary art is post-conceptual, mean-
ing it is morphologically conceptual, being defined by its ideas and relation-
ships within the art system. This constitution will include the explicit distinction
between art and non-art objects. This strict definition would exclude much of
Live Visual art production, as it has not been created within the community of
contemporary art. This is not to say that Live Visuals could not be appropriated
to act as objects within post-conceptual art, but they would need to be activated
by art insiders to achieve this.
Secondly, post-conceptual art must have some form of materialisation in
order to exhibit an aesthetic dimension. Even though the conceptual dimension
of the work dominates, the post-conceptual art object must have experienceable
materialisation, which is located in space and time. Something exists, and we
can experience or feel it. Osborne describes this as the “ineliminable – but radi-
cally insufficient – aesthetic dimension” of the artwork and suggests this points
to a “failure” of the first wave of conceptual art in the late 1960s to produce
purely conceptual work devoid of aesthetic content.75 Osborne presents the term
“failure” bounded in quotation marks, as the project of complete aesthetic evis-
ceration was doomed to fail. This “failure” was understood, demonstrated and
subverted in the minimalist textual artworks of ‘Art & Language’ as they were
presented as aesthetic editioned prints and exhibited in galleries.76
In contrast, Live Visuals are invariably presented outside of the institutional
supports of the gallery system and therefore need to rely on the aesthetic dimen-
sion to attract attention. This is not to suggest that Live Visuals do not contain
conceptual content. Rather, Live Visuals in music, theatre or multimedia con-
texts must compete for attention with other audio-visual stimulation. They are
therefore not marked within culture as primarily conceptual objects, and this
dimension may be overlooked due to their spectacular nature and the context
of consumption. We can think of the spectacular visuals of a U2 or Coldplay
concert, which may contain important messages with regard to the climate cri-
sis, but the images are read as dressing for the music. In contrast, the concep-
tual content of the art installations of Ryoji Ikeda are foregrounded due to the
institutional context and the fact that the aesthetic or spectacle dimension of the
work is dialled down to reveal both the underlying processes and ideas. Osborne
 Live Visuals in Theory and Art 181

describes an “anti-aesthetic use of aesthetic materials” to foreground the concep-

tual, and Ikeda’s work manifests this quality.77
The third criterion of post-conceptual art proposed by Osborne is the con-
cept of “radical distribution,” which suggests that a work of art is “irreducibly
relational,” meaning that the structural and conceptual coherence of a work sits
across all of the materials which contribute to the work.78 Osborne gives two
examples that illustrate the power of this idea. Firstly, he discusses the earthwork
Spiral Jetty constructed within the Great Salt Lake in Utah by the artist Robert
Smithson (1970). Due to the relative inaccessibility of the work, it was presented
in exhibitions through the display of supporting materials, including a documen-
tary film, maps and drawings. Smithson described these exhibitions as ‘nonsites,’
as they were descriptions of the ‘site’ – Spiral Jetty. Through time the idea of the
artwork has inextricably extended across the site and the nonsite and is enmeshed
within the networks of the art world (galleries, critics and audience), which sus-
tains the work’s relevance.79 A more recent example of a distributed artwork is
The Creation Myth (1998) by Jason Rhoades, which is distributed across a wide
range of processes and materials to explore the nature of creation and how we use
systems to describe natural and social processes (see Figure 7.3).

FIGURE 7.3 Jason Rhoades, The Creation Myth (1998/2015). Installation view. BALTIC
Centre for Contemporary Art, Gateshead, UK, Friedrich Christian Flick
Collection, Berlin. Photo: Colin Davison© 2015 BALTIC Centre for
Contemporary Art. Used by permission.
Source: Local image.
182 Paul Goodfellow

Osborne’s second example is broader in scope and relates to photography as a

distributed work on both the level of photography as a field and the photo-
graphic object.80 Firstly, photography as a field is “distributed across a historically
[. . .] determinate, progressive range of technologico-cultural forms” from early
Victorian experiments through to film, television and the computer-generated
image held together by their similar function within culture.81 Secondly, the
singular ‘photograph’ is also distributed in the sense that there can be numerous
negatives and prints for a single photograph that exists across these materials.
And the case for distribution is even stronger with the computer-mediated or
generated image due to the break in the relationship between the photographed
subject and the generated images. From this perspective Live Visuals meet the
criteria of the post-conceptual artwork, as they are both photographic in nature
and radically distributed across the technologies, images and performances of the
Live Visual event.

Ontology: The Nature of Live Visuals

System and Objects

As discussed earlier, the photograph is arguably the dominant art form of post-
conceptual art and is the antecedent of Live Visuals. Consequently, a brief onto-
logical discussion of the photograph will map out the similarities between the
photographic image and Live Visuals and the ways in which they fundamentally
diverge.
If painting is a construction that captures something from the interior life of
the artist, then many early theorists of photography and film have suggested that
the mechanical processes of the camera captured something of external real-
ity. In The Ontology of the Photographic Image (1960), for example, André Bazin
claimed the following:

For the first time, between the originating object and its reproduction
there intervenes only the instrumentality of a nonliving agent. For the first
time an image of the world is formed automatically, without the creative
intervention of man. The personality of the photographer enters into the
proceedings only in his selection of the object to be photographed and by
way of the purpose he has in mind.82

Bazin’s idea of the direct 1–2–1 relationship between the subject, the external
world and the ‘object-image’ is, however, challenged by the increasingly dis-
tributed nature of the photograph and the very act of taking a photograph. As
Osborne observed, the photograph is not a discrete object, but sits across a range
of material substrates, including the institutions and media which support its
employment within culture. Graham Harman, referencing Martin Heidegger,
 Live Visuals in Theory and Art 183

would describe this expanded description as “unconscious holism,” as we see the

photograph in the broader context of what it is doing within culture. Only when
the photographic images are separated from this more comprehensive system and
reappropriated within collage or Live Visuals (or taken out of context) can we
begin to see the content of the original images.83
Harman and his philosophy of object-oriented ontology assert the ontologi-
cal complexity of the artwork but using a fundamentally opposite argument to
Osborne and the relational thinkers (Deleuze, Latour and Bourriaud) whose
writing underpins post-conceptual art. At the root of Harman’s philosophy is
the contention that all material objects and social interactions can be understood
as objects.84 From this perspective, the single physical photograph is an object,
but also the wider distributed and institutionally supported photographic image
is also an object. Harman notes there is a danger within culture to upwardly
reduce or “paraphrase” objects in terms of their relationships, arguing that an
object is always more than the relationships and networks it operates within.85
Harman’s expanded definition of objects would therefore accommodate the
photograph, Live Visuals or the artist performing them as being objects operat-
ing in the world.
Osborne has demonstrated that the post-conceptual artwork performs a com-
plex role within culture and is necessarily distributed and therefore involved in
relationships. However, Harman’s object-oriented ontology reminds us that there
must be an object at the centre of each artwork, which is more than its social
relations, suggesting “a thing acts because it exists rather than existing because it
acts.”86 Harman describes the object as having a “withdrawn” and “inexhaustible”
nature, as they are always more than the roles they play within a given context.87
Returning this to Live Visuals, we can understand that there are images
which are either photographic, painterly or algorithmic in origin, and these can
be generated, performed or remixed in real time, and both the source imagery
and the new images and the wider experience generated by the live event can
be understood as both a distributed system of production in Osborne’s sense
and an art object in Harman’s sense of the term ‘object.’ They are simultane-
ously systems and objects, and this aligns with our lived experience in which
we sense objects as both simultaneously image and experience88 or ‘Living
Visuals.’ This can be illustrated with the work of the art collective Squidsoup,
who create immersive Live Visuals which can be understood as both a system
and object. They describe their ongoing project Murmuration in the following
terms:

Murmuration is an immersive walkthrough public art installation that uses

some 500 individually suspended points of light and sound to create a
responsive data swarm that swirls around visitors’ heads. The piece uses
mesh-networked data systems to connect 500 bespoke orbs that each con-
trol and emit light and sound, creating dynamic f lows of live data, sound
184 Paul Goodfellow

and light. These f lowing behaviours are inspired by the murmurations

(f locking patterns) of starlings.89

This work is both a system and object, and this can be understood on several lev-
els. Firstly, the work simulates the behaviour of the starlings which act individu-
ally to create a system of movement. This behaviour in nature is upwardly causal
in the sense that the overall structure and patterns of the murmuration are built
from the ground up from the movements of the individual bird. Secondly, this
complex f low of movement is simulated using mesh-networked data, and this
can be understood as a system. Thirdly, the work has been presented in numer-
ous locations, and each time the configuration of the orbs has been different due
to conceptual, topographic and architectural differences. However, despite the
physical changes the work remains a coherent artwork or object. Finally, each
person who experiences the work will, on some level, complete the work in their
mind as their bodies move through the space. Each person will have a unique
experience of the work, but this will not, to use Harman’s terms, “exhaust” the
artwork of new meanings, as the full extent of the object is “withdrawn.”
Accepting that a photographic image or Live Visual event is simultaneously a
system of relations and a complex and mysterious object which has an indepen-
dent life from the artist or performer, we can consider its communicatory role
within culture. This will be approached in two ways. First, we can consider Live
Visuals as a system of signs and secondly as a system of communication.

The Live Image as Index, Icon and Communication System

The assumption within this discussion of Live Visuals is that they are either
generated, mediated or projected as digital information. There are therefore
significant differences between a chemically based film or photograph and the
digital image, and in particular the live digital image.90 These differences can be
understood in terms of the three forms of sign originally formulated by Charles
Sanders Peirce: the symbol, index and icon.91 Braxton Soderman highlighted the
key differences between the analogue and digital photograph in The Index and the
Algorithm (1999)92 in which he summarised the three signs as follows: “An icon
resembles or imitates the object it signifies; an index shares a physical connection
with the object it represents; and a symbol is associated with meaning through
convention or habit,” and these distinctions can be applied to both image types.93
Based on these criteria Peirce described the traditional photograph as being
simultaneously index and icon, as it has both a physical (indexical) connection to
the subject, as the film reacts to the light ref lected from the subject, and an iconic
appearance of the subject represented in the exposed photograph.94
In contrast, a digital photograph has a less direct or mediated relationship
with the subject, as the image is not solely created from the ref lected light but
captured as data with the camera’s sensor. There is still an indexical relationship
between the image and the external subject, as Laura Marks states, “If all matter is
 Live Visuals in Theory and Art 185

intimately interconnected by wave-surfing electrons, then all electronic images

have an indexical or analog connection to – matter.”95 However, the final image
is the product of translation, as the ref lected colours are mediated and sampled by
the sensor and the algorithms employed by the camera to ‘improve’ the image.
The indexical relationship to the external material world is further eroded
with Live Visuals, as these have been manipulated through image processing
tools and intuitive live mixing techniques to extract and even generate new
information. The VJ performer, for example, may be manipulating the distri-
bution of colour values to elicit new aesthetic sensations or juxtaposing two
discrete images to create something new in iconic or symbol terms. All such
post-production and remixing processes complicate the provenance of the Live
Visual images, and either erode or distort the indexicality specifically in relation
to the subject of the material.
Consequently, Live Visuals are indexical in several ways. Firstly, like the
traditional photograph, there is “material indexicality,”96 not in relation to
the chemical processes, but in relation to the sensed values used to create the
image. Secondly, the image exhibits indexicality with the algorithms employed
in its production; as Soderman states, “We must begin to recognize that digital
images, as indexical signs, can refer to conceptual objects (computer programs or
algorithmic processes).”97
However, there is a third indexical relationship present within the Live Visual,
which is the movements of the performer as she manipulates the images and how
this interaction is traced and embedded within the performed images. Thus,
the Live Visual image has a threefold indexical relationship between the origi-
nal sensed object, the algorithms which have transformed the data and the live
performance, which channels something of the artist. Live Visuals are therefore
uncanny as they are not recording a past event but offer “ghostly references” to
their complex indexicality.98
Thus, Live Visuals transport both indexical and iconic information, and this
f low of information can be considered by returning to the work of the artists
who first investigated the cybernetic and technological field. In the catalogue
for the seminal systems art exhibition Software (1971) Jack Burnham observed the
move towards information-based art, stating:

In the past few years, the movement away from art objects has been pre-
cipitated by concerns with natural and man-made systems, processes,
ecological relationships, and the philosophical-linguistic involvement of
Conceptual Art. All of these interests deal with art which is transactional;
they deal with the underlying structures of communications or energy
exchanges instead of abstract appearances.99

Burnham knew that to understand the nature of art communication within

increasingly conceptual, ephemeral and digitally mediated works (and this
would include Live Visuals), one needs to think of them in terms of systems and
186 Paul Goodfellow

information ‘f low.’ The concept of information ‘f low’ is important, as it maps

how an idea is transported from the artist via the artwork or performance to the
audience. Similar concepts developed within the natural sciences, cybernetics
and systems theory as they each developed relational descriptions of observ-
able or imagined systems. As noted, earlier Shannon and Weaver described how
information f lows from the sender (the artist) via the channel of communication
(the artwork) to the receiver (the audience). Thus, we can consider what sort of
information is being transmitted, either consciously or unconsciously, through
the artwork by the artist. For example, a largely abstract painting will channel
both aesthetic information and affective information. The aesthetic information
relates to the composition and play of colour, whereas the affective information
is embedded in the surface of the canvas as a record of the physical and emotional
act of painting. In contrast, a conceptual artwork predominantly channels the
conscious ideas of the artist, albeit contained within an object or performance
which has an aesthetic dimension.
This returns us to the accusation levelled at contemporary art by the art
historian Claire Bishop, namely, that contemporary art does not “confront the
question of what it means to think, see, and filter affect through the digi-
tal.”100 Contemporary art has not engaged with digital materials and tools at
a deep level and therefore does not channel and filter the digital experience
in a way equivalent to the Abstract Expressionists’ affective engagement with
paint. As noted in Chapter 5, contemporary art has predominantly engaged
with the digital world at the level of subjugated consumer, as evidenced in the
nostalgia-infused aesthetic of post-Internet art. Such work is looking backward
and merely extends contemporary art’s archival and historicising impulses. In
contrast, media art and Live Visuals in particular express their technological
and informational morphology, equipping them to both channel the excesses
of information of contemporary culture and exhibit the effect of this media-
saturated condition.

Conclusion: The Affect of the Post-Systems Condition

The two art historical narratives of formalism and the avant-garde demonstrate
how conceptual art emerged within the first decades of the 20th century in the
work of Duchamp, but only started to take hold within mainstream institutional
art in the 1960s. During this period new art movements, including minimalism,
conceptual art, pop art and systems art, emerged that dealt with conceptualism;
information; and the proliferation of social, political and technological systems.
After this period there was a bifurcation between mainstream institutional art,
which dealt with conceptualism in terms of the text, and postmodernism and
media art, which dealt with conceptualism in terms of information and its cir-
culation through technology and mass media. Live Visuals are generally located
within media art, and it has therefore been instructive to understand the parallel
histories of media art and contemporary art.
 Live Visuals in Theory and Art 187

Conceptual art and culture more broadly articulate communication in terms

of perceptible messages which can be rationalised and articulated through lan-
guage. However, the brain processes meaningful information in two distinct but
interrelated ways, with the left hemisphere providing a conscious, logical and
linguistic assessment of received information and the right hemisphere offering a
more direct sensual ‘picture.’101 Thus, one could argue that conceptual and post-
conceptual art foregrounds the operations of the left hemisphere, and formalist
art, for example, abstract expressionism, foregrounds the sensual operations of
the right hemisphere. However, Osborne observed that the aesthetic dimension –
and we can include affective information – is irreducible even within conceptual
art, and this is clearly understood within the art world, evidenced in the atten-
tion paid to the aesthetics of the art object and supporting catalogues and the
design and atmosphere of galleries.
Live Visuals occupy an interesting position within the conceptual-aesthetic
debate, as the content of the imagery can be both conceptual and aesthetic.
However, the live creation and performance of these images is fundamentally
a visceral or affective experience for both the artist and audience. It is the very
liveness of the imagery which generates the new affective feelings as the partici-
pants are confronted with new sensory and corporal information. Iain McGil-
christ describes these preconscious reactions as the “primacy of affect,” as they
occur before they can be assessed and translated into conscious thoughts or lan-
guage, whilst Brian Massumi describes this preconscious reaction of the nervous
system as “visceral perception.”102 These seemingly instantaneous reactions have
developed as an evolutionary mechanism to aid survival. We ‘feel’ these things
before we can rationalise them due to the strong communication between the
right hemisphere and the amygdala.103
In an age defined by sensory overload, we feel things before we make sense of
them. Pleasure – we experience something pre-linguistic when we anticipate the
beat dropping in electronic dance music. Corporal horror – we feel foreboding
when we see images of climate collapse – f loods and fire, before we understand
the human-induced source of our nightmare. As Franklin Ginn suggests, we are
not watching apocalyptic films anymore but are participants within the apoca-
lypse104 or existential dread – in the overwhelming circulation of images within
contemporary culture which is both performed and critiqued in the production
of Live Visual experiences. The f low of imagery and information reminds us
that we are losing our sense of self within the matrix of informational excess. As
Morton states, “The more information we acquire in the greedy pursuit of see-
ing everything, the more our sense of a deep, rich, coherent world will appear
unavailable: it will seem to have faded into the past (nostalgia) or to belong only
to others.”105
In an age of complexity, defined by late capital, the climate crisis, pandem-
ics and information excess, we need art forms which are able to channel this
complexity in order to filter the affects of a systems- and technology-mediated
culture. I describe our present state as the “post-systems condition,” as it captures
188 Paul Goodfellow

FIGURE 7.4 KAWS, COMPANION (EXPANDED) in London viewed through the

Acute Art app, 2020, augmented reality. Courtesy of KAWS and Acute
Art. Photo: Isabela Herig. Used by permission.
Source: https://acuteart.com/artist/kaws/

how we are “haunted by the cybernetic and technological extensions of nature,”

with the prefix of ‘post’ referring to the givenness of our systems-mediated state.
This can be illustrated with the final Live Visuals example, the augmented real-
ity (AR) artwork COMPANION (EXPANDED) (2020) by the artist KAWS
(see Figure 7.4). The work is a three-dimensional model of a KAWS character
presented as digital information and viewable only through a smart phone using
an AR app. COMPANION (EXPANDED) could be seen f loating in the air
above cities, including London and New York, at specific times and locations.
He covers his eyes with the palms of his hands to suggest he is either playing
the children’s game hide-and-seek with the viewer or he is overwhelmed by
his own digital transmogrification. He is simultaneously ‘live’ but ‘virtual’ and
‘everywhere’ but ‘nowhere,’ and this cybernetic state reinforces the viewers’ own
corporality as they stand on the Earth at a specific place at a specific time, gazing
upward at the momentary glimpse of simulated reality.
This chapter has employed a bricolage approach to art history and theory to
consider the place of Live Visuals within contemporary art. The overlapping
narratives within art history over the past 100 years have been formalism and
the avant-garde, and it was demonstrated how they converged in the late 1960s
 Live Visuals in Theory and Art 189

to produce new strands of art, including minimalism, conceptual art, pop art
and systems art, and laid the foundations for contemporary art and media art.
Although contemporary art and media art share the same art roots, they have
evolved into two distinct fields of cultural activity with their own artists, insti-
tutions and audience. It was argued that Live Visuals as a form of media art has
been largely ignored by the debates and institutions of contemporary art, and
this should be understood as a dialectical failure – as media art has failed to fully
engage with the philosophical and linguistic concerns of contemporary art and
contemporary art has failed to fully engage with the technological and infor-
mational concerns of media art. A specific criticism raised by Claire Bishop is
contemporary art’s seeming inability to articulate our post-systems condition,
and this chapter has suggested ways in which Live Visuals not only “think, see,
and filter affect through the digital” but how this can be communicated within
the post-conceptual art framework.

Notes
1 Claude Lévi-Strauss, The Savage Mind (Chicago: University of Chicago Press, 1968), 17.
2 Clement Greenberg and John O’Brian, The Collected Essays and Criticism, Volume 1:
Perceptions and Judgments, 1939–1944, Art Criticism (Chicago: University of Chicago
Press, 1988), 23–37.
3 Clement Greenberg and John O’Brian, The Collected Essays and Criticism, Volume 4:
Modernism with a Vengeance, 1957–1969, Art Criticism (Chicago: University of Chicago
Press, 1995), 85–93.
4 Arthur C. Danto, After the End of Art: Contemporary Art and the Pale of History, A.W.
Mellon Lectures in the Fine Arts (Princeton, NJ: Princeton University Press, 1997), 7.
5 Sandford Schwartz, “Clement Greenberg: The Critic and His Artists,” The American
Scholar, 56(4), 1987, 537.
6 Hal Foster, Compulsive Beauty, An October Book (Cambridge, MA: MIT Press, 1995), xiv.
7 Peter Bürger, Theory of the Avant-Garde, Theory and History of Literature (Manchester:
Manchester University Press, 1984), xv.
8 Donald D. Egbert, “The Idea of Avant-Garde in Art and Politics,” Leonardo, 3(1), 1970,
76, https://doi.org/10.2307/1572057.
9 Hal Foster et al., Art since 1900: Modernism, Antimodernism, Postmodernism (London:
Thames and Hudson, 2016), 90–97.
10 Claire Bishop, Artificial Hells: Participatory Art and the Politics of Spectatorship (London/
New York: Verso, 2012), 42–49.
11 Hal Foster et al., Art since 1900: Modernism, Antimodernism, Postmodernism (London:
Thames and Hudson, 2016), 135.
12 Keith Aspley, Historical Dictionary of Surrealism, Historical Dictionaries of Literature and
the Arts (Lanham, MD: Scarecrow Press, 2010), 95.
13 Hal Foster et al., Art since 1900: Modernism, Antimodernism, Postmodernism (London:
Thames and Hudson, 2016), 127–129.
14 Dalia Judovitz, Unpacking Duchamp: Art in Transit (Berkeley: University of California
Press, 1998), 35.
15 Hal Foster et al., Art since 1900: Modernism, Antimodernism, Postmodernism (London:
Thames and Hudson, 2016), 190–195.
16 Boris Groys, In the Flow (Brooklyn: Verso Books, 2016), 153.
17 Boris Groys, In the Flow (Brooklyn: Verso Books, 2016), 153–154.
18 Hal Foster et al., Art since 1900: Modernism, Antimodernism, Postmodernism (London:
Thames and Hudson, 2016), 15–21.
190 Paul Goodfellow

19 Hal Foster et al., Art since 1900: Modernism, Antimodernism, Postmodernism (London:
Thames and Hudson, 2016), 194–195.
20 Sigmund Freud, David McLintock, and Hugh Haughton, The Uncanny, Penguin Clas-
sics (London: Penguin Adult, 2003).
21 Herschel B. Chipp, Theories of Modern Art: A Source Book by Artists and Critics, Revised
edition (Berkeley, CA: University of California Press, 1992), 341.
22 “A Look at the New Installation from J. Spaceman and Jonathan Glazer,” www.vice.com/
en/article/ezakdz/a-look-at-the-new-installation-from-j-spaceman-and-jonathan-
glazer (Accessed November 21, 2021).
23 Michael Betancourt, Structuring Time: Notes on Making Movies (Cabin John, MD: Wild-
side Press, 2004), 66.
24 Lev Manovich, The Language of New Media, Leonardo (Series) (Cambridge, MA: MIT
Press, 2001), 239–243.
25 Lev Manovich, The Language of New Media, Leonardo (Series) (Cambridge, MA: MIT
Press, 2001), 275–276.
26 Claire Bishop, Artificial Hells: Participatory Art and the Politics of Spectatorship (Brooklyn:
Verso, 2012), 42–66.
27 Claire Bishop, Artificial Hells: Participatory Art and the Politics of Spectatorship (Brooklyn:
Verso, 2012), 52.
28 Tor Egil Førland, “Cutting the Sixties Down to Size: Conceptualizing, Historiciz-
ing, Explaining,” Journal for the Study of Radicalism, 9(2), 2015, 125–148, https://doi.
org/10.14321/jstudradi.9.2.0125.
29 Marc Glimcher, Logical Conclusions: 40 Years of Rule-Based Art (New York City: Pace
Wildenstein, 2005), 6–13.
30 Lev Manovich, The Language of New Media, Leonardo (Series) (Cambridge, MA: MIT
Press, 2001), 305–306.
31 Donna De Salvo (ed.), Open Systems: Rethinking Art C.1970, 1st edition (London: Tate
Publishing, 2005), 53.
32 Anne Rorimer, New Art in the 60s and 70s: Redefining Reality, 1st edition (London:
Thames and Hudson Ltd, 2004), 18–19.
33 Hal Foster et al., Art since 1900: Modernism, Antimodernism, Postmodernism (London:
Thames and Hudson, 2016), 394–395.
34 David Hopkins, After Modern Art 1945–2000, Oxford History of Art (Oxford: Oxford
University Press, 2000), 210–211.
35 Katherine N. Hayles, How We Became Posthuman: Virtual Bodies in Cybernetics, Literature,
and Informatics, 74th edition (Chicago: University of Chicago Press, 1999).
36 Professor Fritjof Capra and Pier Luigi Luisi, The Systems View of Life: A Unifying Vision
(Cambridge: Cambridge University Press, 2014), 93.
37 Jack Burnham, Melissa Ragain, and Hans Haacke, Dissolve into Comprehension: Writings
and Interviews, 1964–2004 (Cambridge, MA: MIT Press, 2015), 117.
38 Richard Buckminster Fuller, Operating Manual for Spaceship Earth, edited by Jaime Sny-
der (Baden: Lars Muller Publishers, 2008).
39 Timothy Morton, Hyperobjects: Philosophy and Ecology after the End of the World (Min-
neapolis: University of Minnesota Press, 2013).
40 Erich Horl, General Ecology: The New Ecological Paradigm, edited by James Edward Bur-
ton (London: Bloomsbury Academic, 2017).
41 Gene Youngblood, Expanded Cinema (New York, NY: E. P. Dutton, 1970).
42 Gene Youngblood, Expanded Cinema (New York, NY: E. P. Dutton, 1970), 49.
43 Christiane Paul, A Companion to Digital Art (Hoboken, NJ: John Wiley & Sons, 2016),
463.
44 Christiane Paul, A Companion to Digital Art (Hoboken, NJ: John Wiley & Sons, 2016),
463–478.
45 Sol LeWitt, “Paragraphs on Conceptual Art,” Artforum, Summer 1967, www.artforum.
com/print/196706/paragraphs-on-conceptual-art-36719.
 Live Visuals in Theory and Art 191

46 Sol LeWitt, “Paragraphs on Conceptual Art,” Artforum, Summer 1967, www.artforum.

com/print/196706/paragraphs-on-conceptual-art-36719.
47 Sol LeWitt, “Paragraphs on Conceptual Art,” Artforum, Summer 1967, www.artforum.
com/print/196706/paragraphs-on-conceptual-art-36719.
48 M. McLuhan, Understanding Media: The Extensions of Man, Routledge Classics (Lon-
don: Routledge, 2001).
49 Umberto Eco and David Robey, The Open Work, translated by Anna Cancogni (Cam-
bridge, MA: Harvard University Press, 1989), 84.
50 Rosalind E. Krauss, A Voyage on the North Sea: Art in the Age of the Post-Medium Condi-
tion, Walter Neurath Memorial Lectures (London: Thames & Hudson, 2000).
51 Barry Smart, Postmodernity, Key Ideas (London: Taylor & Francis, 2016), 20.
52 Bruno Latour, Reassembling the Social: An Introduction to Actor-Network-Theory, New edi-
tion (Oxford: Oxford University Press, 2007).
53 Gilles Deleuze and Félix Guattari, A Thousand Plateaus: Capitalism and Schizophrenia,
New edition (London: Continuum International Publishing Group, 2004).
54 Nicolas Bourriaud, Relational Aesthetics (Dijon: Les Presse Du Reel, 1998).
55 Christiane Paul, A Companion to Digital Art (Hoboken, NJ: John Wiley & Sons, 2016),
463.
56 Claire Bishop, “Digital Divide: Contemporary Art and New Media,” Artforum, Septem-
ber 2012, www.artforum.com/print/201207/digital-divide-contemporary-art-and-
new-media-31944.
57 Claire Bishop, “Digital Divide: Contemporary Art and New Media,” Artforum, Septem-
ber 2012, www.artforum.com/print/201207/digital-divide-contemporary-art-and-
new-media-31944.
58 Christiane Paul, A Companion to Digital Art (Hoboken, NJ: John Wiley & Sons, 2016), 467.
59 Guy Debord, Society of the Spectacle: Written by Guy Debord, 2009 Edition (Polegate, UK:
Soul Bay Press, 2009).
60 Guy Debord, Society of the Spectacle: Written by Guy Debord, 2009 Edition (Polegate, UK:
Soul Bay Press, 2009), 2.
61 Jean Baudrillard, Simulacra and Simulation, translated by Sheila Glaser (Ann Arbor: Uni-
versity of Michigan Press, 1994), 79.
62 Jean Baudrillard, Simulacra and Simulation, translated by Sheila Glaser (Ann Arbor: Uni-
versity of Michigan Press, 1994), 81.
63 Walter Benjamin, The Work of Art in the Age of Mechanical Reproduction (London: Pen-
guin Books Limited, 2008).
64 Walter Benjamin, The Work of Art in the Age of Mechanical Reproduction (London: Pen-
guin Books Limited, 2008), 5.
65 Walter Benjamin, The Work of Art in the Age of Mechanical Reproduction (London: Pen-
guin Books Limited, 2008), 21.
66 Rosalind E. Krauss, “Reinventing the Medium,” Critical Inquiry, 25(2), 1999, 289–305.
67 Rosalind E. Krauss, “Reinventing the Medium,” Critical Inquiry, 25(2), 1999, 297.
68 Claire Bishop, “Digital Divide: Contemporary Art and New Media,” Artforum, Septem-
ber 2012, www.artforum.com/print/201207/digital-divide-contemporary-art-and-
new-media-31944.
69 Christiane Paul, A Companion to Digital Art (Hoboken, NJ: John Wiley & Sons, 2016),
466.
70 Terry Smith, “What Is Contemporary Art? Contemporaneity and Art to Come,”
Konsthistorisk Tidskrift/Journal of Art History 71(1–2), April 1, 2002, 3, https://doi.
org/10.1080/002336002320273313.
71 Terry Smith, “What Is Contemporary Art? Contemporaneity and Art to Come,”
Konsthistorisk Tidskrift/Journal of Art History, 71(1–2), April 1, 2002, 3, https://doi.
org/10.1080/002336002320273313.
72 Peter Osborne, Anywhere or Not at All: Philosophy of Contemporary Art, 1 edition (Brook-
lyn: Verso, 2013).
192 Paul Goodfellow

73 Peter Osborne, The Postconceptual Condition: Critical Essays (Brooklyn: Verso, 2018).
74 Peter Osborne, Anywhere or Not at All: Philosophy of Contemporary Art, 1 edition (Brook-
lyn: Verso, 2013), 48.
75 Peter Osborne, Anywhere or Not at All: Philosophy of Contemporary Art, 1 edition (Brook-
lyn: Verso, 2013), 48–49.
76 Charles Harrison, Essays on Art and Language, 2nd Revised edition (Cambridge, MA:
MIT Press, 2002).
77 Peter Osborne, Anywhere or Not at All: Philosophy of Contemporary Art, 1 edition (Brook-
lyn: Verso, 2013), 46–49.
78 Peter Osborne, Anywhere or Not at All: Philosophy of Contemporary Art, 1 edition (Brook-
lyn: Verso, 2013), 46–49.
79 Francis Halsall, Systems of Art: Art, History and Systems Theory, 1st New edition (Bern;
Oxford: Verlag Peter Lang, 2008), 146–151.
80 Peter Osborne, Anywhere or Not at All: Philosophy of Contemporary Art, 1 edition (Brook-
lyn: Verso, 2013), 120–125.
81 Peter Osborne, Anywhere or Not at All: Philosophy of Contemporary Art, 1 edition (Brook-
lyn: Verso, 2013), 123.
82 André Bazin and Hugh Gray, “The Ontology of the Photographic Image,” Film Quar-
terly, 13(4), 1960, 7, https://doi.org/10.2307/1210183.
83 Graham Harman, Art and Objects (Cambridge, UK: Wiley, 2019), 17–18, 32–33.
84 Graham Harman, The Quadruple Object, Reprint edition (Winchester, UK: Zero Books,
2011).
85 Graham Harman, The Quadruple Object, Reprint edition (Winchester, UK: Zero Books,
2011), 7–19.
86 Graham Harman, Immaterialism: Objects and Social Theory (Malden, MA: Polity Press,
2016), 7.
87 Graham Harman, Immaterialism: Objects and Social Theory (Malden, MA: Polity Press,
2016).
88 Richard Grusin (ed.), The Nonhuman Turn (Minneapolis: University of Minnesota Press,
2015), 228.
89 Murmuration – Squidsoup.Org, www.squidsoup.org/portfolio/murmuration-2/ (Accessed
November 21, 2021).
90 Paul Goodfellow, “Reframing the Horizon within the Algorithmic Landscape of
Northern Britain,” Arts, 8(3), 2019, https://doi.org/10.3390/arts8030114.
91 J.D. Johansen and S.E. Larsen, Signs in Use: An Introduction to Semiotics (London:
Taylor & Francis, 2005).
92 Braxton Soderman, “The Index and the Algorithm,” Differences, 18(1), May 1, 2007,
153–186, https://doi.org/10.1215/10407391-2006-026.
93 Braxton Soderman, “The Index and the Algorithm,” Differences, 18(1), May 1, 2007,
156–157, https://doi.org/10.1215/10407391-2006-026.
94 Braxton Soderman, “The Index and the Algorithm,” Differences, 18(1), May 1, 2007,
158, https://doi.org/10.1215/10407391-2006-026.
95 Laura U. Marks, “How Electrons Remember,” MFJ, 34, 1999, 73, www.mfj-online.
org/journalPages/MFJ34/LMarks.html (Accessed November 20, 2021).
96 Janne Seppänen, “Unruly Representation: Materiality, Indexicality and Agency of the
Photographic Trace,” Photographies, 10(1), January 2, 2017, 113–128, https://doi.org/1
0.1080/17540763.2016.1258658.
97 Braxton Soderman, “The Index and the Algorithm,” Differences, 18(1), May 1, 2007,
164, https://doi.org/10.1215/10407391-2006-026.
98 Ingrid Hoelzl and Remi Marie, Softimage: Towards a New Theory of the Digital Image
(Bristol: Intellect, 2015), 54.
99 Charlie Gere, Art, Time and Technology, English edition (Oxford and New York: Berg
Publishers, 2006), 131.
 Live Visuals in Theory and Art 193

100 Claire Bishop, “Digital Divide: Contemporary Art and New Media,” Artforum, Septem-
ber 2012, www.artforum.com/print/201207/digital-divide-contemporary-art-and-
new-media-31944.
101 Iain McGilchrist, The Master and His Emissary: The Divided Brain and the Making of the
Western World (New Haven, CT: Yale University Press, 2019).
102 Brian Massumi, Stanely Fish, and Fredric Jameson, Parables for the Virtual: Movement,
Affect, Sensation, Post-Contemporary Interventions (Durham, NC: Duke University
Press, 2002), 60–61.
103 Iain McGilchrist, The Master and His Emissary: The Divided Brain and the Making of the
Western World (New Haven: Yale University Press, 2019), 42.
104 Franklin Ginn, “When Horses Won’t Eat: Apocalypse and the Anthropocene,” Annals
of the Association of American Geographers, 105(2), March 4, 2015, 359, https://doi.org/
10.1080/00045608.2014.988100.
105 Timothy Morton, Ecological Thought, Reprint edition (Cambridge, MA and London:
Harvard University Press, 2012), 56.
8
LIVE VISUALS
Technology and Aesthetics

Léon McCarthy

Introduction
Live Visual performers rely on technologically enabled workf lows to realise their
art to such a degree that technology is as much their medium as light. This is
evident throughout Live Visuals practice: from the pre-production of motion
graphics to the design of control interfaces through to the selection of media-
diffusion systems. As Part I of this book has already shown, trends in aesthetics
tend to change from decade to decade: what this chapter will argue is that aesthet-
ics tend to converge into trends when a critical mass of artists appropriate tech-
nologies in similar ways. As a result, within Live Visual performance practice,
aesthetics are profoundly coupled to technological processes and inventions.
The term ‘live’ in Live Visual performance emphasises that it is art created
for and in the presence of an audience: the rendering of the artwork happens in
real time as an ephemeral experience shared between performer and audience.
The level of spontaneity that a performer strives for will vary, but in all cases
the presence of the audience impacts on the performance that is realised. This
chapter will discuss how the aesthetics of Live Visuals are the function of three
components: artist, technology and audience.

The Role of Technology

This discussion on the impact that technology has on the practice of the Live
Visual artist begins with reference to J. J. Gibson’s term ‘affordance’: one he
originally coined in order to describe the possibilities for action offered to an
individual by their environment “what it provides or furnishes, either for good
or ill.”1 Gibson conceived of affordances in two ways:

1 One in which affordance is a factor of the environment alone

2 One in which affordance is a factor of both the environment and the artist
DOI: 10.4324/9781003282396-11
 Live Visuals 195

What is of interest to this discussion is the interaction between the artist and the
technologies in use, so it is Gibson’s second type of ‘affordance’ that we can use
as a reference point.
The curiosity of the artist drives them to explore new tools in order to
realise their vision. What emerges as the work of art will have been shaped by
how they reacted to technical limitations as they strove to harness the potential
of a technology. The artwork is therefore a manifestation of the limitations
inherent in the interplay between the artist and their tools, i.e. the ‘affordance’
of their tools. When the artist returns to use the same tools again, their previ-
ous experience is remembered, and so as J. M. Culkin wrote when reviewing
Marshall McLuhan’s media theory: “we shape our tools and thereafter they
shape us.”2
In the mid-20th century by reappropriating old naval missile equipment, the
Whitney brothers were able to generate elaborate visual patterns based on math-
ematical formulae, and thus a new form of algorithmically generated visuals
was born. It was a ground-breaking process, one that foresaw the need for more
powerful computing tools.
It was not until the late 20th century that computers became readily available
to artists; early adopters looked to previous aesthetics that espoused the use of
such processes. In this way the Whitneys’ form of abstraction inf luenced early
exploration of the computer as a tool. Once software tools were created to sup-
port the design of such visuals, abstraction was refined into a style that to this day
underpins the ‘wallpaper’ aesthetic that many nightclub VJs celebrate (as detailed
in Chapter 12).
The mass uptake of a technology by artists tends to distil individual styles into
a trend. A common communicative code is established through which artists
can infer meaning to diverse audiences via a shared set of semiotic principles, so
that what began as a liminal practice (with Whitney) ultimately entered popular
culture. This also happened when video art inf luenced scratch video, which then
in turn inf luenced the style of music video broadcast to the mainstream via MTV
(see Chapter 4).

The Role of the Artist

As previously outlined, technologies facilitate the process of creation and so
can lead to new aesthetics. However, it is also true that technologies can drive
the conceptual concerns of visual artists, as can be seen when they reappro-
priate technologies to question the fundamentals underpinning those very
technologies.
In the late 1950s the general public watched TV for entertainment, but artists
saw in TV the potential for control of the masses through audio-visual propa-
ganda. Fluxus artists were concerned with the ecology of this new technology,
questioning the societal role that TV played by using it in unintended ways. One
such artist was Nam June Paik, who was concerned with how TV can ‘play’ with
the perceptions of the audience. He experimented with various modifications of
196 Léon McCarthy

television technology before premiering his TV Cello performance in 1971, an

event at which Charlotte Moorman ‘played’ a set of TVs as if they were a cello.3
In the 1980s, broadcast TV again became an object of criticism for scratch video
performers: by sampling short clips from TV and reappropriating them as loops
out of their original context, they commented on the public’s obsession with TV.
In other instances, a new technology may be appropriated in order to meet
the vision of a performer. It was video technology that enabled scratch video
performers to mix different video sources together via a bank of synchronised
video decks: if any synchronisation with the music was to be suggested, it had
to be done manually. As discussed in Chapter 4, Coldcut (Figure 8.1) developed
software to mix sound and video by matching the beats per minute (BPM) of the
music to the duration of the video clip, a version of which was publicly released
as VJAMM.4 In doing so, they pioneered a form of audio-visual performance
based upon a tight synchronisation between the visual and sonic events.
With this tight audio-visual coupling, artists approached the synesthetic ideal
espoused by past film-based artists such as Eisenstein and composers such as Scri-
abin (as discussed in Chapter 2). In terms of the language of film, Eisenstein had
a direct inf luence on directors and editors through his theory of cinematic mon-
tage: a theory in which he espoused the portrayal of a meta-narrative through the
juxtaposition of short, contrasting visual clips.5 In the hands of audio-visual per-
formers such as Coldcut, Hexstatic and DJ Yoda, a highly stylised form of juxta-
position emerged: one that came to be shared across Live Visuals and music video.

FIGURE 8.1 Coldcut performing at London Splice Festival (2017). Photo by Jean-
Paul Berthoin. Used by permission.
Source: See correspondence with images rights holder.
 Live Visuals 197

The Role of the Audience

Since the start of the 20th century, there has been an awareness of the role that
the receiver of the artwork plays in forming an aesthetic response to the art-
work. As Marcel Duchamp put it, “The role of the spectator is to determine the
weight of the work on the aesthetic scale.”6 Duchamp’s insight emerged during
the early modernist period, at a time when the camera was a new technology.
Photography had a huge impact on the visual arts, as prior to then realism was
the sole preserve of the artist. If the photograph could reproduce reality in an
instant, then any artistic merit on the part of the photographer had to emerge in
their attempt to portray something more than mere representation. As Umberto
Eco argues, the theories put forward by Charles Sander Peirce in his “Second
Trichotomy of Signs” suggest that as photo-realism is iconic in nature, higher
meaning is imparted when the receiver seeks for symbolic signs as they receive
the artwork.7 With the arrival of photographic technology, an awareness of the
role that the receiver plays in shaping the aesthetic emerged; however, modernist
theory still held that ‘meaning-making’ was primarily determined by the artist.
In the case of the photographer, this was contingent on their choice of subject,
their use of light, etc.
By the mid-20th century, Roland Barthes had come to question modernist
theories of authorial intent by suggesting that any aim on the part of the artist to
impart meaning was of little relevance, as a multitude of meanings will emerge in
the moment when individuals make manifest the artwork as their own aesthetic
experience: “a text is made of multiple meanings . . . but there is one place where
this multiplicity is focused and that place is the reader.”8 It was from formulations
such as these that postmodernist theory emerged: a theory that held the moment
of reception by the receiver as central, and any attempt by artists to diffuse their
own narrative would fracture in the moment of reception. The temporal struc-
ture and form of fine art broke apart: visible in such movements as the no-wave
punk video movement, in which artists such as Vivienne Dick facilitated multiple
readings by juxtaposing disparate video clips in a seemingly haphazard fashion.

Audience Participation
Compared to the participatory nature of much postmodern performance, in
which the audience is called upon to take an active role in shaping the emergent
artwork, the audience attending a contemporary VJ performance is given a less
active role; beyond voicing an opinion through shouting, dancing or booing, the
meta-narrative emerges according to the trajectory the performer has in mind.
This is no different from mainstream musical performance: pop musicians deliver
live shows to the backdrop of audience feedback that merely raises the level of
excitement, but rarely leads to any significant alteration in the artists’ delivery.
This does not have to be the case: audio-visual performers often use audience
reaction as a part of their live strategy, with audience engagement emerging
198 Léon McCarthy

through active interaction rather than muted observation. Modern technologies

facilitate the audience in shaping the emergent aesthetic: across live cinema and
theatre, the second screen (smartphone) has emerged as a way to explore this
paradigm.9 It could be argued that it is with audience participation that technolo-
gies expand the possibilities for contemporary Live Visual practice, with some
contemporary performers following in the footsteps of Fluxus artists by relying
on the audience to play their part in creating the live experience.
When an interplay between artist and audience emerges, is postmodern theory
an adequate reference point for Live Visuals? A. Kirby’s term ‘pseudo-modernism’
is one that warrants some consideration in the context of the aesthetics of Live
Visuals. Kirby introduces pseudo-modernism, saying it “makes the individual’s
action the necessary condition of the cultural product.”10 Placing the individual
at the centre of our contemporary culture has led us to ‘fetishize’ (to use Kirby’s
term) their role as participant over the validity of their message: perhaps this is
the premise upon which fake news has emerged? Kirby puts it thus: “pseudo-
modernism’s typical intellectual states are ignorance, fanaticism and anxiety.”11 It
is the role of the artist to challenge the assumptions of contemporary culture, so
Kirby’s call to arms is important: “it is more useful to find ways of making these
new conditions conduits for cultural achievements instead of the vacuity cur-
rently evident.”12 The next section will consider how audio-visual performance
can challenge audiences by confronting them with their prejudices.

Engaging the Audience

To challenge an audience, the performer must first attract their attention. We
live in an era of ubiquitous multimedia: while highly dynamic audio-visuals
could once draw in an audience, it can now be a challenge to hold their attention.
For the audio-visual performer to attract audiences yet retain the ephemerality of
the experience, the judicious use of media technologies is required.
Let us begin with the challenge of engaging an audience and some examples of
performers harnessing technology in order to enrich the performer–audience rela-
tionship while retaining the ephemerality of the live experience. The music pro-
ducer and performer Ritchie Hawtin sought a way of engaging his audience with
enriched content related to his choice of music in real time. Together with Bryan
McDade, they developed a plug-in called RADR that extracted song information
from the software Traktor and then included this information in tweets posted to
Hawtin’s timeline. This meant those in the audience could follow Hawtin (via
Twitter) and receive updates to their phones about songs that appeared in his set.
Once engaged, how can audio-visual technology be used to challenge the
audience? Across a series of my own performances, I have used Wi-Fi networks
so the audience could collaborate with me via smartphones to develop an emer-
gent audio-visual narrative. In the betav10 live cinema performance (Figure 8.2),
I displayed a question on the venue’s main screen, with the audience then using
their smartphones to comment to the main screen in response.13 As conversations
 Live Visuals 199

FIGURE 8.2 Léon McCarthy, betav10 (2015), a second-screen live cinema performance.
Used by permission.

emerged, this shaped an audio-visual response that both framed and challenged
their thoughts as a multimedia manifestation of their words. Sometimes this
manifestation would magnify an opinion, while at other times its shock would
cause all commentary to cease. As the performance progressed, the audience
became familiar with the relationship between their comments and my reac-
tions, and therefore a game of cat-and-mouse ensued.
Technology facilitates the emergence of even more liminal aesthetics, chal-
lenging in terms of the labels one can use to describe their nature. Tyler Freeman
developed an interactive second-screen–based scenario called Layer Synthesis
Device for large-scale public events.14 Freeman installed several screens across
the event site so that if those nearby to a screen used his website on their phones,
they obtained control over the on-screen visuals. Individuals may have first
found themselves alone with the screen’s canvas all to themselves, but when
more people joined, it became a game of creative participation and competition.
Freeman played no part in the realisation of each individual experience, yet his
software enabled their experiences; the role of the performer was handed over to
the participants, and so the performer–audience relationship fused.

The Influence of Interface

As discussed in Chapters 1 and 2, the 19th century saw efforts to build instru-
ments that could ‘play’ light. Rather than invent new interfaces, contemporary
200 Léon McCarthy

instruments were repurposed into new arrangements. An example is A. W. Rim-

ington’s colour-organ which harnessed the interface of the organ (keyboard),
so the playing of a key triggered the emission of a coloured light. To become
an accomplished performer capable of spontaneous expression, the instrument
in question needs to be mastered in order to establish subconscious mappings
between the brain and the instrument. It helps if the instrument’s interface
remains immutable: then a period of learning can lead to its mastery. By using
the keyboard, the interface of Rimington’s colour-organ remained in a fixed
arrangement and so it could be mastered over time.
When it is possible to interact directly with one’s medium, an artist may need
no control interface. As Chapter 3 outlined, coloured oil lamps were developed
during the psychedelic era of the mid-20th century; vials of coloured oil were
placed in front of a bright light which slowly heated the coloured oil. Convection
currents ensure a continuous rotation of the oil, and so the image projected onto
the nearby surface was a never-ending wave of colour. What is of interest here
is the absence of a control interface; the device was autonomous in its creation
of visuals.
Somewhat related to the previous method of generating Live Visuals is that
of the live manipulation of film. Thirty-five millimetre projection technology
has always been expensive and so remained within the commercial world of the
movie theatre. In the mid-20th century, 16mm and 8mm film technology made
film a viable medium for artists. Armed with multiple projectors (each running
short loops of 16mm film), projectionists could build layered light compositions.
The aesthetic that emerged when artists combined coloured oil projectors with
16mm projections came to be known as ‘liquid visuals’ (detailed in Chapter 3).
Some of the best-known examples are the shows performed by The Joshua Light
Show and Single Wing Turquoise Bird, as well as Andy Warhol’s projections for
the shows of The Velvet Underground. Film projectors run a spool of film in
front of a bright light, which then projects the images that are contained on the
filmstrip. Projectionists discovered that they could slow down the filmstrip with
their finger as it passes the beam of light. If held long enough, it would melt with
the resulting visual a beautiful explosion of texture and colour. The Canadian
musical group Godspeed You! Black Emperor use a projectionist who manipu-
lates filmstrip as a complement to their performance. Like coloured oil lamps,
this process does not require a control interface.
In the 1980s, video-editing technology was developed to facilitate the cre-
ation of content for TV. Several video decks (each with videotapes) were con-
trolled from a mixer: while one deck was playing, another deck was being set
up (via a preview monitor) for the next cue point. It was challenging to work
with such technology in the off line video editing suite; even more so for the
scratch video performer to do so on stage. In comparison to the tactile nature
of manipulating filmstrip, scratch video was a practice founded upon a techni-
cally complex interface, yet one that enabled the performer to work with a large
library of video content and so create short-form narrative. A rather unsightly
 Live Visuals 201

setup, there was a tendency to locate scratch video performers and their equip-
ment out of sight. Being relegated to the off-stage area occupied by sound and
lighting engineers, there was a tendency for them to be considered technicians
rather than performers.
It was a technical development in the world of DJing that gave scratch video
performers an opportunity to take centre stage. Towards the end of the 1990s,
CDJs became an alternative to the vinyl deck as the instrument of choice for DJs.
The premise of a device that could facilitate the mixing and scratching of CDs
inf luenced Pioneer to develop a similar device to manipulate DVDs – the DVJ.
The direct control the DVJ offered was (in terms of gesture) easy for the audi-
ence to follow, and so a sense of spectacle developed around DVJ performance.
With DJs and VJs performing in similar ways, closer collaboration was possible,
and new aesthetics emerged in scratch culture: an example being the music video
mash-up style honed by Eclectic Method.
Around the same time that DVJs emerged, computing technology had devel-
oped to the level that laptops (loaded with VJ software) could manipulate graph-
ics and short video clips. Armed with a vision mixer and two laptops, a VJ could
attempt to emulate the ‘scratch’ performance style associated with DVJ decks.
However, this was a poor substitute as it lacked the spectacle of groups such as
Eclectic Method: laptop VJs were more suited to a different context – that of the
nightclub.
Toward the end of the 20th century rave culture was moving into the main-
stream, with city-based nightclubs the venue of choice. DJs lack the stage pres-
ence that musicians bring, so many nightclubs feature no stage at all. Instead,
there was an effort to embellish the experience with visuals, so projections
became an integral part of the rave aesthetic. Laptop-based VJs, with their muted
performance style and agile setup, were better suited to the context of the night-
club than scratch video performers.
Initially, the laptop VJ’s method of control was the mouse: an interface was
needed to give them more control and tactile feedback. The early 2000s saw the
development of hardware on top of the Musical Instrument Digital Interface
(MIDI) protocol: devices containing pots, knobs and keys that transmitted MIDI
information to the laptop’s software.15 By mapping the MIDI device’s elements
to software-based parameters, the performer could rely less on the mouse/screen
interface and more on the tactile controls in front of them.
As both DJs and VJs took to using MIDI controllers, the potential for expres-
sion through gesture returned, with some performers moving beyond the con-
fines of the nightclub to galleries, venues and festival stages. The DJ’s exaggerated
posturing became the norm during such performances, as it was a way of signal-
ling to the audience that they were manipulating the mix. In a similar effort to
make the connection between gesture and visuals as clear as possible, VJs tended
to change only the most obvious visual attributes such as scale and colour.
Recently, body-worn sensors are being used as a remote-control interface
between performers and computers: dancers have their movements mapped
202 Léon McCarthy

to the diffusion of spatial sound and the placement of the rendered visuals, an
approach used in multimedia performances by groups such as Cirque du Soleil.
Such endeavours require bespoke software and interfaces, so artists often work
alongside programmers; such was the case with the dance-controlled piece enti-
tled Virtual VJ: a collaboration between two co-editors of this volume, Steve
Gibson and Stefan Arisona.16

Diffusion Technologies
Wagner proposed the idea of a ‘total work of art,’ or Gesamtkunstwerk, a perfor-
mance in which the artist would express a unified vision across all media. In
19th century Germany, the media available to Wagner were dramaturgy, music
and set design. In the early 20th century, light-projection technology ushered in
the era of the ‘silent movie.’ Such cinematic technology then facilitated experi-
mentation with light projection at the Bauhaus, where the idea of the Gesamt-
kunstwerk inf luenced both theory and practice. Under the leadership of Walter
Gropius, there was an effort to unify intent across art forms, creating a stronger
aesthetic experience for the receiver of the artwork. The vision of L. Moholy-
Nagy sums up the intent of those practicing at the Bauhaus:

It is time to produce a kind of stage activity, which will no longer permit

the masses to be silent spectators, which will . . . allow them to fuse with
the action on the stage.17

One of the earliest forms of technologically enabled live projection appeared in

K. Capek’s performance entitled R.U.R: it incorporated mechanically operated
screening devices that opened to reveal a cinema with its own projected movie
within as a framing device for the dancers in the foreground.
The volumetric nature of projected light in the cinema means that one can
see the medium as it travels through the dust on its way to the cinema screen.
When an audience occupies the same volume as the projected light, their move-
ments may catch the light, and so they can become part of the resultant spectacle:
a manifestation that audiences tend to shirk from in gallery environments, but
one that in the live context they are more comfortable with. Perhaps it is for
this reason that live light projection brings a social element to the live aesthetic:
something different to the more solitary nature of watching the glowing surface
of the TV or cinema screen.
Cathode ray technology underpinned the development of the TV, which
then found its way into the armoury of the Fluxus artists, both as a site for
critique and as a technology itself. When sampling technology enabled 1980s
video artists to record TV broadcasts and then reappropriate that content as
Live Visuals, the display technology often became walls of TVs: we see this in
the Live Visual show that Emergency Broadcast Network (EBN) developed for
U2’s ZooTV tour.
 Live Visuals 203

The aesthetic qualities of a TV wall never suited the low-ceiling environment

of most nightclubs, so the projector continued as the technology of choice for
VJs. Flat-screen technology did not have a great impact on the practice of the
Live Visual artist, as they were heavy, fragile and expensive (particularly at the
large sizes required for projection in public venues). Nevertheless, the demands
of touring musical acts pushed the envelope, and we saw a hybrid version of the
liquid crystal display (LCD) f lat screen emerge in the large-scale light-emitting
diode (LED) wall. This technology was initially pushed into the live arena by
collectives such as United Visual Artists (UVA) in the form of multimedia stage
sets for musicians like Massive Attack.
Sharkstooth scrim is a material widely used in theatrical set design as a trans-
parent surface that can both ref lect and transmit light: this means it can be placed
to the front of the proscenium to ref lect projections and lights, yet still leave the
stage (actors and set) visible to the audience. Visual artists seeking a subtler pro-
jection surface can also use sharkstooth scrim: a development that adds a layer of
richness to The Light Surgeons’ SuperEverything* performances (see Figure 8.3).18
Projecting onto irregular surfaces is now possible with projection mapping,
where multiple high-lumen projectors are used in tandem with advanced mask-
ing techniques (see Chapter 14 for more on projection mapping and architecture).
It is most often experienced as an outdoor spectacle with pre-rendered projec-
tions used to highlight urban buildings. On some occasions, projection mapping
has been used on stage as part of a live performance, as was the case with Amon
Tobin’s ISAM series of performances. ISAM consisted of a custom cuboid-like

FIGURE 8.3 The Light Surgeons, SuperEverything*, a live cinema performance

projecting onto and through sharkstooth scrim. Used by permission.
204 Léon McCarthy

stage set construction within which Tobin’s performance booth was housed.
These cubes were illuminated with visuals animated in sync with Tobin’s perfor-
mance: a sensory assault recalling Wagner’s vision of a Gesamtkunstwerk.

Contemporary Developments
How will the next wave of technology impact on the aesthetics of Live Visual
practice? What will the next wave of Live Visuals artists perceive as the potential
in their contemporary technologies? Some of the technologies that are currently
shaping the commercial world may soon have an impact on the aesthetic of Live
Visuals.
Data visualisation is a technique used to reveal trends in information mined
from data. While the world of commerce tends to use data visualisation to gain
insights, artists could explore the information that underpins our digital world
through forms of visual narrative. An urge on the part of contemporary visual
artists to comment on the nature of our media-saturated society may drive them
to then consider data visualisation as part of the Live Visual aesthetic. An exam-
ple of what might emerge can be seen in Small Global: a series of installations
and performances by D-Fuse that visualised data (albeit off line information) on
global economic activity.
The broad take-up of social media, exemplified by platforms like Facebook
and Instagram, has come to impact not only on the way we order our lives but
also on artistic concerns. Just as television inf luenced the concerns of Fluxus art-
ists and the World Wide Web inf luenced the emergence of net.art, we now see
artists ref lecting on the nature of social media as a mainstream paradigm. While
artists initially explored the rich stories that could be visualised about people and
their connections to each other, that early fascination has moved onto a concern
with the insidious nature of social media, as apparent in the popular Netf lix
series Black Mirror. In terms of audio-visual performance, a critique may emerge
either through the content performers produce, the meta-narratives they sug-
gest or the reappropriation of social networks as devices supporting performance
itself.
Social networks facilitate the establishment of specific virtual communities:
this can bring benefits to members but leaves them open to new forms of propa-
ganda (advertising, political or otherwise). Artists can claim some of this terri-
tory and communicate with specific cohorts through social media and network
technologies. As mentioned earlier, Richie Hawtin has used Twitter to deliver
enriched content to the audience’s phones through second-screening. This is an
activity relatively new to both the commercial and art worlds, but as more artists
explore the potential of the second screen, we can expect to see it impact the
Live Visual aesthetic. Perhaps the second screen could facilitate subsets within
an audience to collaborate in how they consume, suggest and use such enriched
content. A richer form of social interaction could emerge, one mixing the physi-
cal, the tactile and the virtual.
 Live Visuals 205

Just as social media can be mined to decipher sentiment, second-screen com-

ments from an audience can be mined by passing the textual data through natu-
ral language processing (NLP) algorithms. In my own work I have used the
Python programming language to automatically mine meaning from audience
comments. My interest is in now exploring the aesthetic that would emerge were
I to fully automate the process of mining, aggregating and selecting audio-visual
content. If I were to perform alongside a content-aware social media bot,19 what
role would I then play?
The previous discussion about interface touched on the challenge that the
laptop performer encounters when seeking to relate their screen-based actions to
the audience. The previous section outlined an alternative in which, rather than
have the audience seek the mappings between software and the diffused audio-
visuals, they are instead given that control. An alternative could be to reveal the
underlying code that is rendering the audio-visuals. This is already emerging in
the practice of live coding, with the term algorave being used to describe such
events (see Figure 8.4). The performer’s real-time manipulation of computer
code changes the sonic output, with the live code presented on screen in a way
that frames the on-stage performers.
While live coding practice established itself as a novel means to generate
sound, we are now seeing live coders generate both the visual and the sonic.
Shawn Lawson is an example of a performer using a customised web browser

FIGURE 8.4 B. Swift and G. Muzio performing Hyperkosmo, an algorave performance,

at ICLI Conference. © 2020 Robin Parmar. Used by permission.
Source: www.f lickr.com/photos/rparmar/49565348691
206 Léon McCarthy

which, when seen on the main screen, displays both his code and the visuals
being rendered in real time. As with sound generated through live coding, high
fidelity and temporal development of structure are (necessarily) sacrificed for
the extreme ‘liveness’ that both performer and audience perceive. While art-
ists such as Lawson develop their own programming platform, there are now
fully f ledged live-coding platforms (such as Cyril), and as more emerge a larger
community will develop. The aesthetic of live coding may yet impact on audio-
visual aesthetics at large.

What Next?
To conclude, let us return to the aesthetics of Live Visual practice; there has
been the ‘post-conceptual’ wallpaper movement, and now the ‘pseudo-modern’
in which the intent of the artist shapes the framework upon which both they
and the audience interact to realise an aesthetic. This juncture has come about
through the interaction of artists with new technologies, and we can expect
technologies to further embolden artists as they seek alternative connections
with the audience. Artists may choose to collaborate with the audience, as it is
by enabling the audience that the performer can encourage them to interrogate
meta-narrative on their terms.
What will the next aesthetic forms be? Will narrative come to dominate: an
aesthetic direction already apparent in the emerging practice of live cinema?
Will it be a concern for spontaneity, be that via collaboration, virtual networks
or media bots? Will it be a grappling with the control interface: the perennial
challenge to any artist that has come to rely on the evolving MIDI interface?
Whatever is to come, it will emerge from a critical mass of artists reappropriating
alternative technologies in similar ways. As is outlined in Chapter 13, augmented
reality (AR) and virtual reality (VR) are all finding their place in the workf lows
of cutting-edge Live Visuals artists. The Live Visual aesthetic of the future is
sure to be mediated by the technologies used by the next wave of Live Visual
performers.

Notes
1 James J. Gibson, The Ecological Approach to Visual Perception (New York: Psychology
Press, 2013).
2 John M. Culkin, “A Schoolman’s Guide to Marshall McLuhan,” Saturday Review, March
18, 1967, 51–53.
3 Nam June Paik and Charlotte Moorman, TV Cello (New York, 1971).
4 Matt Black, VJAMM (Cambridge: Camart, 1997).
5 Sergei Eisenstein and Jay Leyda, The Film Sense (London: Faber and Faber, 1986).
6 Marcel Duchamp, The Creative Act (Houston, TX: Convention of the American Fed-
eration of Arts, 1957).
7 Umberto Eco, A Theory of Semiotics (Bloomington: Indiana University Press, 1978).
8 Roland Barthes, “The Death of the Author,” in Image Music Text (London: Fontana
Press, 1993).
 Live Visuals 207

9 Teresa Cerratto-Pargman, Chiara Rossitto, and Louise Barkhuus, “Understanding

Audience Participation in an Interactive Theater Performance,” in Proceedings of the 8th
Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational (NordiCHI ’14)
(New York: Association for Computing Machinery, 2014), 608–617. DOI: https://doi.
org/10.1145/2639189.2641213.
608–617.
10 Alan Kirby, “The Death of Postmodernism and Beyond,” Philosophy Now, (58), 2013, 2.
11 Alan Kirby, “The Death of Postmodernism and Beyond,” Philosophy Now, (58), 2013, 6.
12 Alan Kirby, “The Death of Postmodernism and Beyond,” Philosophy Now, (58), 2013, 3.
13 Léon McCarthy, “Betav10,” A Live Cinema Performance, 2015, [Online], https://vimeo.
com/album/2619225/video/127494372 (Accessed November 15, 2020).
14 Tyler Freeman, “Outsourcing the VJ: Collaborative Visuals Using the Audience’s Smart-
phones,” in Lanfranco Aceti, Steve Gibson, and Stefan Müller-Arisona, co-eds., Live
Visuals for Performance, Gaming, Installation, and Electronic Environments (San Francisco:
Leonardo Electronic Almanac, 2013), https://www.leoalmanac.org/vol19-no3-
outsourcing-the-vj/.
15 See Chapter 12 for more on MIDI technology.
16 Steve Gibson, in Lanfranco Aceti, Steve Gibson, and Stefan Müller-Arisona, co-eds.,
Live Visuals for Performance, Gaming, Installation, and Electronic Environments (San Fran-
cisco: Leonardo Electronic Almanac, 2013), 214–229, www.leoalmanac.org/vol19-
no3-simulating-synesthesia/.
17 Roselee Goldberg, Performance: Live Art, 1909 to the Present (New York: H. N. Abrams,
1979).
18 See Chapter 12 for a detailed discussion, as well as Chapter 17 for an interview with
Chris Allen of The Light Surgeons.
19 One like Apple’s Siri but tailored to the context of audio-visual performance.
9
AVUIS
Audio-Visual User Interfaces: Working With
Users to Create Performance Technologies

Nuno N. Correia and Atau Tanaka

Introduction
The evolution of live visual performance has paralleled the evolution of live
electronic music. Artists worked directly with physical materials in the 1960s:
magnetic tape for audio and coloured gels and oils in psychedelic light shows.
The 1980s brought a focus on devices, with musicians performing on hard-
ware synthesizers, while early VJs worked with videocassette decks and analogue
video synthesizers (as described in Chapters 3 and 4). The laptop then became a
prominent element in both electronic music and live visual performances from
the 1990s onward. This stripped away the material specificities and instrumental
affordances to performing both music and visuals on a general-purpose device.
While the generic nature of the computer took away idiomaticity, a common
platform uniting sonic and visual media has created opportunities for tighter
coupling between the ocular and the aural. Without device or material speci-
ficities, the interface has become the determining factor in the affordances of a
computer-based audio-visual performance system.
A common criticism of computer music performances is the lack of visual
feedback for audiences. This situation has led audiences to view the laptop’s
use as a musical instrument “a violation of the codes of musical performance”1
due to a feeling of a lack of authenticity. This has created “a rift between the
performer and the audience.” 2 A strategy to compensate for this rift has been
the use of Live Visuals in electronic music performances. 3 This can be classi-
fied as audio-visual (AV) performance, a practice exploring “an interconnec-
tion between sound and image, which sometimes becomes apparent and at
other times remains intuitive.”4 AV performance can help solve the issue of the
lack of visual feedback in electronic music performance or can be thought of
as a holistic total artwork (Gesamtkunstwerk) unifying multiple media. Creat-
ing successful interfaces consists not just of considering the constituent media
DOI: 10.4324/9781003282396-12
 AVUIs 209

but also of the intended mode of performance and the needs of the per-
former or user.
We identify three broad types of user interface connecting sight and sound
in a performance system: 1) live coding, where the command-line interface is
shown to an audience; 2) tangible interfaces, where physical objects are used as
the interface and often augmented with visualisation; and 3) Audio-Visual User
Interfaces (AVUIs) that extend the concept of the graphical user interface (GUI)
into creative applications. We developed the concept and practice of AVUIs as a
functional way to create tighter coupling between music and visual, with neither
being subservient to the other in terms of performance. The systematic pairing
of media through a performative interface made the coupling operational, and
in this way took the association of visual and sonic media beyond visualisa-
tion or sonification, opening up a potential for ease of use and access.5 Here,
we will assess strengths and weaknesses of AVUIs compared to other interface
approaches, identify successful methods for designing AVUIs and suggest a user-
centric design process that engages with performers.
We will start by presenting background on the three categories of interface
approaches identified. We then focus on the last one and present our project,
Enabling AVUIs,6 first in terms of design and then in an audience evaluation.
From these, we draw conclusions to map the territory of AVUIs, in dialogue
with related concepts, and discuss best practices for AVUIs, including fostering
audience understanding.

Background
Artists have used a variety of approaches for real-time audio-visual art in elec-
tronic music performances. Ribas7 has created a taxonomy that classifies these
different systems, taking into account the interactive aspect:

• Audio-visual entities, assuming “distinct procedural behaviours and

responses to interaction”: pieces composed of distinct individual ele-
ments, “mostly graphic shapes or moving pixels,” which have “associated
sound excerpts or loops, either to graphic forms or to an overall visual
configuration.”
• Interactive sounding shapes, where specific audio-visual elements are
not necessarily created through the user’s interaction, “but rather chosen,
selected, altered, added or activated – reconfigured within the possibilities
given by an existing repertoire devised within the system.”
• Sounding figurations, consisting of visual elements that can be drawn or
created by “screen-based and mouse-operated systems” and “whose proper-
ties are mapped to the production of synthetic sounds.” In this category,
nothing happens without human interaction, since “it is exactly human
expression that the system is devised to integrate and express as its subject
matter,” producing “consistent responses to user input.”
210 Nuno N. Correia and Atau Tanaka

• Audio-visual reactions to interactions, where “changes to the audio-

visual surface are a response to the participants’ combined actions,” often
gestural. Reactions to the behaviour are indeterminable: “there can be no
linear correspondence” between an interaction and an audio-visual reaction
“due to the f luctuating nature of the input data.”

Meanwhile, in the field of human-computer interaction (HCI) research, Preece

et al.8 identify 20 main types of human-computer interfaces that have emerged
since the personal computer revolution of the 1980s. From these we highlight
three that have had a relevant adoption in terms of audio-visual performance:
command-line interfaces (CLI), tangible user interfaces (TUI) and GUIs. These
are not exhaustive, as there are a multitude of hybrid approaches. We then draw
upon Ribas’s analysis of electronic music to apply these three archetypal HCI
modes of interaction to fit the needs of AV performers.

Visualised Live Coding

Live coding is a practice “where source code is edited and interpreted in order
to modify and control a running process.”9 It is therefore situated in the broader

FIGURE 9.1 Slub performing live at the Roebuck pub in London. 2009. Photo by
Philippa. Licensed under the Creative Commons Attribution-ShareAlike
2.0 Generic license.
Source: https://commons.wikimedia.org/wiki/File:Slub_live_Roebuck.jpg
 AVUIs 211

type of CLIs. Live coding in music emerged from searching for new ways of
expression in computer music, outside of the “rigid interfaces of performance
software like Ableton Live or Reason.”10 Functional programming is considered
the most common paradigm in live coding.11 Live coding artists aim to “explore
the interfacing of the language itself as a novel performance tool, in stark contrast
to pretty but conventional user interfaces,” or in other words, to explore “the
act of programming as an expressive force for music closer to the potential of
the machine.”12 An early example is the work of Slub (Figure 9.1): “Slub often
project their laptop screens to give some impression of the processes in motion
that form the music. This allows the audience to see the live quality of the code
as well as hear it.”13
Concerns with the perception of code by audiences have been an important
driver in the aspect of visualising live code. There is a risk that “audience members
may feel distracted, or perhaps even excluded by the projection of code written
in language they do not necessarily understand.”14 In their article “Visualisation
of Live Code” McLean et al. report on a number of works exploring “ways of
visualising code development that allows non-programmers to enhance their
enjoyment and understanding of a live coded piece.”15 Magnusson also presents
examples of visualised live coding works as “graphical representation of algo-
rithmic music scores.”16 Purcell et al. evaluated the contributions to the audience
experience of visualisations of two live coding music systems, with emphasis on

FIGURE 9.2 Thor Magnusson, Threnoscope screenshot. @2021 by Thor Magnusson.

Used by permission.
212 Nuno N. Correia and Atau Tanaka

understanding and enjoyment.17 They entitled these systems “didactic,” focus-

ing more on informative aspects of visualisation, and “aesthetic,” focusing more
on aesthetic appeal. They concluded that the “overall effect of visualisations on
enjoyment was high for both the aesthetic and didactic visualisations,” with
more inconclusive results regarding understanding.18
Threnoscope is an example of visualised live coding by Magnusson that com-
bines live coding, graphical score and GUI, aiming “to create a helpful graphical
representation of the sonic texture in microtonal drone music”19 (Figure 9.2).
In a performance setting, the graphical score and the live coding window are
displayed side by side. However, live coding is the main form of interaction with
the system, not the GUI: “the representational score in the Threnoscope can be
considered to be one large graphical user interface controller, although it serves
mostly as a representation of the state of the music.”20 Threnoscope demonstrates
that user interfaces do not always fit within rigid classifications and that there are
works that push the boundaries between those.
The live coding community develops tools for creating Live Visuals through
real-time coding, often in combination with live coding for music. Hydra 21 is a
“browser-based platform for live coding visuals, in which each window of the
browser can be used as a node of a modular and distributed video synthesizer.”22
Hydra takes an approach that is inspired by analogue video synthesizers from the
1970s “in which visuals are created by routing, transforming, and recombin-
ing multiple sources and outputs in real time.”23 Fragment is an audio-visual live
coding environment based on spectral synthesis. It follows a pixel-based, real-
time, image-synth approach to sound synthesis: “the sound synthesis is powered
by pixels data produced on the graphics card by live GLSL code.”24 It consists
of a “canvas” panel of visuals and a code editor panel, which allows the user to
“generate the visuals which are fed to the audio synthesis engine.”25 This prac-
tice of ‘live coded visuals’ is different from the ‘visualised live coding’ presented
in McLean’s work.26 In ‘live coded visuals,’ the main output of the code are the
graphics. In ‘visualised live coding,’ the visuals represent the code, or the music
generated by the code.

Augmented Tangible User Interfaces

To address the two problems identified with laptop performances by Patten – the
lack of visibility by the audience of the performer’s actions and the lack of physi-
cality, modularity and expressivity of interfaces27 – artists and designers have
developed TUIs for music, with visual augmentations.
Audiopad is a tabletop tangible interface for musical performance that aims to
“combine the modularity of knob-based controllers with the expressive charac-
ter of multidimensional tracking interfaces.”28 Audiopad uses electromagnetically
tracked physical objects as input devices. Visual information is projected to the
tabletop surface from above, so that “information corresponding to a particular
physical object on the table appears directly on and around the object.”29 The
 AVUIs 213

system provides physical form to the digital parameters of a synthesizer and also
visual feedback about the synthesis process. According to Patten et al., the pair-
ing of physical input and graphical output “can yield a musical interface that has
great f lexibility and expressive control.”30 Its creators focused on the “legibility”
of the interaction, that is: “onlookers should be able to understand how users
were interactive with the system.”31
Reactable follows a similar approach to Audiopad of a tabletop tangible inter-
face composed of physical objects and corresponding visual augmentations.
According to Patten et al., “perhaps the most important difference between
the Reactable and the Audiopad is that the Reactable uses modular synthesis,
while the Audiopad uses loop-based synthesis.”32 This leads to different chal-
lenges for the interface designer and the performer. Reactable has different
levels of dynamic visual feedback: “auras around the physical objects bring
information about their behaviour, their parameters values and configuration
states, while the lines that draw the connections between the objects, convey
the real waveforms of the sound f low.”33 The visualisation of the sound f low
is an important aspect of Reactable due to its modular nature. In terms of
the visual feedback, some of the guiding principles were: “avoid any type of
textual or numerical information, while banishing at the same time any dec-
orative display”34 (Figure 9.3). In other words, graphics should be simple, rel-
evant and informational. In performances, the manipulation of the Reactable

FIGURE 9.3 The Reactable. Photo by Daniel Williams. 2007. Licensed under the
Creative Commons Attribution-Share Alike 2.0 Generic license.
Source: https://commons.wikimedia.org/wiki/File:Reactable_Multitouch.jpg
214 Nuno N. Correia and Atau Tanaka

has often been displayed to the audience by projecting the video feed of an
overhead camera.
Soft Revolvers by Myriam Bleau consists of acrylic spinning tops with light-
emitting diode (LED) lights, each mapped to sound. Each spinning top “controls
musical algorithms in Pure Data through their internally built gyroscope and
accelerometers.”35 The LED lights are synced to the sound being mapped to, and
manipulated by, each individual top. A camera captures Bleau’s performance,
and the live video is projected behind the artist, giving the audience a closer
look at the lighted spinning tops. The design of the tops was “intentionally built
to enhance audience experience with the music and performance aesthetics.”36
According to Bleau:

Transparency in the interactions is really important for me, I like to under-

stand what’s going on when I see a performance (. . .). The light patterns
are generally linked to the sounds that each top plays, making it easier for
the audience to understand which top is playing and how I compose the
music live.37

Toward AVUIs
From the previous approaches, different requirements for interfaces for AV per-
formance emerge. Managing the ambition of the project’s complexity to match
the technical limitations of the hardware/software environment creates a more
cohesive, ‘instrumental’ result. Carrying on the musical instrument metaphor,
enabling physical affordances in the interaction makes for a compelling perfor-
mance. Combining and recombining elements of the system in a modular way
enables a number of different works to emerge from an instrument, with per-
haps each piece exploring or highlighting different qualities or components of
the system. The legibility of performance can be aided by exposing, or making
visible, parts of the system, the compositional and performative processes. This
helps demystify what for the audience might be an incomprehensibly complex
series of relations between performer action, sound and visual output. Creating
this legibility of system and performance result in, we argue, a more expressive
instrumental system.
We propose the following qualities of a more legible and expressive AV
system:

1 Get closer to the potential of the computer38

2 Physicality of interfaces39
3 Modularity40
4 Visualise the live quality of the process41
5 Visualisation leading to the understanding of the piece by the audience42
6 Visibility by the audience of the performer’s actions43
7 Expressive character of interfaces44
 AVUIs 215

Some of these requirements are more centred on the performer (1–3), while oth-
ers are more audience-oriented (4–6). Expressiveness of interfaces (7) relates to
both performer and audience. Explicitly or implicitly, a common concern is to
bridge an identified divide between electronic AV practitioners and their audi-
ences. This list is not meant to be met entirely by a single approach, but to inform
the discussion around interfaces for performance. Due to the diverse origin of
these requirements, it will be nearly impossible to meet all with a single interface
type. For example, physicality of interfaces can be challenging to implement
with live coding. Therefore, thinking about context of use and about the user,
whether they be musician or visual artist, performer or audience, is primordial.
This creates the need for design methods that focus on the user.
There have been alternative approaches to live coding and tangible interfaces
that followed the requirements noted earlier, with less emphasis on code and on
physicality of interfaces. In live coding, there is an identified risk of distraction
or feeling of exclusion by the audience due to a lack of understanding of the
code45 – which many live coding artists have aimed to address. Tangible inter-
faces afford the immediacy of physicality but are not always viable or scalable in
some scenarios, such as audience participation or online performance. Specialised
tangible interfaces are not universally available, which “is not a minor feature
when considering the design of popular and ‘democratic’ new music interfaces.”46
Therefore, we can introduce a new criterium to the aforementioned list, that of:

8 Democratic interfaces47

Democratic aspects may include legibility and accessibility, resulting in systems

adapting GUIs to incorporate sound visualisation. To describe these approaches,
we propose the concept of AVUI, where “the interaction of sound and image in
the interface extends the concept of GUIs.”48 Here we will show how AVUIs fulfil
the criteria of modularity, visibility, expressiveness and democracy from the list.

Precursors of AVUIs
We begin by presenting related works that prefigure the AVUI concept: FMOL
and the work of ixi software. Another precursor, the work of Correia on Inter-
active Audio-Visual Objects (IAVOs), will be presented in the following sec-
tion. FMOL is an earlier project of Reactable’s Jordà, developed in 1997 as “an
Internet-based music composition system that could allow cybercomposers to
participate in the creation of the music for (. . .) F@ust 3.0,”49 a show by Catalan
theatre group La Fura dels Baus. In its default “rest” setting, FMOL looks like
a 6 × 6 grid. Each of the six vertical lines is associated with one sound genera-
tor (synthesis engine or sample players), while the horizontal lines are associ-
ated with processors (filters, reverbs, etc.). These lines function “both as input
devices (controllers) that can be picked and dragged with the mouse, and as
output devices that give dynamic visual and ‘sonic’ feedback.”50 The motivation
216 Nuno N. Correia and Atau Tanaka

for the project was “to construct an abstract visualisation so tightly related to the
synthesis engine architecture, that almost every feature of the synthesizer would
be ref lected in a symbolic, dynamic and non-technical way in the interface.”51
FMOL later inspired the development of Reactable, mentioned earlier.
In the early 2000s, the collective ixi software (ixi) developed screen-based
instruments aiming to “represent sound or sound processing with graphical
objects placed in a two-dimensional space on the Screen.”52 With these screen-
based instruments, ixi aimed to “bridge the gap between physical instruments
and controllers and the abstract programming environments which people use
in their work.”53 An example is SpinDrum, an instrument consisting of rotat-
ing wheels, triggering a sound file when they reach a certain position. The
size of the wheels represents the amplitude of associated sounds, while the
vertical and horizontal placement represent pitch and panning, respectively.
The user can manipulate speed of rotation and add, move or delete wheels.54
ixi also developed ixiQuarks “a graphical user interface (GUI) environment
of audio tools and instruments for live improvisation that allow for user inter-
action on both the GUI and the code level.”55 An example from ixiQuarks is
GrainBox, a “two-dimensional parameter space for granular synthesis,” where
“boxes with related parameters are connected with lines.”56 With these tools,
ixi aim to “represent musical patterns visually and create intuitive spontane-
ous instruments in forms of graphical user interfaces that allow for quick reac-
tions in a live situation,” something that software such as SuperCollider and
Pure Data lack.57

The Enabling AVUIs Project

We have reported on the Enabling AVUIs project in previous articles. In “User-
Centered Design of a Tool for Interactive Computer-Generated Audiovisu-
als,”58 we describe in detail the initial interviews and ideation process involving
a brainstorming and sketching workshop. In that paper, we introduce a novel
brainstorming method, Re-boot, further presented in Stage 2 later. In “Proto-
typing Audiovisual Performance Tools: A Hackathon Approach,”59 we focused
on hackathons as a method for rapid prototyping systems for AV performance
based on our experience with the first AVUI hackathon. In “AVUI: Designing a
Toolkit for Audiovisual Interfaces,”60 we presented the user-centric design pro-
cess, from scoping interviews to design, development and deployment of AVUIs,
focusing on the iterative evaluation by users at each stage. This resulted in the
development of audio-visual systems, an AV toolkit (ofxAVUI) and AVUI guide-
lines. In “The Role of Live Visuals in Audience Understanding of Electronic
Music Performances,”61 we reported on the audience studies of the two perfor-
mances, presenting 10 audio-visual projects. We distributed a questionnaire to
the audience, containing Likert-scale and open-ended questions. Seventy-nine
audience members filled in the questionnaire. The paper focused on the analysis
of the question “Did you find the connection between the performer’s actions
 AVUIs 217

and the audio-visual result understandable?” In “From GUI to AVUI: Situating

Audiovisual User Interfaces Within Human-Computer Interaction and Related
Fields,”62 we discuss the concept of AVUI within the field of HCI, particularly
relating it to GUI. Here we take a broader perspective: we situate the project
within other related approaches to interface in AV performance, particularly
where audience understanding is relevant. We also describe in detail a user-
centric approach to map out the design space of AVUIs and discuss the meth-
odological implications of working with audio-visual practitioners in creating
technologies for artistic performance. Here we also provide a synthetic overview
of the entire two-year project.

Interactive Audio Visual Objects

Between 2007 and 2010, Correia developed three audio-visual projects combin-
ing GUI, sound and visualisation, using a modular object-based approach, lead-
ing to the concept of IAVOs. IAVOs are “modules that form a cohesive whole,
where the GUI is embedded and aesthetically integrated in the visualisations.”63
IAVOs allow users to manipulate and visualise individual audio stems, following
“a similar logic to ‘tracks’ or ‘mixer strips’ in digital audio workstations.”64
In the first of these projects AVOL (2007), the user interface (UI) was com-
posed of mute, solo, volume, track change and randomise functions.65 AVOL
allows the sequencing of a total of 28 pre-composed loops. It was an initial proof
of concept of the IAVO approach. In the last of the IAVO projects, AV Clash
(2010), the diversity of sounds and manipulation options are vastly increased.
AV Clash contains four IAVOs, each representing a different ‘tag’ from the
online sound database Freesound.66 Each IAVO in the project contains a GUI
with options for selecting Freesound files within that tag, trimming the sound,
sequencing sounds, volume control, sound selection, audio effect selection, effect
volume, tag selection, stop, solo and visualisation selection. IAVOs in AV Clash
can be dragged and thrown around the screen, creating further audio-visual
effects when clashing with each other.67
Although IAVOs are composed of individual objects, they are meant to be
harmonious: the combination of the different IAVOs “aims to create a coherent
whole, where the manipulation options afforded by the GUI allow for numerous
variations of the audiovisual content.”68 The concept of IAVOs overlaps with
the category of “audiovisual entities” in Ribas’s taxonomy of real-time audio-
visual art systems presented earlier: “pieces composed of distinct individual ele-
ments . . . mostly graphic shapes or moving pixels, and have associated sound
excerpts.”69 IAVO projects can be considered precursors of our AVUI work.

User-Centred Design Approach in Enabling AVUIs

The Enabling AVUIs project took place over a two-year period and can be
divided into eight stages:
218 Nuno N. Correia and Atau Tanaka

1 Scoping interviews with performers

2 Brainstorming and sketching workshop
3 Two hackathons for AVUI prototype development, with audio-visual
performers
4 Two performances with AVUI prototypes and respective audience studies
5 Evaluation of prototypes by other audio-visual performers
6 A ‘consolidation prototype’ gathering best practices and expert evaluation
7 Creation of an AVUI development toolkit, released as open source
8 Toolkit evaluation in a final hackathon and with online users

We followed a user-centred design (UCD) approach, a “design processes in

which end-users inf luence how a design takes shape.” 70 UCD is typically com-
posed of four core processes: 1) understanding context of use, 2) specifying user
requirements, 3) producing design solutions and 4) evaluating against require-
ments71 (see Figure 9.4). These can be roughly mapped to our four first stages.
Hackathons and interviews were used as methods throughout the studies with
artists. The interviews were analysed using thematic analysis.72 Questionnaires
were used for the audience studies. The iterative cycle using hackathons enabled
“a dynamic process of invention, distributed across events.” 73

Stage 1: Scoping Interviews With Performers

In an initial stage, we conducted interviews with 12 audio-visual performers (11
male, 1 female), asking them about their practice, their tools and their needs and
desires as performers. The performers had between 4 and 18 years of performance

FIGURE 9.4 UCD core processes.74

 AVUIs 219

experience. Three of the interviewees were involved in the organization of AV

events in London. Another three were academic researchers in the field of sound
and audio-visuals. Four were full-time media artists and audio-visual perform-
ers. And two were younger practitioners, combining their AV practice with
studies or other work. This group represents a wide range of approaches, such as
audio-visuals with analogue equipment, live coding, do-it-yourself approaches
with live camera feeds, live cinema and generative audio-visuals with custom-
made software.
The analysis of the interviews brought forth a series of key requirements:
modularity, f lexibility and reconfigurability; ease of hardware/software inte-
gration; instrument-like expressivity and f luidity; integration of environmental
elements; generative capabilities and diversity; communication of process to the
audience; and reliability and speed.75
The 12 interviewees provided us with a group of experts. These experts all
have extensive experience as AV performers, with diverse backgrounds (academ-
ics, event organisers, full-time and part-time performers) and artistic approaches.
We would consult these experts throughout the different peer evaluations of the
research – the evaluators of stages 5 and 6 were from this same group. In our Dis-
cussion section, under “User-Centered Design Process in AVUIs,” we further
elaborate on the role of these experts in our research.

Stage 2: Brainstorming and Sketching: Reboot

The requirements gathered from the interviews were used as a starting point
for a sketching and brainstorming workshop at Goldsmiths, University of Lon-
don, with 19 participants (including 2 from the previous interview stage). The
participants (12 male and 7 female) were divided into five groups. The one-day
workshop structure consisted of two half-day parts. The first part adopted the
“bootlegging” idea generation technique.76 Bootlegging is a “structured brain-
storming technique particularly suited to multidisciplinary settings.” 77 It aims
to mix familiar concepts in a way that stimulates creativity. We then extended
the bootlegging method with a focused, structured process that revisited and re-
examined the concepts from the first round. We call this new method “reboot.” 78
Instead of relying on aleatoric mixing and combining variations, in reboot there
is a deliberate planning of the variants, following a similar structure to “boot-
legging”: “after having stimulated the creation of new application ideas with
the bootlegging session, reboot allows the participants to concentrate on more
specific solutions.” 79
The combination of bootlegging followed by reboot resulted in five con-
cepts for future AV performance technologies. Workshop participants materi-
alised these concepts through storyboards and annotated sketches. Of the five,
two were considered by us particularly successful, as they met several of the
requirements set out earlier: Gestural Touchscreen and Meta/Vis. Gestural Touch-
screen is a touchscreen-based application controlled entirely by gestures. Users
220 Nuno N. Correia and Atau Tanaka

can only load vector graphics files as visual content, and there is a built-in physics
engine. Meta/Vis also relies on multitouch interaction but adds a configuration
panel. This panel adopts a data-f low paradigm, although substantially simplified.
Objects such as sound, visuals, control, generative and physics can be connected
and configured.
This process, combining bootlegging with reboot, generated ideas that we,
as designers, would not have conceived ourselves. Additionally, it enabled AV
performers, who have limited experience in interface development, to imag-
ine future systems. When discussing their needs and desires, this confirmed the
requirements set out in the previous stage. This reinforces the pertinence of our
UCD approach.

Stage 3: Two Hackathons for AVUI Prototype Development

The next step was to take the concepts generated in the earlier phase and to move
from storyboards and sketches to functioning technology. In order to do so, we
organised rapid prototyping sessions based on the hackathon tradition. Our aim
was to bring together our subjects to build prototypes informed by the earlier
requirements and sketches. We organised two hackathons with AV perform-
ers who responded to a call for participation where coding knowledge was a
prerequisite. Each hackathon took place over two days and followed a common
structure, in three stages: 1) introduction: a presentation on the previous stages
of the project and results achieved so far, goals and structure of the hackathon; 2)
conceptualisation and sketching; and 3) software development sprint.
The first hackathon had 23 participants (5 female and 18 male), while the
second had 13 (2 female and 11 male), 3 of whom had taken part in the first. We
divided participants into six groups, based on preferred programming languages
and development environments, distributing prior programming experience
evenly across groups. One of the groups from the first hackathon did not man-
age to finish their prototype, resulting in 11 prototypes.
The hackathons started with a presentation on the previous stages of the
study: interviews, workshop and also results of hackathon 1 in the second one.
The sketches produced in the reboot workshop were presented in detail, and
participants were invited to adopt features of the sketches into their projects.
Therefore, the earlier sketches acted more as inspiration rather than a strict plan
to follow. The structure for the hackathon was outlined. Each hackathon started
with the development of concepts and sketches, followed by software program-
ming. The group work was f luid in both hackathons, and all groups achieved the
desired outcomes (with the exception of one). The atmosphere was enthusiastic,
with a fervour of activity and an important networking element. Some of the
participants knew each other from the London AV community, while others
benefited from this opportunity to integrate further. The second hackathon was
more f luid due to less time spent on the introductory session (a lesson learned
from the first event).
 AVUIs 221

From Hackathon 1, five prototypes were developed, which we describe.

ABP is an animation engine and sound visualiser, where the user can define
colour, geometry and animation parameters. DrawSynth allows for controlling
sound and image – users can draw vector shapes and select colours, which are
sonified by a synthesis engine. Esoterion Universe consists of a 3D space that can
be filled with planet-like audio-visual objects, each with a GUI to modify
their visual and sonic properties. GS.avi is an instrument that generates con-
tinuous spatial visualisations and music from the gestural input of a performer.
Modulant allows for drawing images, using a paintbrush type of tool, which are
then sonified.
In Hackathon 2, six prototypes were created. Butterfly is an audio visual-
iser which allows for the control of four audio synthesizers by manipulating
icons distributed in four XY pads on the screen. Cantor Dust generates, dis-
plays and sonifies Cantor set type fractals as sound and visuals. EUG further
develops Esoterion Universe from hackathon 1, adding 3D gestural control with
a Leap motion sensor. OnTheTap plays with the tactile, analogue feel of tap-
ping surfaces as interaction input, captured as audio. residUUm allows for the
creation and manipulation of AV particles with a variable lifespan by clicking
and dragging on the screen. Wat creates a chaotic 3D texture based on cellular
automata.
All the projects were uploaded to GitHub as open-source code.80 The diversity
of the 11 projects demonstrates that the hackathons were a successful approach
to cover a large portion of the design space for AVUIs. These 11 projects cover
all the categories of Ribas’s taxonomy for audio-visual pieces,81 with at least two
examples per category. Butterfly, Esoterion Universe, EUG and residUUm can be
considered ‘audio-visual entities,’ as they are composed of individual elements,
with distinctive graphic shapes and associated sounds, and also a related user
interface. ABP, Cantor Dust and OnTheTap can be classified as ‘interactive sound-
ing shapes,’ as they are not intended to represent specific sounds, but consist of
generic graphics reacting to the overall sonic landscape, with the possibility of
reconfiguring both audio and visuals by means of an interface independent from
the visual output. DrawSynth and Modulant belong to the ‘sounding figurations’
category, as they consist of graphical elements that are drawn and mapped to
the production of synthetic sound. GS.avi and Wat can be considered part of
the ‘audio-visual reactions to interactions’ category, as changes to the audio-
visual surface are a response to the performer’s gestural actions, in a non-linear
correspondence.
The hackathon process allowed us to quickly develop 11 systems for AV per-
formance, across the design space for AVUIs, in a short amount of time. Some
of these systems incorporated surprising and interesting solutions, particularly in
terms of UI, that we would otherwise not have conceived ourselves. By involv-
ing developers who are themselves performers, and by connecting the develop-
ment to previous user-centric stages of the project, we ensured that the systems
that emerged are relevant to the AV artistic community.
222 Nuno N. Correia and Atau Tanaka

Stage 4: Performances With the AVUI Prototypes

and Audience Studies
Once we had functioning prototypes, we validated them in a real-world set-
ting, using a performance-led research in the wild approach.82 We organized
two public events, one per hackathon, where each project was presented in a
10-minute performance, followed by a short discussion by the authors (around
five minutes). The goal was to create a setting similar to that of a gig, but one
where we had some control over the environment and where we could invite the
audience for feedback in a form useful for the next steps in our research.
We distributed questionnaires to the audience, with questions targeting 10
projects (Figure 9.5), 4 from Hackathon 1 and 6 from Hackathon 2 (one of the 5
projects from the first hackathon, DrawSynth, was left out of the questionnaires
due to late completion and last-minute addition to the event). In the first per-
formance, 45 respondents answered the questionnaire (27 male, 12 female and 6
empty answers). The average age of audience members was 29; 69% had experi-
ence as a practitioner in audio and/or visuals: 27% had experience as a visual artist,
visual designer or VJ; 18% as musician or DJ; and 24% in both. The second perfor-
mance had 34 respondents (19 male, 14 female, 1 empty answer). The average age
was 33; 79% had experience as practitioners: 26% as a visual artist, visual designer
or VJ; 38% as a musician or DJ; and 15% in both. The event was promoted at
the location of the event, Goldsmiths, University of London, a university with a
strong arts and music tradition, and at the London Music and Video Hackspace
communities. Therefore, the profile of the audience fits with our expectations –
an audience familiar with diverse music and audio-visual performance practices.
The audience completed the questionnaires in the intervals between project
performances. The questionnaires consisted of three pairs of questions, repeated

FIGURE 9.5 Images from the 10 projects resulting from the hackathons, presented at
performances and evaluated by audiences. Left to right, top row: ABP,
Esoterion Universe, GS.AVI, Modulant, Butterfly; bottom row: Cantor
Dust, EUG, OnTheTap, residUUm, Wat.
 AVUIs 223

for each project. Each pair consisted of a 5-point Likert scale and an open-ended
question. Two of the pairs asked concerned variety/diversity of audio and visual
content and relatedness between both modalities:

1 Did you find that the audio-visuals were varied and diverse?
Complete the sentence: The audio and visuals were . . .
2 Did you find that sounds and visuals were well related?
Complete the sentence: The relationship between sounds and visuals was . . .

A third pair of our questions related to audience understanding of the perform-

er’s actions:

3 Did you find the connection between the performer’s actions and the audio-
visual result understandable?
Complete the sentence: The performance was . . .

In “The Role of Live Visuals in Audience Understanding of Electronic Music

Performances,” 83 we studied the role of visuals in audience understanding of
electronic music performances. For that purpose, we analysed the answers to
the third pair of questions noted earlier. The results showed that projects Eso-
terion Universe, Modulant, Butterfly, EUG and residUUm obtained the highest
results (Modulant with a median of 5, the others with a median of 4). These are
the projects corresponding to the ‘audio-visual entities’ and ‘sounding figura-
tions’ categories from Ribas’s taxonomy.84 This showed that projects in these
categories have higher potential for audience understanding of the performer’s
actions.

Stage 5: Evaluation of Prototypes by Other Audio-Visual

Performers
After the performances, the projects were tested for ease of installation and
robustness. Based on those criteria, six projects were chosen to be evaluated
by other performers (invited from our group of experts from stage 1): Esoterion
Universe, GS.avi and Modulant from Hackathon 1 and Butterfly, residUUm and
Wat from Hackathon 2. We analysed the expert interviews, leading to the iden-
tification of strengths and weaknesses of the projects. Among the good practices
identified, we highlight the XY quadrants and modular code organisation of
Butterfly, the gestural affordances of GS.avi, the object-oriented organisation of
media in Esoterion Universe, the possibility to load files in GS.avi and Modulant
and the powerful manipulation options of Wat and Modulant. Experts expressed
the wish for haptic or parallel interaction, to reconfigure interfaces more easily,
to control more parameters of the projects from the UI, to have an expandable
canvas, to be able to channel visuals to other applications with Syphon and to
visualise the current state and parameter space of the UI. A detailed report on
224 Nuno N. Correia and Atau Tanaka

that evaluation is included in “AVUI: Designing a Toolkit for Audiovisual Inter-

faces.”85 The evaluation from these prototypes originated revised requirements,
which were used in stage 6.

Stage 6: Consolidation Prototype

The results from the previous stages fed into design guidelines for a final proto-
type, an iPad app for AV performance entitled AV Zones. We call this a “con-
solidation prototype” as it brought together ideas and concepts spanning several
prototypes from the earlier phase. This sought to gather individual ideas that
emerged from bottom-up UCD stages of the research into a single application
organised in a classical, top-down development process. The AV Zones app has
been released and its code published as open source.86 It incorporates several
successful features of the prototypes: the XY quadrants of Butterfly, the gestural
affordances of GS.avi, the possibility to load and manage files and powerful
AV manipulation options. Special attention was also given to visualising cur-
rent state and parameter space of the UI. It adopts the object-based concept of
‘zones’: rectangular areas that incorporate UI elements producing and manipu-
lating a specific sound and a visualisation of that sound. This type of object-
based approach was identified as successful in the audience study and the expert
evaluation (particularly with Esoterion Universe, in the latter). Due to this object-
oriented approach, the app fits in the audio-visual entities category. The app has
three vertical zones, each with three XY pads for audio manipulation, control-
ling pitch shift, delay and filter. Each zone has a sequencer, which can record
touch information and visualise it. There are nine sounds available per zone,
which can be switched at runtime and replaced in the code. Different touch
inputs create different results: tapping for triggering sounds, touch movement for
manipulating the sound, two-finger tap to switch on and off and double tap to
trigger special function – a sound menu or sequencer.
The app was developed using openFrameworks87 and the Maximilian audio
library.88 Both are open source and cross-platform. AV Zones was also evalu-
ated by performers from our group of experts from stage 1, completing one
cycle in the research, where practitioners we had interviewed at the beginning
of the UCD process were able to see and try the technology we created based
on their input. Experts were pleased with the affordances of the UI to convey
the performer’s agency, with the aesthetics of the UI and with the manipulation
possibilities. The evaluators wished to have the option to hide the UI, to be able
channel the visuals to other applications and to further control the application
with additional hardware for a better fit with their usual workf low. There is a
wish for more customisation options in terms of sound and visuals. Some of the
functionalities, which were dependent on gestures and not on GUI, were con-
sidered too hidden. A full report on that evaluation can be found in our article
“AVUI: Designing a Toolkit for Audiovisual Interfaces.”89
 AVUIs 225

Stage 7: AVUI Development Toolkit

We next wanted to generalise the interface concepts in AV zones. Rather than
simply publish a compiled end-user application, we wanted to make available its
building blocks in the form of a code library. This would make available AVUI
concepts as enabling technologies to allow developers and creative coders to
incorporate AVUI principles in their own projects.
Lessons learned from the evaluation in stage 6 (in addition to the previous
stages) led to new design requirements that informed the development of an
AVUI toolkit for combining UI with AV content. This toolkit aimed to assist
in streamlining the development of AV work, better integrate interfaces in AV
performances and make interaction more understandable for audiences. The
toolkit was developed using openFrameworks and Maximilian and was released
as an openFrameworks ‘add-on,’ ofxAVUI, following the naming convention for
openFrameworks plug-ins.
We generated design specifications based on evaluations from previous stages.
They were distilled into a new set of requirements, which in turn contributed to
the definition of the feature set of our toolkit. This led to a technical specifica-
tion and software architecture, where the design specifications for the toolkit
were to allow for parallel and visible interaction; integrate sound, image and UI
following an object-oriented approach; enable reconfigurable interfaces, with
f lexible mappings; ensure both clarity and aesthetic appeal of interface, har-
monised with visuals; allow for powerful media manipulation, with procedural
content; and adopt f lexible media management.
The result is an interface builder framework designed specifically for creat-
ing real-time audio-visual software. We adopted the ‘zones’ concept from AV
Zones to allow a developer to create dedicated regions in the screen for the
manipulation of an associated sound source and to display its visualisation.
Each zone has only one sound and one visualisation to reinforce its individual-
ity as an autonomous entity. Different UI elements can be added to a zone: but-
tons, toggles, XY pads, sliders, range sliders, drop-down menus and labels. The
number of zones can be defined, as well as size, position, colour palette and UI
elements. Any parameter from the UI can be rerouted to any audio feature of
the zone or any other aspect of the software (for example, any graphic on the
screen). We kept the minimal UI aesthetics of the prototype (Figure 9.6). We
incorporated the Syphon protocol to enable interapplication communication,
so that media could be channelled, with or without UI, to other applications.
We added two built-in visualisations and also facilitated the creation of new
ones, making the visualisation module extensible. We released the add-on in
versions for personal computer and mobile multitouch devices. ofxAVUI was
released as open source in our GitHub repository 90 and was made available to
the openFrameworks community in their add-on directory, under the cat-
egory GUI.91
226 Nuno N. Correia and Atau Tanaka

FIGURE 9.6 ofxAVUI zones example, with labels identifying different UI elements.

Stage 8: Toolkit Evaluation

In the final stage of the research cycle, we evaluated the ofxAVUI add-on by
organising a one-day hackathon. Our aim was to assess the ease of use and
effectiveness of development of the toolkit. Eight participants took part in the
hackathon (five male, three female). Their profile was similar to participants in
previous hackathons: audio-visual performers and developers. We asked partici-
pants to develop a small project using ofxAVUI. Four of the participants managed
to complete a small project during the event. The projects were FFT/MFCC,
audio frequency analysers and visualisers; Step Sequencer for creating rhythmic
patterns; Background Image for customising zones; and Lisajous and Grid, two addi-
tional visualisers. These projects were added to the ofxAVUI online repository.
In order to obtain further feedback, we contacted ofxAVUI users on GitHub.
Although software downloads are anonymous, 12 individuals had “starred” the
repository – a form of following repository updates on GitHub. Of those 12, 8
had contact information in their GitHub profiles and were contacted by us. We
sent an email asking if they would like to participate in a study. We obtained four
replies, and two developers agreed to participate. They developed two projects:
Multisampler, a four-zone sampler, and ShaderUI, an implementation of sound-
responsive shaders. These were also added to our repository.

Enabling AVUIs Guidelines

We have synthesised the insight gained over the overarching Enabling AVUIs
research cycle in the form of two sets of design guidelines: general guidelines for
 AVUIs 227

AVUI design92 and best practices for the design of AV systems leading to better
audience understanding in performances.93 We summarize those guidelines next.

AVUI Guidelines
The best practices identified allow us to propose the following design guidelines
for use by designers who wish to implement AVUIs. They may be useful for
designers who wish to use sound and image together in the interface, particularly
for AV performance. These guidelines are divided into three main topics:94

1 Maximizing AV Experience
a Develop AVUIs that can be implemented across multiple platforms and
interaction modalities
b Consider the potential of AVUIs for facilitating visualisation of interac-
tion when sharing/showing a screen
c Adopt an object-oriented approach, for a harmonious, coherent and
interrelated convergence of audio, image and UI
d Facilitate different types of display, allowing for different performer-
audience display configurations and hardware
2 Optimizing Interface Functionality and Aesthetics
a Use reconfigurable interfaces that allow users to remap elements of the
UI to different sonic features and visual properties
b Explore not simply one-to-one but also one-to-many mappings
between UI, audio and visual features
c Adopt a minimalist interface aesthetics that does not detract from the
visuals
d Reinforce interface clarity by ensuring visibility of all UI elements,
their state and parameter space
e Allow for hierarchical interfaces, with the possibility of a master con-
trol, and communication between modules
3 Media Strategies
a Allow for powerful manipulation of sound and image: different forms
of media generation and multiple audio and visual effects
b Make use of generative media due to its variety, f lexibility and economy
of resources
c Try different visualisation and sonification approaches, using informa-
tion retrieval techniques from audio and image
d Visualisation should ref lect not only audio but also the multiple interac-
tions afforded by the UI
e Leverage powerful media management features, such as networked con-
tent, and content sharing between applications
228 Nuno N. Correia and Atau Tanaka

Best Practices in AVUI Design for Audience Understanding

In addition to following generic AVUI guidelines (particularly 1b, 1c, 2c and
2d), we identify best practices in AVUI design leading to audience understand-
ing of the performer’s actions.95 The first is to adopt a design pattern based on
‘audio-visual entities’ or ‘sounding figurations.’ Both patterns facilitate a more
direct link between interaction and audio-visual result. For the ‘audio-visual
entities’ category, ensure that the agency of the performer is present on the
screen, but in a stylised way to avoid a “demo effect” (for example, with a cus-
tomised cursor or symbolic representation of multitouch). Regarding a ‘sound-
ing figurations’ category, do not employ a simplistic approach, which might
appear overly demonstrative. This can be accomplished by avoiding direct one-
to-one mappings between points on the screen and audio properties or by intro-
ducing dynamic elements (speed, orientation, etc.) from the gesture generating
the drawing.
Together, the guidelines and best practices consolidate the insights gained
throughout the participatory research that led to the development of the AVUI
toolkit. They point out the qualities we sought to embody in the design of the
software and also point out to users the ways in which the toolkit can be used
to create compelling audio-visual instruments and multimedia performances
where the UI is a holistic part of the creative output. The guideline list is a
distillation of the design thinking that drove the project from its inception to
the release of the code. The guidelines follow the application of third-wave
HCI and Fallmans’s design-oriented HCI in a creative context, as we describe
in music HCI in Tanaka’s “Embodied Musical Interaction.” 96 In the arc of
the research, we span Fallman’s three accounts of design.97 The guidelines list
can be seen as a conservative account, where we broach a broad and arguably
vague problem (“ill defined, unstructured” in Fallman’s words) through a
rational, transparent process. The recommendations for best practice repre-
sent a pragmatic account, where we seek to aid the user-as-bricoleur to create
an audio-visual instrument that is “unique to the situation.” Here we follow
traditions of digital musical instrument building, described by Magnusson as
a digital organology.98 With the instrument, the performer will have the means
to engage in Fallman’s final, romantic account, where the actual problem
of audio-visual interaction becomes subordinate to the artwork or perfor-
mance itself. However, with a foundational design rigour and best practice,
we hope that the body of creative work produced using technologies like the
AVUI toolkit will not just be artistically compelling but will communicate
to audiences.
We believe that these guidelines will help developers in creating AV systems
that solve the issues identified earlier: the “violation of the codes of musical per-
formance”99 due to a feeling of lack of authenticity by audiences in electronic
performances and “a rift between the performer and the audience”100 due to the
lack of understanding of the performer’s actions.
 AVUIs 229

FIGURE 9.7 Territory map of interfaces for AV performance, centred on AVUIs and
Enabling AVUIs prototypes.

Discussion

Mapping the Territory of AVUIs

The Enabling AVUIs project demonstrates that audio-visual user interfaces can
be a fruitful approach to AV performance in terms of design options for perform-
ers and also in terms of audience understanding of the performer’s actions. The
12 prototypes developed (11 by participating artists and a consolidation proto-
type by us) cover a broad design space, taking into account Ribas’s categories
for audio-visual performance.101 Figure 9.8 shows a territory map centred on
AVUIs but also showing other identified interface approaches for AV perfor-
mances. These approaches are not meant to be exhaustive, but are shown as
case studies; there are further examples of hybrid approaches combining mul-
tiple interfaces. The map takes interface types as a starting point, arriving at the
different prototypes from the Enabling AVUI project. The graphic shows that
AVUIs not only adapt windows, icons, menus, pointer (WIMP) GUIs but also
other interface types. Touch input devices are well suited for AVUIs, as demon-
strated by AV zones. To a certain extent, the air-based gestures of EUG, using a
Leap Motion device, demonstrate the f lexibility of AVUIs in terms of 3D user
interfaces. Further work on AVUIs might expand the adaptability of AVUIs to
other interface types identified in Figure 9.7, such as virtual reality, information
visualisation and multimodal interfaces.
Revisiting our requirements for interfaces for AV performance earlier, AVUIs
are more democratic (adopting terminology from Jordà102) interfaces: they
require less technical knowledge (from performer and audience) than live coding
230 Nuno N. Correia and Atau Tanaka

and work with more universal devices (e.g., WIMP, touchscreen devices) com-
pared to dedicated tangible interfaces. However, they have less direct access to
computational processes compared to live coding and do not have the physical
immediacy of TUIs.

User-Centred Design Process in AVUIs

The eight stages in the research cycle represent a three-year trajectory, from
ideation to realisation, to software releases. We engaged with users at each step.
The iterative methods created smaller cycles within the main research thrust.
This demonstrates that user-centred design approaches can be operationalised
in domain-specific, creative technologies. We can embrace emergent, bottom-
up ideation techniques to inform subsequent organised development efforts that
result in the release of enabling technologies. The hope here is that by including
users at the beginning of our design cycles, the resulting plug-in technology we
released will be found to be useful by those same creative communities.
The user, in our case, is the performer. In electronic music performances,
and consequently in AV performances, the performer often builds their own do-
it-yourself (DIY) systems, a phenomena Jordà has entitled as digital lutherie.103
These users therefore can be considered “expert users.” While most technology
development efforts seek to lower the barrier of entry or cater to the lowest com-
mon denominator, there is an increasing interest in HCI for the specific insights
that domain specialists can bring to interaction research.104
We aimed to create an abstraction out of the different AV systems built during
the project – a toolkit to build other AV systems. For that purpose, our ‘expert’
performers, despite their domain-specific knowledge, were not all developers –
they did not necessarily build AVUIs but are potential users of an AVUI toolkit.
Their ‘outside’ view on the AVUI prototypes, free of the partial and idiosyn-
cratic views of the prototype developers, was important in order to extract more
generic requirements for the toolkit, however, where “generic” did not mean
“lowest common denominator.” It is from this perspective of balancing indi-
vidual needs and customisability with a democratic approach of broad use that
the different prototypes were created and tested.
We had two types of performers: the ‘luthier’ users (performers that partici-
pated in the development of prototypes) and the ‘advisor’ users (performers that
set out initial prototype requirements and evaluated the resulting prototypes,
leading to toolkit requirements). This does not mean that our ‘advisor’ users
are not also luthiers – it means that in this process they assumed the role of an
‘advisor.’ Finally, the luthier users also evaluated the AVUI toolkit by means of
a hackathon and subsequent interviews. Figure 9.8 shows a diagram of our user-
centred design process. The figure is meant to highlight the UCD elements,
zooming in on those.
We believe this dual approach of involving luthiers and advisors in UCD can
be relevant when developing tools for more generic use, both in electronic music
 AVUIs 231

FIGURE 9.8 Diagram with the roles of the two types of users (‘luthiers’ and ‘advisors’)
in the Enabling AVUIs UCD process.

and related fields outside of music and visual arts. It can put into a productive
dialogue the perspectives of tool builder and end user. It can facilitate more
widespread usefulness and adoption than simply relying on the idiosyncratic
input of the luthiers.
The lessons learned with applying UCD to the development of the AVUI
toolkit led us to propose adapting the traditional UCD cycle (Figure 9.4). We
added three stages we consider necessary in UCD to scale from prototype design
to toolkit design: specify toolkit requirements from prototype evaluations, pro-
duce toolkit design solution and evaluate toolkit design against requirements
(Figure 9.9). In a sense, the steps “specify requirements,” “produce design solu-
tion” and “evaluate design” are run twice: once for the prototype and mirrored
again for the toolkit (in a more abstracted and generalisable way). With these
amendments, we were able to reconcile the bottom-up ideas emerging from
the workshop activities with the top-down tech development thrust of the con-
solidation prototype. By proposing a methodological extension to user-centred
design that enables merging these otherwise opposing forces to converge, the
result is a design process that is sensitive to user needs and desires, all while
being completely operational in the dynamics and pressures of software product
development.

Beyond Enabling AVUIs

Several of the prototypes that were developed in the scope of Enabling AVUIs
were presented in other contexts. Wat and residUUm were performed at the
Sound/Image 2015 colloquium at the University of Greenwich. The authors
of residUUm wrote an article on the project, which was published at the NIME
conference.105 The first author of this article has been performing and presenting
AV Zones in 10 events, half of which after the conclusion of the Enabling AVUIs
project, in festivals such as Seeing Sound (Bath), New Art Fest (Lisbon), Splice
(London) and ICLI (Brighton).106
232 Nuno N. Correia and Atau Tanaka

FIGURE 9.9 Adapted UCD processes for toolkit design (additions in bold).

Conclusion
AVUIs enable artists to create f lexible audio reactive interfaces for their perfor-
mances. They fulfil the identified key criteria of modularity, visibility, expres-
siveness and democracy in interfaces for performance. AVUIs lead not only to
more effective UIs for the performer’s practice (by aligning and integrating UIs
with the audio-visual content) but also to more transparent and legible perfor-
mances for the audience. Displaying AVUIs to an audience allows one to convey
the agency of the performer by showing the UI and the actions that are performed
with it in a way that is aesthetically harmonious with the audio-visual content.
This can assist in bridging the identified gap between the performer and audi-
ence in digital performances. With the Enabling AVUIs project, we proposed a
set of guidelines and best practices for implementing AVUIs. We also developed
and made available a toolkit, the ofxAVUI add-on, that facilitates the adoption
of AVUIs, using the popular creative coding environment openFrameworks.
With this article, we achieved our stated aims: we assessed the strengths and
weaknesses of AVUIs compared to other approaches, we identified successful
methods for designing AVUIs and we propose a user-centred design process with
performers. We first identified requirements for interfaces for AV performance
across the key interface approaches identified. We created a territory map of
AVUIs: taking interface types as a starting point, passing through the different
identified key interface approaches for AV performance and arriving at the dif-
ferent prototypes from the Enabling AVUI project, grouped around an existing
taxonomy of AV systems. These analyses allowed us to assess the strengths and
weaknesses of AVUIs compared to other interface approaches.
 AVUIs 233

We then highlighted two additions to traditional UCD practices. We pro-

posed a separation into two types of users: ‘luthiers’ and ‘advisors,’ in order to
obtain a degree of abstraction from idiosyncratic practices. We adapted the tra-
ditional four-process UCD model to a seven-process model for toolkit design,
where three of the processes are ran twice (for prototypes and for toolkit). To
arrive at these insights, we captured a longitudinal series of studies spanning
two years that include several cycles of iteration. We went from an initial
scoping of the domain with users to the release of a software product to that
community.
Future work could focus on comparing AVUIs to other interface approaches
for performance using audience studies. Additionally, there is further work to be
done in mapping the territory of interfaces for AV performance beyond the three
typologies we identify in this article.
The evolution to the UCD cycle we present will be useful for practitioners
who wish to develop AV performance systems and toolkits, in particular with
UCD methods. We believe some of these conclusions are not limited to adopting
the AVUI approach but can also be extended to other typologies. We also hope
to have further established AVUIs as a well-defined approach for designing AV
performance systems.

Acknowledgements
We would like to thank all who collaborated in this research: workshop and
hackathon participants, evaluators and interviewees and colleagues, and in par-
ticular Borut Kumperščak for programming assistance and Alessandro Altavilla
for documentation. This work was supported by the EU Marie Curie fellow-
ship FP7 REA grant 627922 and also partly by LARSyS (Project – UIDB/
50009/2020).

Notes
1 Kim Cascone, “Grain, Sequence, System: Three Levels of Reception in the Perfor-
mance of Laptop Music,” Contemporary Music Review, 22(4), December 1, 2003, 101–
104, https://doi.org/10.1080/0749446032000156900.
2 James Patten, Ben Recht, and Hiroshi Ishii, “Audiopad: A Tag-Based Interface for
Musical Performance,” in Proceedings of the 2002 Conference on New Interfaces for Musical
Expression, NIME ’02 (Dublin, Ireland: National University of Singapore, 2002), 1–6.
3 Vitor Joaquim and Álvaro Barbosa, “Are Luminous Devices Helping Musicians to Pro-
duce Better Aural Results, or Just Helping Audiences Not To Get Bored?,” in XCoAx
2013: Proceedings of the First Conference on Computation, Communication, Aesthetics and X
(2013), http://2013.xcoax.org/#p36.
4 Ana Carvalho, “Live Audiovisual Performance,” in Ana Carvalho and Cornelia Lund,
eds., The Audiovisual Breakthrough (Berlin: Fluctuating Images, 2015).
5 Nuno N. Correia and Atau Tanaka, “AVUI: Designing a Toolkit for Audiovi-
sual Interfaces,” in Proceedings of the 2017 CHI Conference on Human Factors in Com-
puting Systems, CHI ’17 (Denver, CO, USA: ACM, 2017), https://dl.acm.org/doi/
abs/10.1145/3025453.3026042.
234 Nuno N. Correia and Atau Tanaka

6 “Enabling AVUIs,” http://avuis.goldsmithsdigital.com/ (Accessed March 27, 2021).

7 Luísa Ribas, The Nature of Sound-Image Relations in Digital Interactive Systems (PhD the-
sis, Universidade do Porto Faculdade de Belas Artes, 2011).
8 Jenny Preece, Helen Sharp, and Yvonne Rogers, Interaction Design: Beyond Human-
Computer Interaction, 4th edition (Chichester: John Wiley, 2015).
9 Alex McLean, “Making Programming Languages to Dance to: Live Coding with
Tidal,” in Proceedings of the 2nd ACM SIGPLAN International Workshop on Functional Art,
Music, Modeling & Design, FARM ’14 (Gothenburg, Sweden: Association for Comput-
ing Machinery, 2014), 63–70, https://doi.org/10.1145/2633638.2633647.
10 Nick Collins et al., “Live Coding in Laptop Performance,” Org. Sound, 8(3), December
2003, 321–330, https://doi.org/10.1017/S135577180300030X.
11 Alex McLean, “Making Programming Languages to Dance to: Live Coding with
Tidal,” in Proceedings of the 2nd ACM SIGPLAN International Workshop on Functional Art,
Music, Modeling & Design, FARM ’14 (Gothenburg, Sweden: Association for Comput-
ing Machinery, 2014), 63–70, https://doi.org/10.1145/2633638.2633647.
12 Nick Collins et al., “Live Coding in Laptop Performance,” Org. Sound, 8(3), December
2003, 321–330, https://doi.org/10.1017/S135577180300030X.
13 Nick Collins et al., “Live Coding in Laptop Performance,” Org. Sound, 8(3), December
2003, 321–330, https://doi.org/10.1017/S135577180300030X.
14 Alex McLean et al., “Visualisation of Live Code,” in Proceedings of the 2010 International
Conference on Electronic Visualisation and the Arts, EVA’10 (London, UK: BCS Learn-
ing & Development Ltd., 2010), 26–30.
15 Alex McLean et al., “Visualisation of Live Code,” in Proceedings of the 2010 International
Conference on Electronic Visualisation and the Arts, EVA’10 (London, UK: BCS Learning &
Development Ltd., 2010), 26–30.
16 Thor Magnusson, “Algorithms as Scores: Coding Live Music,” Leonardo Music Journal,
(21), November 16, 2011, 19–23, https://doi.org/10.1162/LMJ_a_00056.
17 Arrian Purcell, Henry Gardner, and Ben Swift, “Visualising a Live Coding Arts
Process,” in Proceedings of the 26th Australian Computer-Human Interaction Conference
on Designing Futures: The Future of Design, OzCHI ’14 (Sydney, New South Wales,
Australia: Association for Computing Machinery, 2014), 141–144, https://doi.org/
10.1145/2686612.2686634.
18 Arrian Purcell, Henry Gardner, and Ben Swift, “Visualising a Live Coding Arts
Process,” in Proceedings of the 26th Australian Computer-Human Interaction Conference
on Designing Futures: The Future of Design, OzCHI ’14 (Sydney, New South Wales,
Australia: Association for Computing Machinery, 2014), 141–144, https://doi.
org/10.1145/2686612.2686634.
19 Thor Magnusson, “Improvising with the Threnoscope: Integrating Code, Hardware,
GUI, Network, and Graphic Scores,” in Proc. NIME ’14 (London, 2014).
20 Thor Magnusson, “Improvising with the Threnoscope: Integrating Code, Hardware,
GUI, Network, and Graphic Scores,” in Proc. NIME ’14 (London, 2014).
21 “Hydra,” https://hydra.ojack.xyz (Accessed March 27, 2021).
22 Olivia Jack, “Hydra: Live Coding Networked Visuals,” in Proc. International Confer-
ence on Live Coding (Madrid: International Conference on Live Coding 2019 (ICLC),
2019), 353, https://doi.org/10.5281/zenodo.3946269.
23 Olivia Jack, “Hydra: Live Coding Networked Visuals,” in Proc. International Confer-
ence on Live Coding (Madrid: International Conference on Live Coding 2019 (ICLC),
2019), 353, https://doi.org/10.5281/zenodo.3946269.
24 “Fragment Synthesizer,” www.fsynth.com (Accessed March 27, 2021).
25 “Fragment Synthesizer: Documentation,” www.fsynth.com/documentation/index.
html (Accessed March 27, 2021).
26 Alex McLean et al., “Visualisation of Live Code,” in Proceedings of the 2010 International
Conference on Electronic Visualisation and the Arts, EVA’10 (London, UK: BCS Learning &
Development Ltd., 2010), 26–30.
 AVUIs 235

27 James Patten, Ben Recht, and Hiroshi Ishii, “Audiopad: A Tag-Based Interface for
Musical Performance,” in Proceedings of the 2002 Conference on New Interfaces for Musical
Expression, NIME ’02 (Dublin, Ireland: National University of Singapore, 2002), 1–6.
28 James Patten, Ben Recht, and Hiroshi Ishii, “Audiopad: A Tag-Based Interface for
Musical Performance,” in Proceedings of the 2002 Conference on New Interfaces for Musical
Expression, NIME ’02 (Dublin, Ireland: National University of Singapore, 2002), 1–6.
29 James Patten, Ben Recht, and Hiroshi Ishii, “Audiopad: A Tag-Based Interface for
Musical Performance,” in Proceedings of the 2002 Conference on New Interfaces for Musical
Expression, NIME ’02 (Dublin, Ireland: National University of Singapore, 2002), 1–6.
30 James Patten, Ben Recht, and Hiroshi Ishii, “Audiopad: A Tag-Based Interface for
Musical Performance,” in Proceedings of the 2002 Conference on New Interfaces for Musical
Expression, NIME ’02 (Dublin, Ireland: National University of Singapore, 2002), 1–6.
31 James Patten, Ben Recht, and Hiroshi Ishii, “Interaction Techniques for Musical Per-
formance with Tabletop Tangible Interfaces,” in Proceedings of the 2006 ACM SIGCHI
International Conference on Advances in Computer Entertainment Technology, ACE ’06 (Hol-
lywood, CA, USA: Association for Computing Machinery, 2006), 27-es, https://doi.
org/10.1145/1178823.1178856.
32 James Patten, Ben Recht, and Hiroshi Ishii, “Audiopad: A Tag-Based Interface for
Musical Performance,” in Proceedings of the 2002 Conference on New Interfaces for Musical
Expression, NIME ’02 (Dublin, Ireland: National University of Singapore, 2002), 1–6.
33 Sergi Jordà et al., “The ReacTable: Exploring the Synergy between Live Music Perfor-
mance and Tabletop Tangible Interfaces,” in Proceedings of the 1st International Conference
on Tangible and Embedded Interaction (Baton Rouge, Louisiana: ACM, 2007), 139–146,
http://portal.acm.org/citation.cfm?id=1226998.
34 Sergi Jordà et al., “The ReacTable: Exploring the Synergy between Live Music Perfor-
mance and Tabletop Tangible Interfaces,” in Proceedings of the 1st International Conference
on Tangible and Embedded Interaction (Baton Rouge, Louisiana: ACM, 2007), 139–146,
http://portal.acm.org/citation.cfm?id=1226998.
35 Nicole Delager, “Soft Revolvers,” 21st Century Digital Art, 2016, www.digiart21.org/
art/soft-revolvers.
36 Nicole Delager, “Soft Revolvers,” 21st Century Digital Art, 2016, www.digiart21.org/
art/soft-revolvers.
37 Nicole Delager, “Soft Revolvers,” 21st Century Digital Art, 2016, www.digiart21.org/
art/soft-revolvers.
38 Nick Collins et al., “Live Coding in Laptop Performance,” Org. Sound, 8(3), December
2003, 321–330, https://doi.org/10.1017/S135577180300030X.
39 James Patten, Ben Recht, and Hiroshi Ishii, “Audiopad: A Tag-Based Interface for
Musical Performance,” in Proceedings of the 2002 Conference on New Interfaces for Musical
Expression, NIME ’02 (Dublin, Ireland: National University of Singapore, 2002), 1–6.
40 James Patten, Ben Recht, and Hiroshi Ishii, “Audiopad: A Tag-Based Interface for
Musical Performance,” in Proceedings of the 2002 Conference on New Interfaces for Musical
Expression, NIME ’02 (Dublin, Ireland: National University of Singapore, 2002), 1–6.
41 Nick Collins et al., “Live Coding in Laptop Performance,” Org. Sound, 8(3), December
2003, 321–330, https://doi.org/10.1017/S135577180300030X.
42 Alex McLean et al., “Visualisation of Live Code,” in Proceedings of the 2010 International
Conference on Electronic Visualisation and the Arts, EVA’10 (London, UK: BCS Learning &
Development Ltd., 2010), 26–30.
43 James Patten, Ben Recht, and Hiroshi Ishii, “Audiopad: A Tag-Based Interface for
Musical Performance,” in Proceedings of the 2002 Conference on New Interfaces for Musical
Expression, NIME ’02 (Dublin, Ireland: National University of Singapore, 2002), 1–6.
44 James Patten, Ben Recht, and Hiroshi Ishii, “Audiopad: A Tag-Based Interface for
Musical Performance,” in Proceedings of the 2002 Conference on New Interfaces for Musi-
cal Expression, NIME ’02 (Dublin, Ireland: National University of Singapore, 2002),
1–6.
236 Nuno N. Correia and Atau Tanaka

45 Alex McLean et al., “Visualisation of Live Code,” in Proceedings of the 2010 International
Conference on Electronic Visualisation and the Arts, EVA’10 (London, UK: BCS Learning &
Development Ltd., 2010), 26–30.
46 Sergi Jordà, “FMOL toward User-Friendly, Sophisticated New Musical Instruments,”
Computer Music Journal, 26, 2002, 23–39.
47 Sergi Jordà, “FMOL toward User-Friendly, Sophisticated New Musical Instruments,”
Computer Music Journal, 26, 2002, 23–39.
48 Nuno N. Correia and Atau Tanaka, “AVUI: Designing a Toolkit for Audiovi-
sual Interfaces,” in Proceedings of the 2017 CHI Conference on Human Factors in Com-
puting Systems, CHI ’17 (Denver, CO, USA: ACM, 2017), https://dl.acm.org/doi/
abs/10.1145/3025453.3026042.
49 Sergi Jordà, “Sonigraphical Instruments from FMOL to the ReacTable*,” in NIME03
(Montreal, 2003), www.nime.org/proceedings/2003/nime2003_070.pdf.
50 Sergi Jordà, “Sonigraphical Instruments from FMOL to the ReacTable*,” in NIME03
(Montreal, 2003), www.nime.org/proceedings/2003/nime2003_070.pdf.
51 Sergi Jordà, “Sonigraphical Instruments from FMOL to the ReacTable*,” in NIME03
(Montreal, 2003), www.nime.org/proceedings/2003/nime2003_070.pdf.
52 Thor Magnusson, “Ixi Software: Open Controllers for Open Source Audio Soft-
ware,” in Proceedings of ICMC 2005 (Barcelona, 2005), www.ixi-software.net/thor/ixi_
software2.pdf.
53 Thor Magnusson, “Ixi Software: Open Controllers for Open Source Audio Soft-
ware,” in Proceedings of ICMC 2005 (Barcelona, 2005), www.ixi-software.net/thor/ixi_
software2.pdf.
54 Thor Magnusson, “Ixi Software: Open Controllers for Open Source Audio Soft-
ware,” in Proceedings of ICMC 2005 (Barcelona, 2005), www.ixi-software.net/thor/ixi_
software2.pdf.
55 Thor Magnusson, “The IxiQuarks: Merging Code and GUI in One Creative Space,”
in Proceedings of ICMC 2007 (Copenhagen, 2007), www.ixi-software.net/thor/
ixiQuarksICMC.pdf.
56 Thor Magnusson, “The IxiQuarks: Merging Code and GUI in One Creative Space,”
in Proceedings of ICMC 2007 (Copenhagen, 2007), www.ixi-software.net/thor/
ixiQuarksICMC.pdf.
57 Thor Magnusson, “Ixi Software: Open Controllers for Open Source Audio Soft-
ware,” in Proceedings of ICMC 2005 (Barcelona, 2005), www.ixi-software.net/thor/ixi_
software2.pdf.
58 Nuno N. Correia and Atau Tanaka, “User-Centered Design of a Tool for Interactive
Computer-Generated Audiovisuals,” in Proc. 2nd ICLI (Lisbon: ICLI, 2014), 1–14,
www.liveinterfaces.org/proceedings2014.html.
59 Nuno N. Correia and Atau Tanaka, “Prototyping Audiovisual Performance Tools: A
Hackathon Approach,” in Proc. NIME 2015 (Louisiana: Baton Rouge, 2015), 1–3,
www.nime.org/proceedings/2015/nime2015_230.pdf.
60 Nuno N. Correia and Atau Tanaka, “AVUI: Designing a Toolkit for Audiovi-
sual Interfaces,” in Proceedings of the 2017 CHI Conference on Human Factors in Com-
puting Systems, CHI ’17 (Denver, CO, USA: ACM, 2017), https://dl.acm.org/doi/
abs/10.1145/3025453.3026042.
61 Nuno N. Correia, Deborah Castro, and Atau Tanaka, “The Role of Live Visuals in
Audience Understanding of Electronic Music Performances,” in Audio Mostly (New
York, NY, USA: ACM Press, 2017), 1–8, https://dl.acm.org/citation.cfm?id=3123555.
62 Nuno N. Correia and Atau Tanaka, “From GUI to AVUI: Situating Audiovisual User
Interfaces Within Human-Computer Interaction and Related Fields,” EAI Endorsed
Transactions on Creative Technologies (in press).
63 Nuno N. Correia, Interactive Audiovisual Objects (Doctoral thesis, Helsinki, Aalto ARTS
Books, 2013), https://aaltodoc.aalto.fi/handle/123456789/11341.
64 Nuno N. Correia, Interactive Audiovisual Objects (Doctoral thesis, Helsinki, Aalto ARTS
Books, 2013), https://aaltodoc.aalto.fi/handle/123456789/11341.
 AVUIs 237

65 Nuno N. Correia, “AVOL: Towards an Integrated Audio-Visual Expression,” Journal of

Visual Art Practice, 10(3), 2012, 201–214.
66 “Freesound,” https://freesound.org/ (Accessed March 27, 2021).
67 Nuno N. Correia, “AV Clash, Online Audiovisual Project: A Case Study of Evaluation in
New Media Art,” in Proc. Int. Conf. Advances on Computer Entertainment Technology (ACE)
(Lisbon: ACM Press, 2011), http://dl.acm.org/citation.cfm?id=2071423.2071478.
68 Nuno N. Correia, Interactive Audiovisual Objects (Doctoral thesis, Helsinki, Aalto ARTS
Books, 2013), https://aaltodoc.aalto.fi/handle/123456789/11341.
69 Luísa Ribas, The Nature of Sound-Image Relations in Digital Interactive Systems (PhD the-
sis, Universidade do Porto Faculdade de Belas Artes, 2011).
70 Chadia Abras, Diane Maloney-Krichmar, and Jenny Preece, “User-Centered Design,”
in William Bainbridge, ed., Encyclopedia of Human-Computer Interaction (Thousand
Oaks: Sage Publications, 2004).
71 Timo Jokela, “Making User-Centred Design Common Sense: Striving for an Unam-
biguous and Communicative UCD Process Model,” in Proceedings of the Second Nordic
Conference on Human-Computer Interaction, NordiCHI ’02 (New York, NY, USA: ACM,
2002), 19–26, https://doi.org/10.1145/572020.572024.
72 Virginia Braun and Victoria Clarke, “Using Thematic Analysis in Psychology,” Qualita-
tive Research in Psychology, 3(2), 2006, 77–101, https://doi.org/10.1191/1478088706q
p063oa.
73 Carl DiSalvo and Melissa Gregg, “Issue-Oriented Hackathons,” Carl DiSalvo, 2013,
http://carldisalvo.com/posts/issue-oriented-hackathons/.
74 Timo Jokela, “Making User-Centred Design Common Sense: Striving for an Unam-
biguous and Communicative UCD Process Model,” in Proceedings of the Second Nordic
Conference on Human-Computer Interaction, NordiCHI ’02 (New York, NY, USA: ACM,
2002), 19–26, https://doi.org/10.1145/572020.572024.
75 Nuno N. Correia and Atau Tanaka, “User-Centered Design of a Tool for Interactive
Computer-Generated Audiovisuals,” in Proc. 2nd ICLI (Lisbon: ICLI, 2014), 1–14,
www.liveinterfaces.org/proceedings2014.html.
76 Lars Erik Holmquist, “Bootlegging: Multidisciplinary Brainstorming with Cut-Ups,”
in Proceedings of the Tenth Anniversary Conference on Participatory Design 2008, PDC ’08
(Indianapolis, IN, USA: Indiana University, 2008), 158–161, http://dl.acm.org/citation.
cfm?id=1795234.1795260.
77 Lars Erik Holmquist, “Bootlegging: Multidisciplinary Brainstorming with Cut-Ups,”
in Proceedings of the Tenth Anniversary Conference on Participatory Design 2008, PDC ’08
(Indianapolis, IN, USA: Indiana University, 2008), 158–161, http://dl.acm.org/citation.
cfm?id=1795234.1795260.
78 Nuno N. Correia and Atau Tanaka, “User-Centered Design of a Tool for Interactive
Computer-Generated Audiovisuals,” in Proc. 2nd ICLI (Lisbon: ICLI, 2014), 1–14,
www.liveinterfaces.org/proceedings2014.html.
79 Nuno N. Correia and Atau Tanaka, “User-Centered Design of a Tool for Interactive
Computer-Generated Audiovisuals,” in Proc. 2nd ICLI (Lisbon: ICLI, 2014), 1–14,
www.liveinterfaces.org/proceedings2014.html.
80 “GitHub – AVUIs,” GitHub, https://github.com/AVUIs (Accessed March 27, 2021).
81 Luísa Ribas, The Nature of Sound-Image Relations in Digital Interactive Systems (PhD the-
sis, Universidade do Porto Faculdade de Belas Artes, 2011).
82 Steve Benford et al., “Performance-Led Research in the Wild,” ACM Transac-
tions on Computer-Human Interaction, 20(3), July 1, 2013, 14:1–14:22, https://doi.
org/10.1145/2491500.2491502.
83 Nuno N. Correia, Deborah Castro, and Atau Tanaka, “The Role of Live Visual
in Audience Understanding of Electronic Music Performances,” in Audio Mostly
(New York, NY, USA: ACM Press, 2017), 1–8, https://dl.acm.org/citation.
cfm?id=3123555.
84 Luísa Ribas, The Nature of Sound-Image Relations in Digital Interactive Systems (PhD the-
sis, Universidade do Porto Faculdade de Belas Artes, 2011).
238 Nuno N. Correia and Atau Tanaka

85 Nuno N. Correia and Atau Tanaka, “AVUI: Designing a Toolkit for Audiovi-
sual Interfaces,” in Proceedings of the 2017 CHI Conference on Human Factors in Com-
puting Systems, CHI ’17 (Denver, CO, USA: ACM, 2017), https://dl.acm.org/doi/
abs/10.1145/3025453.3026042.
86 “GitHub – AVZones,” November 27, 2018, https://github.com/AVUIs/AVZones-beta.
87 “OpenFrameworks,” 2015, http://openframeworks.cc.
88 “GitHub – Maximilian,” March 25, 2021, https://github.com/micknoise/Maximilian.
89 Nuno N. Correia and Atau Tanaka, “AVUI: Designing a Toolkit for Audiovi-
sual Interfaces,” in Proceedings of the 2017 CHI Conference on Human Factors in Com-
puting Systems, CHI ’17 (Denver, CO, USA: ACM, 2017), https://dl.acm.org/doi/
abs/10.1145/3025453.3026042.
90 “GitHub – OfxAVUI,” December 14, 2020, https://github.com/AVUIs/ofxAVUI.
91 “Ofxaddons.Com – GUI,” n.d., https://ofxaddons.com/categories/1-gui.
92 Nuno N. Correia and Atau Tanaka, “AVUI: Designing a Toolkit for Audiovi-
sual Interfaces,” in Proceedings of the 2017 CHI Conference on Human Factors in Com-
puting Systems, CHI ’17 (Denver, CO, USA: ACM, 2017), https://dl.acm.org/doi/
abs/10.1145/3025453.3026042.
93 Nuno N. Correia, Deborah Castro, and Atau Tanaka, “The Role of Live Visual in
Audience Understanding of Electronic Music Performances,” in Audio Mostly (New
York, NY, USA: ACM Press, 2017), 1–8, https://dl.acm.org/citation.cfm?id=3123555.
94 Nuno N. Correia and Atau Tanaka, “AVUI: Designing a Toolkit for Audiovi-
sual Interfaces,” in Proceedings of the 2017 CHI Conference on Human Factors in Com-
puting Systems, CHI ’17 (Denver, CO, USA: ACM, 2017), https://dl.acm.org/doi/
abs/10.1145/3025453.3026042.
95 Nuno N. Correia, Deborah Castro, and Atau Tanaka, “The Role of Live Visual in
Audience Understanding of Electronic Music Performances,” in Audio Mostly (New
York, NY, USA: ACM Press, 2017), 1–8, https://dl.acm.org/citation.cfm?id=3123555.
96 A. Tanaka, “Embodied Musical Interaction,” in S. Holland, T. Mudd, K. Wilkie-
McKenna, A. McPherson and M. Wanderley, eds., New Directions in Music and Human-
Computer Interaction, Springer Series on Cultural Computing (Cham: Springer, 2019),
https://doi.org/10.1007/978-3-319-92069-6_9.
97 D. Fallman, “Design-Oriented Human: Computer Interaction,” in Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems, CHI ’03. ACM (New York,
NY, USA, 2003), 225–232, https://doi.org/10.1145/642611.642652.
98 Thor Magnusson, “Musical Organics: A Heterarchical Approach to Digital Organol-
ogy,” Journal of New Music Research, 46(3), 2017, 286–303, DOI: 10.1080/09298215.
2017.1353636.
99 Kim Cascone, “Grain, Sequence, System: Three Levels of Reception in the Perfor-
mance of Laptop Music,” Contemporary Music Review, 22(4), December 1, 2003, 101–
104, https://doi.org/10.1080/0749446032000156900.
100 James Patten, Ben Recht, and Hiroshi Ishii, “Audiopad: A Tag-Based Interface for
Musical Performance,” in Proceedings of the 2002 Conference on New Interfaces for Musical
Expression, NIME ’02 (Dublin, Ireland: National University of Singapore, 2002), 1–6.
101 Luisa Ribas, “Sound-Image Relations and Dynamics in Digital Interactive Systems,” in
6th International Conference on Digital Arts – ARTECH 2012 (Faro, 2012).
102 Sergi Jordà, “FMOL toward User-Friendly, Sophisticated New Musical Instruments,”
Computer Music Journal, 26, 2002, 23–39.
103 Sergi Jordà, “Instruments and Players: Some Thoughts on Digital Lutherie,” Journal of
New Music Research, 33(3), September 1, 2004, 321–341, https://doi.org/10.1080/092
9821042000317886.
104 Michel Beaudouin-Lafon, “Towards Unified Principles of Interaction,” in Proceed-
ings of the 12th Biannual Conference on Italian SIGCHI Chapter, CHItaly ’17 (New
York, NY, USA: Association for Computing Machinery, 2017), 1–2, https://doi.
org/10.1145/3125571.3125602.
 AVUIs 239

105 Ireti Olowe, Giulio Moro, and Mathieu Barthet, “ResidUUm: User Mapping and
Performance Strategies for Multilayered Live Audiovisual Generation,” in Proceedings of
International Conference on New Interfaces for Musical Expression (NIME) (Brisbane, 2016),
www.nime.org/proceedings/2016/nime2016_paper0053.pdf.
106 Nuno N. Correia, “AV Zones: Tablet App for Audiovisual Performance,” in Proc. 3rd
ICLI (Brighton, 2016).
10
A PARAMETRIC MODEL FOR
AUDIO-VISUAL INSTRUMENT
DESIGN, COMPOSITION AND
PERFORMANCE
Adriana Sá and Atau Tanaka

Introduction
We may think of an audio-visual performance as a construction of experienced
time, which depends on multiple intertwining variables. While the performer
may compose and perform by thinking directly or indirectly about these vari-
ables, the audience will likely prefer to focus on the experience itself and not try
to glean insight on low-level details; however, creative compositional processes
can benefit from analysis and understanding how a work is perceived.
There are many ways of elaborating a performance: creating design specifica-
tions, scoring the sonic and visual materials, and planning the arrangement of the
physical space. Each of those activities requires its own process and method, and
some decisions are more subject to change than others, depending – or not – on
complementary aspects. As practitioners, we feel need for a unifying, comple-
mentary method which can be used to set out our general approach for those
different work stages at once, as well as in parallel.
We introduce the notion of the parametric visualisation model as a tool for
the compositional process. By systemising a set of variables (i.e. parameters in
a graphical way), a parametric visualisation model can reveal relationships and
interdependencies between those parameters. Instruments and performance situ-
ations can then be represented with a set of axes and analysed accordingly. As an
example, the model created by Birnbaum et al. reveals relations between interac-
tion, sound organisation, physical distribution in space and semantics of sound.1
Similarly, the framework created by Thor Magnusson reveals how digital music
devices condition interaction and sonic results,2 while the one created by Marko
Ciciliani reveals how an electronic music performance might draw the focus to
the performer or the environment.3
In this chapter we propose a parametric model that is useful in audio-visual
instrument design, composition and performance. We draw a separation between
DOI: 10.4324/9781003282396-13
 A Parametric Model 241

those activities, but in practice that separation might not be so obvious: ulti-
mately, the iterative creation process must always consider the final, global expe-
rience. We take a perceptual approach in conceiving our model and intend it to
be applicable to the broad diversity of aesthetic options and technical platforms.
The model is modular: one can discard part of the parameters so as to analyse
any time-based work, including recorded audio pieces and films. On the one
hand, the model enables the separate analysis of performer-instrument interac-
tion, sound, image, audio-visual relationship and physical setup. On the other, it
enables the analysis of how the combination conducts the audience’s experience.

A Panorama of Creative Questions

We use the term ‘spatial presence’ to describe the subjective experience of being
physically located in a mediated space. Studies in sound art4,5 and media the-
ory6,7,8,9 indicate that spatial presence is inf luenced by individual predispositions,
the characteristics of sound and image and audio-visual relationships. Other dis-
cussions focus on how the psychological space of the work is informed by loud-
speaker placement.10,11 From these theories, we conclude that spatial presence
depends on multiple variables, the combination of which informs the semantics
of a work. We define semantic typologies to assess the global semantics of a
composition or performance based on their relative weights. In addition, we
draw upon Simon Emmerson’s notion of ‘performative arena’12 to define a high-
level parameter, which can provide cues that other, detailed parameters may not
provide.
Michel Chion observed that the combination of sound and image generates
a third audio-visual element.13 However, only a part of the sensory information
reaches conscious awareness. The question is, how can that combination ben-
efit the experience of music, be it sonic or visual? The abstract filmmaker Stan
Brakhage created silent films out of a feeling that sound dominates the subtler
rhythms of vision. In the last two decades of his life he created audio-visual
movies in which he breaks any direct connection between picture and sound.14
The converse exists in the musical domain, where Salomé Voegelin points out
how vision dominates over audition in our oculocentric culture.15 From Pierre
Schaeffer16 to recent acousmatic composers, many people have argued that
sounds must be detached from their originating cause to be fully experienced.
Jeff Pressing, a composer and cognitive researcher who investigated the audio-
visual relationship in digital 3D environments, noted that perception operates
from vision to audition whenever a direction of causation is discernible.17 The
graphic performer Meghan Stevens believes that the music remains dominant
when the audio-visual relationship is partially congruent, yet she is the first to
admit that her theories rely on limited evidence.18
Questions of sensory dominance therefore pose a challenging problem. In
neuroscience, Sinnett et al. found that it depends on attention and that attention
can be manipulated so that one sense dominates over another.19 He does not say
242 Adriana Sá and Atau Tanaka

how, but extrapolating from creative practice, audio-visual theory and the sci-
ence of perception enables us to identify the variables that inf luence how the
audience’s experience can be driven through the sound, the image or the audio-
visual composite. Our model considers not only the audio-visual fit but also the
sonic and the visual dynamics.
Interaction – between media, between performer and material and between
performance and spectatorship – is fundamental in the exploration of perceptual
experience. Different artists address interaction through a wide range of strate-
gies and with very different creative motivations. Tod Machover and his team
at MIT Media Lab developed interfaces intended to compensate for “people’s
limitations”20; this also implies that the software prescribes which output results
are desirable and which are not. In contrast, Joel Ryan (who pioneered the digi-
tal signal processing of acoustic instruments) encourages us to make control as
difficult as possible, linking musical expression to effort.21 Indeed, one inter-
face might allow for unpredictability so as to convey the reciprocal interaction
between performer and instrument, whilst another interface prescribes the out-
come beforehand. Interaction design is thus determined by different creative
principles corresponding to different notions of expression. Our model should
include a distinguishing parameter applicable to any sonic, visual and audio-
visual system and instrument. We can quantify effort relative to the amount of
cognitive processing required in a given task. Effort might further ref lect in the
dynamics and the semantics of sound and image.
In the following sections we begin by presenting each parameter indepen-
dently. We illustrate their use by giving a range of artistic examples. We then
explain how their combination facilitates the analysis of expression, spatial pres-
ence and sensory dominance. Finally, we demonstrate how the model can be
used in creative practice and show its usefulness as a compositional tool.

Interaction Effort
One of our objectives is to facilitate the analysis of how interaction designs con-
vey expression, be it in the sonic, the visual or the audio-visual domain. The
term ‘design’ is here not constrained to the activity of designers; we purposefully
do not distinguish between idealising, crafting, composing and performing.
Birnbaum et al.22 and Magnusson 23 propose parametric models to analyse
interaction with digital music devices. Both their models include parameters
related to the performer’s control over the device and the prior system knowl-
edge required for interaction. Here, we summarise those variables into a single
parameter: the performer’s cognitive effort, that is their mental information pro-
cessing, conscious and unconscious.
In previous work on new musical instruments, we elaborated on how differ-
ent levels of effort convey different notions of musical expression.24 This under-
standing is equally applicable to the sonic, visual and audio-visual domains. As a
parameter, effort can be characterised as follows:
 A Parametric Model 243

 Little effort means one of two things: either the work does not depend
much on real-time interaction or the relationship between intention and
resulting output is linear and clearly perceivable.
 Medium effort means that the performer needs particular skills to play the
instrument, but a sense of immediacy conveys f luency and timing and/or
technical configurations rule out undesired outcomes.
 High effort implies particular skills and/or high cognitive demand; the
interaction with the system does not feel immediate and/or the system does
not rule out any outcomes.

In our parametric visualisation model, a single axis suffices to represent the real-
time interaction effort, motor and/or conceptual.
An example of ‘low interaction effort’ can be found in Phill Niblock’s Move-
ments of People Working, performed since 1973.25 These works show repetitive
movements of manual labour combined with massive drones of sound, rich in
harmonics and overtones. The images are created in advance, and Niblock’s
graphic scores for the sound have been interpreted by many musicians. His inter-
action with sound and image in performance is very sparse. A different example
of low effort can be found in Music for Solo Performer by Alvin Lucier (1965),26
where he uses a brain interface to activate multiple percussion instruments. The
interface was crafted so that the lesser the brain wave energy, the stronger the
actuation over the instruments. This work raises an interesting issue: the interac-
tion with an effortful interface can be effortless. Indeed, brain waves are hard to
control. But in an interview Lucier explains that he didn’t want to show mind
control, because he preferred the discovery of how his brainwaves sounded.27
To him, composition is about how to deploy the loudspeakers and what instru-
ments to use. For him, brain music performance was not about making an effort
to create certain brain activity, but rather to enter into a meditative state of
biofeedback.
‘Medium effort’ can be manifested in a range of behaviours over time, an
adaptation to unpredictable conditions, a monitoring of results in relation to a
reference source or an anticipation of changes in oneself or the environment. Jeff
Pressing coined the term ‘dynamic complexity’28 to describe this in music. We
can say that medium effort implies behavioural deviations and reactions to those
deviations.
A musical example of medium effort is in a performance by Joel Ryan (elec-
tronics) and Evan Parker (soprano saxophone).29 Ryan’s instrument is a digital
signal processing of the saxophone where processing parameters are performa-
tively manipulated. The two instruments are therefore in an interdependent rela-
tionship. The sounds of the saxophone and the electronics converge when their
loudness and tone are the same; then they cause attention to focus on subtle tonal
shifts, diverging progressively as one timbre emerges from the other, so as to
converge again. Each performer plays their instrument with its normal cognitive
load, but also must pay attention to the consequences that their play has on the
244 Adriana Sá and Atau Tanaka

material provided to the other performer. Another example of medium effort is

found in the Violin Power series of audio-visual performances by Steina Vasulka,30
where she controls laser-disc video from an electric MIDI violin 31 (Figure 10.1).
Vasulka creates audio-visual tension whenever certain musical gestures on her
instrument cause the video player to seek different segments of video footage.
These transitions are abrupt at the visual level, although not at the sonic level.
The subsequent return to audio-visual synchrony creates a convergence, which
causes a sensation of release.
A high level of effort conveys yet another type of expression. A paradigmatic
example can be found in the work of Martin Howse, where he uses the con-
ductivity of earth as part of a system to perform noise and electronic music.32
He investigates the links between geophysical phenomena, software and the
human psyche, proposing a return to animism within a critical misuse of scien-
tific technology. In performance, his interfaces combine a diversity of chemical
substances, earth materials and computers. Similar to Lucier, Howse empha-
sises discovery as opposed to control. But very differently, the output is highly
dependent on his real-time decisions and actions, on direct intervention with his
materials – literally digging into the ground (Figure 10.2).

FIGURE 10.1 Steina Vasulka playing her audio-visual instrument. From the STEIM
“Waisvisz archive.” Photographer unknown. Used by permission of
STEIM.
Source: https://screenfestival.no/post/75982085903/artist-talk-with-steina-vasulka
 A Parametric Model 245

FIGURE 10.2 Martin Howse, Earth Voice, Media Mediums, 2014, Paris. Used by
permission.
Source: [email protected]

Continuity
Turning to the audience’s experience, one important concern is how the dynam-
ics of sound and image drive attention. Drawing from neuroscience and psy-
chology, we created a taxonomy of continuities and discontinuities related to
intensity and attention.33 We defined intensity as the psychophysical impact of
any change in the chain of stimuli causing an increase in neural activity. That
neural activity is a measure of attention, which means that we can quantify
intensity based on how attention works.
Attention is automatic when driven by salient events, such as the sudden
appearance or disappearance of a stimulus. Such events counteract biophysical
expectations, causing a great increase in neural activity; that increase is consistent
with primary survival instincts. Conversely, attention is under individual con-
trol when expectations are fulfilled; it evokes less neural activity, then, because
there are no significant changes in sensory information. It is important to note
that expectations depend greatly on the panorama – previous and simultane-
ous events, as well as the time length of experience. Meanwhile, the threshold
between deliberate and automatic attention can be fuzzy, as attention causes us to
optimise perceptual resolution so as to better process information related to the
attention target.34 As deliberate attention makes detail changes more intense, we
also become more susceptible to automatic attention.
246 Adriana Sá and Atau Tanaka

Intensity is therefore proportional to perceived discontinuity. It depends on

the event itself, on the panorama and on a person’s perceptual resolution. We
propose the following taxonomy:

 Steady continuity has no intrinsic motion; it is of lowest intensity, dis-

pensing with attention. Attention is likely to deviate and focus upon any
simultaneous stimuli or upon internal states.
 Progressive continuity occurs when successive, non-abrupt events display
a similar interval of motion. It fulfils the expectation that once something
begins to move in a certain direction, it will continue to move in that direc-
tion (Gestalt principle of good continuation).
 Ambivalent discontinuity refers to the threshold between continuity and
discontinuity. At low perceptual resolution, a predictable logic is shifted
without disruption. At high resolution, discontinuities become more intense.
Higher intensity implies greater attention, and lower intensity implies less
attention.
• Radical discontinuity violates psychophysical expectations, prompting
automatic attention. It is of the highest intensity, implying greater neural
activity. Radical discontinuities are always prioritised in the stimuli competi-
tion to reach conscious awareness.
• Endogenous continuity corresponds to the mental representation of per-
ceptual motion; it can embrace all the other types of continuities and discon-
tinuities. We use the term endogenous to stress that perceiving a coherent
relationship between them depends greatly on the individual.

Our parametric visualisation model employs two axes to represent the sonic and
visual dynamics, as shown in Figure 10.8 later in this chapter; “SC” means steady
continuities, “PC” means progressive continuities, “AD” means ambivalent dis-
continuities and “RD” means radical discontinuities. Endogenous spans all the
dynamics. Other types of continuities and discontinuities can be illustrated with
paradigmatic examples.
A musical example of steady continuity is Elaine Radigue’s Triologie de la Mort
(1998),35 a three-hour drone piece where we hardly perceive any overtones;
the work relates strongly to Tibetan Buddhism. An audio-visual example is
La Monte Young and Marian Zazeela’s Dream House,36 an ongoing installation
(since 1962) that defines a vibratory space through the combination of continu-
ous sound and light frequencies, experimenting on how people are drawn to
inhabit it.
The notion of progressive continuity can be illustrated with any gradual
increase or decrease in loudness, tonality, brightness, colour, density, rhythm
or time length. An example can be found in Gary Hill’s film Black and White
Text (1980),37 which explores a relationship between geometric black and white
figures and human voice. As the work unfolds, the intervals between the words
and the visual shifts become progressively shorter, while sound layers accumulate
 A Parametric Model 247

and rectangles multiply on screen. Importantly, progressive continuity implies

perceived motion. If the progression happens so slowly that we cannot apprehend
any change (as happens in Radigue’s Triologie), it would instead be considered a
case of steady continuity.
While progressive continuity entails motion in a clearly perceivable direction,
ambivalent discontinuity entails multidirectional motion. An example is seen
in Thomas Wilfred’s Lumia from the 1930s.38 The Lumia are dynamic light-
paintings created with a visual instrument that combines projectors, ref lectors
and coloured slides so as to produce polymorphous streams of colour, which
invite attention to focus on subtle detail changes. We can revisit Phil Niblock’s
Movements of People Working39 to see how perception focuses on the nuances of
sound and image – the repetitive movements of manual labour and the continu-
ous mass of sound. As we increase perceptual resolution, the detail variations
become more intense. The fuzzy threshold between ambivalent discontinuity
and radical discontinuity is particularly evident in Stellar (1993), one of Stan
Brakhage’s silent abstract films drawn directly on the film strip. The film has
elements of continuity; all frames are made with the same technique and the
same colours. And yet, as we focus on the visual detail changes proliferating in
the fast-changing frames, we also become very sensitive to how the interplay of
discontinuities grounds the construction of time.
Radical discontinuities can be used so as to create rhythmic patterns, as hap-
pens in Vasulka’s Heraldic View (1974),40 where visual patterns are created by an
audio synthesizer. When the duration of the experience is short, each abrupt
event prompts automatic attention, causing a sudden increase in neural activ-
ity. Yet after a while, the sequence of elements fulfils expectations, as happens
with any pattern; in this way, radical discontinuity becomes steady continuity.
Alternatively, radical discontinuities can be explored so as to tease and counter-
point expectations. For example, Ryoji Ikeda’s performance superposition41 cre-
ates radical discontinuities with sudden blackouts; the contrast with moments
of progressive and steady continuity makes those discontinuities more intense
(Figure 10.3).

Audio-Visual Fit
The way perception prioritises sensory information is inf luenced by the dynam-
ics of sound and image, but the audio-visual relationship is equally important.
In audio-visual theory, Chion coined the term ‘added value’ to describe the
surplus of synchronisation.42 It is crucial not to misinterpret the term, because
the meaning of the audio-visual composite is not really added to the meanings of
the sound and the image. On the contrary, it tends to override those meanings.
In experimental psychology, Kubovy and Schutz showed that the aural discounts
the visual and the visual discounts the aural based on concepts of causation.43,44
They coined the term ‘ecological fit’ to describe how automatic interactions
between the senses are governed by those concepts.
248 Adriana Sá and Atau Tanaka

FIGURE 10.3 Ryoji Ikeda, superposition, 2012 performance. Kyoto Experiment, Kyoto
Art Theatre Shunjuza, Kyoto, 2013. Photo by: Kazuo Fukunaga.
Courtesy: Kyoto Experiment. © Ryoji Ikeda. Used by permission.
Source: Satori Kita <[email protected]>

The greater the ecological fit, the more we ignore any diverging sensory
information. We can explain this in terms of cognitive ‘efficiency.’ A high level
of fit leads to integrated perceptual encodings and representations, which require
less neural activity than separate ones.45 Drawing from the science of perception
and audio-visual theory, we investigated perceived audio-visual relationships46
and identified three levels of ecological fit:

• High fit means that the audio-visual relationship conveys conclusive infor-
mation about causes and effects. Our perceptual mechanisms prioritise
information that comply with those conclusions, producing integrated men-
tal representations. High fit is of low intensity, because it does not require
much cognitive processing.
• Medium fit means that one senses causation without understanding the
base cause and effect relationships. It is of medium intensity and requires a
medium level of cognitive processing. It conveys perceptual chunking, but
the process of audio-visual binding remains ambivalent: one can form inte-
grated as well as separated representations of the sounds and the images.
• Low fit means that the pairing of sound and image does not activate prior
memories of causation. Perceptual binding is weak, requiring perception to
create new chunks of memory, with a large amount of cognitive processing.
That means high intensity.
 A Parametric Model 249

Our parametric visualisation model uses two axes to represent the extent to
which what we see fits with what we hear: one for the fit between sound and
image, and the other for the fit between physical gesture and system output.
‘High ecological fit’ can be illustrated with Norman McLaren’s abstract ani-
mation film Dots (1940),47 where sound and image are synchronised one-to-one.
The visual elements consist of dots, which McLaren painted directly onto clear
frames of film. The sounds were created in the same way, with dots painted
directly into the area on the filmstrip usually reserved for the soundtrack. Another
example is Noise Fields (1974),48 a fully synchronised video by Steina and Woody
Vasulka. Made with analogue video synthesis processors, this work visualises
and sonifies the energy of the electronic signal. Beyond the film and video art,
many systems and instruments were designed to emphasise the union of audition
and sight through one-to-one synchronisation. The Ocular Harpsichord created
by Louis Castel (1730) is an early example. It consisted of a harpsichord with
coloured glasses and curtains; when a key was struck, a corresponding curtain
would lift brief ly to show a f lash of corresponding colour.49 As a contemporary
example, 3D positional audio-effects used in video games are intended to create
a high audio-visual fit, and creative works such as Tarik Barri’s Versum50 explore
this as a means of composition.
From an earlier study, we coined the term ‘fungible mapping’ to describe
an audio-visual mapping exhibiting medium fit. It combines synchronised and
non-synchronised components, exhibiting complexity enough to be confusing.
In our study, participants were aware of a causal relationship and aware of not
distinguishing the base cause-and-effect relationships. As they could not segre-
gate converging and diverging information, their sense of causation extended to
the mapping as a whole. The study was greatly motivated by the development
of an audio-visual instrument,51 which combines an acoustic string instrument
and 3D software that operates based on the acoustic input (Figure 10.4).52,53,54
It clarified how the instrument could confound the cause-effect relationships
in spite of using a 3D engine – a technical platform intended to maximise the
audio-visual fit. It also enabled extrapolations into the physical setup: in perfor-
mance the relation between physical gesture and instrument output is sometimes
synchronised and at other times not. Additionally, two stereo audio pairs crossed
in space blur the relation between the visible sound emitters on the screen and
the corresponding sounds emitted through the loudspeakers.
Another example of ‘medium fit’ in our own creative work can be found in
the performances of Tanaka’s group, Sensors_Sonics_Sights.55,56 The trio uses
sensor-based digital musical instruments, capturing performer gesture to modu-
late 3D imagery and synthesised sound (Figure 10.5). Two members play sound
and one plays image, with the connection between the media taking place
through the traditional ensemble practice of synchronising by eye contact and
gesture. The audience senses a causal connection between the performers’ gestures
and the sonic and visual outputs. Nevertheless, the nature of the instruments
confounds the base cause-and-effect relationships. There is no technological
250 Adriana Sá and Atau Tanaka

FIGURE 10.4 Top: Performance by Adriana Sá and John Klima at Maga Festival/
Silos Contentor Criativo, Caldas da Rainha, 2018 (photo by Susana
Valadas, courtesy of Grémio Caldense). Bottom: studio setup of Sá’s
audio-visual instrument.
Source: [email protected]

connection between sound and image, but points of sensory unison convey per-
ceptual binding. As perception doesn’t segregate the elements that produce a
sense of causation from the elements that do not, the feeling of causation extends
to the audio-visual relationship as a whole. The practice of small ensemble cham-
ber music performance therefore is extended to audio-visual performance.
 A Parametric Model 251

FIGURE 10.5 Performance of Atau Tanaka’s group, Sensors_Sonics_Sights, 3 Legged

Dog, New York 2007.
Source: Chapter authors.

Another way of creating ‘medium fit’ can be found in many live coding
events, where textual programming onstage generates music and/or visuals as
it is written live in performance. The code is usually projected on a screen so
that people can see the process, but the cause-and-effect relationships are often
confounding. Alex McLean sometimes purposefully obscures his code to make
it more difficult to read, while still showing some of the activity of the edits.57 In
Thor Magnusson’s performances with the Threnoscope58 the digital cause-effect
relationships are exposed with a graphic notation system and real-time program-
ming code, yet even coders won’t fully understand the cause-effect relationships
because the code is relatively high-level and the system is complex.59 In other
words, medium fit is also compatible with consistent synchrony.
Laptop performances are often criticised for having a low fit between physical
gesture and system output. But in watching an audio-visual performance or a
film, we are driven to perceive – and imagine – connections between the sounds
and images, even when their fit is low and perceptual binding is weak. Often one
can extrapolate meanings from video images of one thing coupled with sounds
from something completely different, even if there is no synchrony.

Performer Position Relative to Image

We propose an additional parameter for characterising the performer’s physical
position relative to the image. It is useful to distinguish the following three types
of arrangements:

• Integrated means that the image and the performer’s physical body form a
single visual scene, as happens when an image is projected upon a performer.
252 Adriana Sá and Atau Tanaka

• Separated means that the image is separated from the performer, who is
nevertheless visible. This type of arrangement can divide attention or deviate
attention from the performer.
• Hidden means that the performer is not visible. The audience does not see
their agency, but knowing what type of interface is being used can inf luence
how the work is perceived.

Our parametric visualisation uses three discrete points to represent these types
of arrangements.
The way the physical setup inf luences attention also depends on speaker
placement, lighting, audience distribution and physical architecture. All these
details count in the audience’s experience. However, in order to facilitate the
application of the parametric model, we do not parameterise each aspect inde-
pendently. Instead, our model includes two high-level parameters – ‘semantics’
and ‘performative arena’ – which provide cues about variables that the other
parameters do not address.

Semantics
Semantic organisation can be described with respect to causes and concepts, but
attention dynamics have intrinsic semantics as well: every experience has mean-
ing, including when we focus on perceptual motion itself.60,61 The mental rep-
resentation of the work as a whole can be considered an endogenous continuity.
The notion of endogenous continuity expands Jeff Pressing’s semantic charac-
terisation of sounds62 so as to embrace the visual and the audio-visual domains.
He distinguished ‘expressive,’ ‘informational’ and ‘environmental’ sounds, stress-
ing that these typologies normally overlap. Expressive sounds would include all
kinds of music and song. Examples of informational sounds would be speech,
alarms and sonified data. Examples of environmental sounds would include ani-
mal calls, wind sounds and the noises of machinery. We adapt these semantic
typologies as follows:

• Informational semantics prompt causal percepts, shifting attention to a

meaning.
• Expressive semantics means that the focus of attention is upon a central
target.
• Environmental semantics means a focus upon a context or environment.

These three semantic dimensions can be quantified independently. We can

quantify the informational dimension by assessing our conclusiveness about a
cause or meaning. It might be useful to look at semiotics, where the notions of
icon, index and symbol characterise different types of relations between signi-
fier and meaning. But these notions do not suffice to quantify expressive and
environmental semantics, which also exist when the informational load is low.
 A Parametric Model 253

Regardless of concepts and interpretations, the more attention that is focused

on a specific target, the less it spreads through the environment, and vice versa.
Our parametric visualisation model also takes advantage here; the expressive and
environmental dimensions of a work can be represented in a single axis. As such,
two axes suffice to represent the three types of semantics.
When analysing a creative work, we might consider the semantics of interac-
tion, sound, image, audio-visual relationship and physical setup. One can assess
the semantics of each element and estimate their relative weight in the global
meaning of any particular work.
If we think of interaction in terms of cognitive effort, predictable, clearly
perceivable interface behaviours provide a large amount of information about
how the system should be interacted with, making the interaction effortless. A
system that does not depend on real-time control is effortless as well, but the
information content of the interaction is low, as the audience does not perceive
to what extent the performer inf luences system output. Furthermore, perceiving
effort implies interpreting causes and meanings. Perceived effort tends to attract
attention, supporting expressive semantics.
Sounds and images have informational content whenever they evoke some-
thing beyond themselves. Symbolic systems such as programming code might
provide a large amount of information if one understands the code and very
little if one does not. In other cases, the informational dimension can support
the expressive or the environmental dimensions. For example, a piano record-
ing leads us to imagine a piano and a pianist, and the recording of singing birds
evokes a natural environment. Semantic categorisation might be less obvious in
narrative film; it is useful to draw from Leo Braudy’s distinction between ‘closed
frame’ and ‘open frame.’63 A wide shot provides all the information necessary to
interpret the image, leading attention to focus within the limits of the frame.
Conversely, a close-up does not describe a scene, leading imagination to build
what is not seen within the frame. In other words, a wide-shot of a landscape
conveys expressive semantics, and a close-up of a person or object conveys envi-
ronmental semantics.
The dynamics of sound and image have their own semantics, be it expressive
or environmental. To parameterise these dynamics, we use the taxonomy of con-
tinuities and discontinuities. Steady and progressive continuities create a sense
of environment because they fulfil expectations; attention can draw us towards
the context/environment. Ambivalent discontinuities also leave attention under
individual volition, but they entail more pathos; they attract more attention,
reinforcing the expressive dimension of the work. Radical discontinuities make
expressive semantics very strong: they prompt automatic attention, monopolis-
ing conscious awareness.
The informational load of the audio-visual relationship is proportional to its
ecological fit, determining the strength of perceptual binding. The binding is
informed by concepts of causation,64 which have informational load by defini-
tion. A high level of audio-visual fit provides a large amount of information,
254 Adriana Sá and Atau Tanaka

enabling integrated representations. Medium fit conveys a sense of causation,

but the informational load is not very high, as one does not understand the base
cause-effect relationships. Finally, low fit provides little information about cau-
sation, making the binding weak.
The semantics of the physical setup are also important in performances and
interactive installations. The central position of a performer, a sound source
placed next to them or a spotlight over them will have the effect of directing
attention to a central target, conveying expressive semantics. Conversely, the
distribution of sound and light sources in space will emphasise the environment.
A performer’s position relative to a moving image is equally inf luential. A
hidden performer confounds the extent to which the work is created in real time,
which means a decrease in informational semantics. For example, Ikeda rarely
takes the stage and is instead an invisible performer. The informational content
of his work relies not on the performative gesture, but in the data information
content of the sound and image – in his case often invoking a kind of informa-
tion overload.
With a separated arrangement, the semantics of the physical setup are more
expressive when the focus is on the performer, and more environmental when
the focus is upon the spatial relation between his body and the moving image.
We might consider the different arrangements of DJ/VJ setups as an example of
this. If the focus is on a DJ on stage, the interaction between sound and image
find an expressive arrangement. If, on the other hand, the DJ and VJ are not cen-
tre stage but are behind the dance f loor in the DJ booth, the visuals and music
might create a more environmental effect.
Finally, the semantics of an integrated arrangement can be expressive or envi-
ronmental. They are environmental whenever the visual output functions like
a stage scene and expressive whenever the physical scale of the work equals that
of the human body. In Chikashi Miyama’s Modulations, the performer’s body
movements create movement in a particle system (Figure 10.6). The performer’s
position, coincident with the centre point of the image, with projection as much
on the body as on the screen behind him, create an expressive intensity. As the
piece evolves and the particle system expands beyond the bounds of the body,
with the sound spatialised around the audience, the same audio-visual material
becomes environmental.
Sometimes, the semantics of the system output are highly expressive due to
sonic and visual discontinuities, while the integrated arrangement brings an
environmental quality to the work. An example is in Ikeda’s superposition,65 where
the performers are in front of a large visual projection, surrounded by multiple
video monitors. Another example is in a performance by Metamkine at the Lau-
sanne Underground Film & Music Festival (2012).66 The performers sit in front
of a large projection, using a Super 8 projector, colour filters and various devices
to create a multitude of light effects and noises (Figure 10.7). In contrast with
these works, Guy Sherwin created a series of silent performance films where
 A Parametric Model 255

FIGURE 10.6 Chikashi Miyama performing Modulations in the Kubus, ZKM, 2016.
Photo: Chikashi Miyama. Used by permission.

FIGURE 10.7 La Cellule d’Intervention Metamkine performing during Kill Your

Timid Notion tour, 2008. Photo: Bryony McIntyre. Used by permission.
Source: Bryony McIntyre <[email protected]>
256 Adriana Sá and Atau Tanaka

an integrated arrangement creates expressive semantics and the image exhibits

continuity. In these works, called Man with Mirror (1976–2011),67 he forged a kind
of ‘exquisite corpse’ by exploring the relation between his physical body and the
changing angles of its ref lection on a live-manipulated mirror.

Performative Arena
To complete our parametric model we need a final parameter: one that enables
us to summarise how the different semantic dimensions of a creative work inter-
twine so as to shape a performative arena. The performative arena corresponds to
how the work creates the potential space of presence. It can contract and expand,
inextricably related to attentional processes. Those processes might depend on
the characteristics of the sound, the image and the audio-visual relationship, on
the performer’s interaction with the system, the speakers’ placement, the spatial
relation between performer and visual projection, if any, the lighting, the physi-
cal architecture and the audience location. Individual predisposition might be
inf luential as well, but it does not depend on the work, and we do not intend to
parameterise experience itself. We rather consider verifiable variables and pro-
vide methods to interpret their relationships.
This final high-level parameter complements the other parameters, facilitat-
ing the disambiguation of certain aspects. It provides cues about elements that
have no direct representation, such as the placement of speakers and the lighting.
We distinguish three types of performative arena, which are not mutually
exclusive:

• Local arena means that the work conveys a focus upon the performer.
Expressive semantics are dominant.
• Distributed arena means that the work conveys a focus upon the environ-
ment. Environmental semantics are dominant.
• Extended arena means that the work conveys a subjective sense of presence
beyond the physical performance space. It requires perceptual cues, which
imply informational semantics.

Our parametric visualisation model uses three discrete points to represent these
three types of arenas.
The local arena relates to Ciciliani’s notion of ‘centripetal’ performance ten-
dencies, where the focus is upon the performer.68 An unequivocal example is
when a sound source is placed next to a musician. We can also revisit Sherwin’s
Man with Mirror,69 in which he integrates his physical image and its ref lection on
a mirror by using a light projector in a dark room, without any light ref lections
on the wall. The expressive scale of the work is reinforced with informational
load, as the interaction is clearly perceivable. Furthermore, a large visual pro-
jection can convey the local arena as well. It happens when the performer is
separated from the image and the image shows their interaction with the system.
 A Parametric Model 257

An example is in an expanded cinema performance by Arnont Nogyao (2017),70

where the visual projection shows his interaction with the modified surface of a
vinyl record.
Meanwhile, the distributed arena relates to Ciciliani’s ‘centrifugal’ performance
tendencies, where the focus is upon space.71 Niblock’s performances serve as
examples that create immersive environments. His drone music is played through
multiple loudspeakers distributed in space, and our eyes are directed to large
visual projections rather than to the performer, who sits in darkness.
While Birnbaum’s model72 and Ciciliani’s73 model consider the relationships
between the physical and the psychological space of a creative work, they do not
address how a work might expand one’s presence beyond physical space. Shift-
ing spatial presence beyond physical space requires perceptual cues derived from
the informational load of the sound, the image, or the audio-visual relationship.
An example of the extended arena is seen in Vasulka’s performance dedicated to
Nam June Paik,74 where the imagery recalls real-life situations: trees shaking in
the wind, Vasulka playing the violin, Michel Waisvisz playing The Hands. The
informal semantic dimension of that imagery is strong – by definition, video can
emulate how we naturally see the world. That enables a subjective sense of pres-
ence beyond the physical performance space.
The local and the distributed arena are often combined. In Lucier’s Music
for Solo Performer,75 attention is driven to the performer, who sits still in a cen-
tral position, using his brain interface to activate percussion instruments. At the
same time, attention is driven to the environment, because the instruments and
the loudspeakers are distributed in the room, amongst the audience. Another
imaginary example would be a performer carrying a TV monitor displaying
footage from the surrounding environment. Here, the distributed arena does not
require a distributed physical setup. Furthermore, a creative work can combine
all three types of arena. For example, Mick Grierson created a 3D composition
and improvisation system that behaves like first-person computer games.76 The
user can create, adapt and combine elements with varying physical attributes
to produce musical structures. The system does not exhibit pre-determined
constraints on the environment, object properties and interactions. It responds
to user behaviour through adaptive algorithms. We can speak of a local arena
because the performer is visible and the interaction design highlights their pos-
sible skills. We can also speak of a distributed arena because the system uses
a multichannel audio system. And finally we can speak of an extended arena
because the system looks and behaves like a first-person video game – it extends
the sense of presence to the digital world.
While the three types of arena ref lect the three semantic typologies, the
inverse is not necessarily true. That is to say, the local arena can be assumed
to be expressive, and the distributed area to be environmental. However, an
environmental semantic does not presuppose a distributed arena. For example,
in Steina Vasulka’s performance there is no local arena because the performer is
not visible, yet the visual discontinuities and sonic deviations create expressive
258 Adriana Sá and Atau Tanaka

semantics. Similarly, there can be environmental semantics without the arena

being distributed or extended, such as a sonic mass of drones emitted through a
single loudspeaker. Also, the semantics of an audio-visual performance can be
informational without the arena being extended, as happens in live coding per-
formances. Indeed, the parameters in our model complement each other without
redundancy. The high-level ones can be used to assess how any low-level vari-
able informs the meaning of the work and how the product informs the feeling
of presence.

Expression, Sensory Dominance and Spatial Presence

As part of a discussion about expression in audio-visual performance, it may be
useful to look at how the amount of interaction effort inf luences the semantics
and dynamics of the work. Our model visualises those expressive dimensions
on three axes: interaction, dynamics and semantics. This could play an important,
objective role in the development of an idiosyncratic, personal instrument. It was
the case in the individual development of our own instruments, as mentioned
earlier, which we plan to use in a future collaboration.77,78 We will present them
in the final section of this chapter so as to illustrate how our model can be used
not just in analysis of existing work but in the planning, composition and cre-
ative practice.
Sensory dominance is another dimension for creative manipulation, partic-
ularly if one wants the audience to experience complex sonic or visual con-
structions along with the audio-visual composite. We have sought to bring
clarification to the problem by drawing upon knowledge in neuroscience and
psychology.79,80 Accordingly, the parametric model reveals sensory dominance
through the axes ‘Audio-Visual Fit’ and ‘Visual Dynamics.’
Highly congruent audio-visual relationships produce integrated perceptual
representations, where vision subordinates audition.81 In fact, visual dominance
occurs at the moment we conceptualise causation.82,83 Perceptual prioritisation
can be inverted, or just become ambivalent when concepts are inconclusive.84
When the fit is too low to produce any sense of causation, one can bind sepa-
rate representations of sound and image, but that becomes demanding in terms
of cognitive load; hence, attention is likely to focus on making sense of the
audio-visual relationship, rather than on the music. Furthermore, sudden visual
discontinuities cause vision to subordinate audition, as they attract automatic
attention.85 Importantly, the threshold between ambivalent and radical disconti-
nuities should ref lect a particular timescale, since over time, the same behaviours
can appear more continuous or discontinuous. The parametric model can be
used to represent instantaneous moments as much as summarising large-scale
compositional structures.
An important element in any performance is a consideration of the audi-
ence’s subjective sense of presence. We adopt the notion of spatial presence as a
cognitive sense related to semantics and attention. The parameter ‘Performative
 A Parametric Model 259

Arena’ represents how the mediated space of an audio-visual performance may

draw focus on the performer, on the environment or on imaginary spaces beyond
the physical performance space. As a high-level parameter, it also encapsulates
information from other elemental parameters. By consolidating them, the per-
formative arena may allow the model to convey forms of experience that a single
parameter by itself may not specify. In this way, spatial presence can be inferred
from the model as a whole.

Using the Model in Creative Practice

We have shown how the parametric visualisation model can be used to analyse
existing audio-visual instruments. It also provides a theoretical perspective from
which to create new audio-visual performances and develop new audio-visual
systems.86 We have applied the model in the development of the audio-visual
instrument mentioned earlier in this chapter, which processes 3D sound and
image based on an acoustic zither input.87,88 We also used the model to imag-
ine how a set of musical collaborations would affect the audio-visual perfor-
mance work as a whole.89,90 The interaction effort, the visual dynamics, the level
of audio-visual fit and the physical setup would remain similar, but the sonic
dynamics and the global semantics would be quite variable. The model can put
in objective perspective how an audio-visual instrument might create a particu-
lar type of subjective experience and simultaneously be expansive in enabling the
discovery of new creative possibilities.
The model can also be used as a compositional tool. As an example, it has
served as a communication medium in a remote, long-distance discussion to plan
future creative collaborations. In this work, we intend to articulate a new version
of Sá’s audio-visual instrument and Tanaka’s EMG biosignal instrument. In Fig-
ure 10.8 we see a schematisation of a performance structured in three sections.
The first section (Figure 10.8 top) has no visual projection; there are only two
small lights on stage. The performative arena is both local and distributed because
the audience sits directed to the performers and the loudspeakers are distributed
in the room. The sonic construction explores ambivalent discontinuities, invit-
ing attention to focus on the subtle intertwining of sonic emissions – the zither
played with a bow and the electronic sound textures created with the electro-
myography (EMG). Both instruments require medium interaction effort, and
there is a fungible relationship between the visible physical gestures and the sonic
output: the audience senses causation without understanding the base cause-
and-effect relationships. The recognisable sound of an acoustic string instrument
conveys informational semantics, but the sound produced by the EMG does not,
and the two are often undistinguishable. Whilst the local arena and the musical
volatility convey expressive semantics, the distributed arena and the immersive
sonic continuity convey environmental semantics.
In the second section (Figure 10.8 middle) the sonic construction entails
radical discontinuities, which prompt automatic attention. These alternate with
260 Adriana Sá and Atau Tanaka

FIGURE 10.8 Top: first section of the performance. Middle: second section of the
performance. Bottom: third section of the performance.
Source: Chapter authors.
 A Parametric Model 261

silences of variable duration; the combination does not really form a pattern,
because the music maintains its organic, volatile qualities. While the EMG pro-
duces electronic sounds that are silenced in sudden manner, the zither is played
with pick and slider, activating pre-recorded urban sounds. The interaction with
the audio-visual instrument becomes slightly more effortful when the visual
projection comes in: the digital mappings rule out radical visual discontinui-
ties so as to avoid visual dominance, but the performer’s attention must spread
beyond the sonic construction, even if only occasionally to press a button. The
projected image is abstract, and its shape is much smaller than the screen. It
dislocates around and in between the performers, creating a wealth of progres-
sive continuities. The physical bodies are sometimes integrated, and other times
separated from that visual shape, forming a wealth of simple and simultaneously
complex visual effects. The audio-visual relationship remains fungible: there are
synchronised and non-synchronised audio-visual events, and the global com-
plexity circumvents the human tendency to prioritise causal percepts. Whilst the
sonic discontinuities strengthen the expressive dimension of work, the recogni-
sable street sounds strengthen the informational dimension in a way that extends
the performative arena beyond the physical performance space.
In the third section (Figure 10.8 bottom) the image increases in size, with
progressive continuity; it ends up covering the whole screen. The large-scale
projection over the performers becomes a reactive stage scene, and this new type
of integrated arrangement reinforces the environmental qualities of the work.
The semantics of the sonic construction are strongly environmental as well. The
zither is dribbled, activating sounds of nature, and these merge with electronic
continuities produced by the EMG. The soundscape is dense and rich in ambiva-
lent discontinuities. Attention is invited to focus on the wealth of sonic details,
while the multiple emissions interlace like a braid, emerging and submerging
from each other. The audio-visual relationship remains fungible, but overall,
there is a decrease in informational semantics because the sensorial complexity
makes the cause-and-effect relationships now definitely indistinguishable – the
audience should not even try to understand, just feel.

Conclusion
We have presented a parametric visualisation model as a means to represent in
abstract form audio-visual artworks. Each parameter has been described sepa-
rately and in their interdependent relationship with other parameters. Examples
from the history of audio-visual art have been used to illustrate the parameters.
Furthermore, we showed that the model provides an operational means to anal-
yse the relationship between sonic expression, sensory dominance and spatial
presence. That relationship is crucial in any audio-visual performance language,
regardless of its particular sphere of creative concerns.
Clearly, the model is very useful as an analytical tool, applicable to any tech-
nical platform, aesthetical approach and physical setup. By using this tool in
262 Adriana Sá and Atau Tanaka

analysis, we are able to discuss together a disparate series of works using a com-
mon framework.
The model has another useful function: it can work as a compositional tool. In
the final section of the chapter, we showed how it has been applied in planning a
new duo performance by the authors. The representation serves as a kind of score
for sections of the new work. It serves to create a common terminology to tie
together performance practice on two very different instruments – an electrified
zither and a muscle EMG instrument – and two distinct forms of output – digital
audio processing of acoustical signals connected to 3D computer graphics and
pure sound synthesis.
The model has been useful to set out an overall performance structure. It can
be further used to score the piece with more detail, leaving an open space for the
choice of sonic and visual materials, audio-visual mappings and individual tim-
ings; each parameter can summarise several aspects of the work, and one can also
use the model to analyse each aspect independently. This is particularly useful
because we desire to rely on a grounding structure and simultaneously believe in
real-time motivations for expression.
Beyond our creative work, the model is potentially useful to any audio-visual
practitioner. It can be used to analyse existing instruments, create new audio-
visual performances and develop new audio-visual systems.

Notes
1 David Birnbaum, Rebecca Fiebrink, Joseph Malloch, and Marcelo M. Wanderley,
“Towards a Dimension Space for Musical Devices,” in Proceedings of the International Con-
ference on New Interfaces for Musical Expression (Vancouver, BC, Canada, 2005).
2 Thor Magnusson, “An Epistemic Dimension Space for Musical Devices,” in Proceedings
of the Conference on New Interfaces for Musical Expression (Sydney, Australia, 2010).
3 Marko Ciciliani and Zenon Mojzysz, “Evaluating a Method for the Analysis of Perfor-
mance Practices in Electronic Music,” in Proceedings of the Second International Conference
for Live Interfaces (Porto, Portugal, 2015).
4 Salome Voegelin, Listening to Noise and Silence: Towards a Philosophy of Sound Art (New
York, London: Continuum, 2010).
5 Francisco López, official Web Site, “Against the Stage,” February 2004, www.
franciscolopez.net/stage.html (Accessed February 7, 2020).
6 John V. Draper, David B. Kaber, and John M. Usher, “Telepresence,” in Human Factors,
40(3), 1998, 354–375.
7 Thomas W. Schubert, “A New Conception of Spatial Presence: Once again, with Feel-
ing,” in Communication Theory, 19(2), 2009, 161–187.
8 Paul Dourish, Where the Action Is: The Foundations of Embodied Interaction (Cambridge,
MA: MIT Press, 2004).
9 Ana Sacau, Jari Laarni, and Tilo Hartmann, “Influence of Individual Factors on Pres-
ence,” in Computers in Human Behavior, 24(5), Elsevier Science Publishers B. V., 2008,
2255–2273.
10 Marko Ciciliani and Zenon Mojzysz, “Evaluating a Method for the Analysis of Perfor-
mance Practices in Electronic Music,” in Proceedings of the Second International Conference
for Live Interfaces (Porto, Portugal, 2015), ISBN 978–989–746–060–9.
11 Simon Emmerson, Living Electronic Music (Hampshire, UK: Ashgate Publishing Limited,
2007).
 A Parametric Model 263

12 Simon Emmerson, Living Electronic Music (Hampshire, UK: Ashgate Publishing Limited,
2007).
13 Michel Chion, Audio-Vision: Sound on Screen, translated by C. Gorbman (New York:
Columbia University Press, 1994).
14 Merilyn Brakhage, “On Stan Brakhage and Visual Music,” Vantage Point, January 31,
2008, http://vantagepointmagazine.wordpress.com/2008/01/31/on-stan-brakhage-
and-visual-music.
15 Salomé Voegelin, Listening to Noise and Silence: Towards a Philosophy of Sound Art (New
York, London: Continuum, 2010).
16 Pierre Schaeffer, Traité des Objets Musicaux: Essai Interdisciplines (Paris: Éditions du Seuil,
1966).
17 Jeff Pressing, “Some Perspectives on Performed Sound and Music in Virtual Environ-
ments,” Presence, 6(4), 1997, 8.
18 Meghan Stevens, “Music and Image in Concert,” Music and Media, 2009, 3.
19 Scott Sinnett, Charles Spence, and Salvador Soto-Faraco, “Visual Dominance and
Attention: The Colavita Effect Revisited,” in Perception & Psychophysics, 69(5), 2007,
673–686.
20 Tod Machover, “Beyond Guitar Hero: Towards a New Musical Ecology,” In RSA Jour-
nal, London, 2009.
21 Joel Ryan, “Some Remarks on Musical Instrument Design at STEIM,” Contemporary
Music Review, 6(1), 1991, 3–17.
22 David Birnbaum, Rebecca Fiebrink, Joseph Malloch, and Marcelo M. Wanderley,
“Towards a Dimension Space for Musical Devices,” in Proceedings of the International Con-
ference on New Interfaces for Musical Expression (Vancouver, BC, Canada, 2005).
23 Thor Magnusson, “An Epistemic Dimension Space for Musical Devices,” in Proceedings
of the Conference on New Interfaces for Musical Expression (Sydney, Australia, 2010).
24 Adriana Sá, “Designing Musical Expression,” in Proceedings of xCoAx: Conference on
Computation, Communication, Aesthetics & X (Lisbon, Portugal, 2017). ISBN 978–989–
746–128–6.
25 Phill Niblock’s, Movements of People Working, www.youtube.com/watch?v=mCKtqsy9gcY
and www.youtube.com/watch?v=LJtIZskdHOc.
26 Alvin Lucier, Music for Solo Performer, http://daily.redbullmusicacademy.com/2017/05/
alvin-lucier-music-for-solo-performer.
27 1986 interview with Ev Grimes for Yale’s Oral History of American Music.
28 Jeff Pressing, “Cognitive Complexity and the Structure of Musical Patterns,” in J. Slo-
boda, ed., Generative Processes in Music: The Psychology of Performance, Improvisation, and
Composition (New York: Oxford University Press, 1987), 129–178.
29 Performance by Joel Ryanand Evan Parker – www.youtube.com/watch?v=IQ4Dq
RgtbHc.
30 Performance by Steina Vasulka – www.youtube.com/watch?v=9mg1weOHWSs.
31 Steina Vasulka´s Violin Power series – www.vasulka.org/Steina/Steina_ViolinPower/
ViolinPower.html.
32 Video about Martin Howse – https://vimeo.com/54006161.
33 Adriana Sá, “How an Audio-Visual Instrument Can Foster the Sonic Experience,” Live
Visuals for Performance, Gaming, Installation, and Electronic Environments, Leonardo Electronic
Almanac, 19(3), MIT Press, 2013.
34 Eric I. Knudsen, “Fundamental Components of Attention,” Annual Review of Neurosci-
ence, 30(1), 2007, 57–78.
35 CD released by Experimental Intermedia Label.
36 La Monte Young and Marian Zazeela’s Dream House – www.youtube.com/watch?v=
3ahgq-zVQLc.
37 Gary Hill’s film Black and White Text – www.youtube.com/watch?v=bg1O3NcPwBg.
38 Thomas Wilfred’s Lumia – www.youtube.com/watch?v=ojVX8FWYc4g.
39 Phil Niblock’s Movements of People Working – https://expcinema.org/site/en/dvd/
phil-niblock-movement-people-working.
264 Adriana Sá and Atau Tanaka

40 Steina Vasulka’s Heraldic View – www.fondation-langlois.org/html/e/page.php?Num

Page=481.
41 Ryoji Ikeda’s Superposition – https://vimeo.com/49873167.
42 Michel Chion, Audio-Vision: Sound on Screen, translated by C. Gorbman (New York:
Columbia University Press, 1994).
43 Michael Schutz and Michael Kubovy, “Causality and Cross-Modal Integration,” Journal
of Experimental Psychology: Human Perception and Performance, 35(6), 2009, 1791–1810.
44 Michael Kubovy and Michael Schutz, “Audio-Visual Objects,” Review of Philosophy and
Psychology, 1(1), Springer Science and Business Media B.V., 2010, 41–61.
45 Scott W. Brown and Marilyn Boltz, “Attentional Processes in Time Perception: Effects
of Mental Workload and Event Structure,” Journal of Experimental Psychology: Human
Perception and Performance, 28, 2002, 600–615.
46 Adriana Sá, Baptiste Caramieux, and Atau Tanaka, “The Fungible Audio-Visual Map-
ping and Its Experience,” Journal of Science and Technology of the Arts, 6(1), 2014, 85–96,
http://dx.doi.org/10.7559/citarj.v6i1.131.
47 Norman McLaren’s Dots – http://vimeo.com/15919138.
48 Steina Vasulka’s Noise Fields – www.fondation-langlois.org/html/e/page.php?NumPage=483.
49 William Moritz, “The Dream of Color Music, and Machines That Made It Possi-
ble,” Animation World Magazine, April 1, 1997, www.awn.com/mag/issue2.1/articles/
moritz2.1.html (Accessed August 30, 2016).
50 Tarik Barri’s Versum – http://tarikbarri.nl/projects/versum.
51 Audio-visual instrument by Adriana Sá – http://adrianasa.planetaclix.pt/research/practice
Overview.htm.
52 Adriana Sá, “How an Audio-Visual Instrument Can Foster the Sonic Experience,” Live
Visuals for Performance, Gaming, Installation, and Electronic Environments, Leonardo Electronic
Almanac, 19(3), MIT Press, 2013.
53 Adriana Sá, “Repurposing Video Game Software for Musical Expression: A Perceptual
Approach,” in Proceedings of NIME: New Interfaces for Musical Expression (London, 2014),
331–334.
54 Adriana Sá, “Designing Musical Expression,” in Proceedings of xCoAx: Conference on
Computation, Communication, Aesthetics & X (Lisbon, Portugal, 2017). ISBN 978–989–
746–128–6.
55 Performance by Sensors Sonics Sights – https://babiole.net/en/sss-sensors-sonics-sights/.
56 Atau Tanaka, “Back to the Cross-Modal Object: A Look Back at Early Audiovisual
Performance Through the Lens of Objecthood,” Live Visuals for Performance, Gaming,
Installation, and Electronic Environments, Leonardo Electronic Almanac, 19(3), MIT Press,
2013.
57 Adriana Sá, Joel Ryan, Edwin van der Heide, Atau Tanaka, Andrew McPherson, Thor
Magnusson, Alex McLean, Miguel Carvalhais, and Mick Gierson, “Live Interfaces:
Seeds of Debate,” in Proceedings of INTER-FACE: ICLI 2014, 2015, 14–28.
58 Thor Magnusson, “Improvising with the Threnoscope: Integrating Code, Hardware,
GUI, Network, and Graphic Scores,” in Proceedings of the Conference of New Interfaces for
New Expression (London, 2014).
59 Adriana Sá, Joel Ryan, Edwin van der Heide, Atau Tanaka, Andrew McPherson, Thor
Magnusson, Alex McLean, Miguel Carvalhais, and Mick Gierson, “Live Interfaces:
Seeds of Debate,” in Proceedings of INTER-FACE: ICLI 2014, 2015, 14–28.
60 John Dewey, Art as Experience (New York: Penguin Putnam. Harvard (18th ed.), 1980).
61 Mark Johnson, The Meaning of the Body: Aesthetics of Human Understanding (Chicago:
University of Chicago Press, 2007).
62 Jeff Pressing, “Some Perspectives on Performed Sound and Music in Virtual Environ-
ments,” Presence, 6(4), 1997.
63 Leo Braudy, The World in a Frame (Garden City: Anchor Books, 1977).
64 Michael Kubovy and Michael Schutz, “Audio-Visual Objects,” Review of Philosophy and
Psychology, 1(1), Springer Science and Business Media B.V., 2010, 41–61.
 A Parametric Model 265

65 Ikeda’s Superposition – https://vimeo.com/49873167.

66 Performance by Metamkine – www.youtube.com/watch?v=xWwVvICGeR4.
67 Guy Sherwin’s Man with Mirror – https://vimeo.com/31609396.
68 Marko Ciciliani and Zenon Mojzysz, “Evaluating a Method for the Analysis of Perfor-
mance Practices in Electronic Music,” in Proceedings of the Second International Conference
for Live Interfaces (Porto, Portugal, 2015), ISBN 978–989–746–060–9.
69 Guy Sherwin’s Man with Mirror – https://vimeo.com/31609396.
70 Performance by Arnont Nogyao – https://vimeo.com/242720751.
71 Marko Ciciliani and Zenon Mojzysz, “Evaluating a Method for the Analysis of Perfor-
mance Practices in Electronic Music,” in Proceedings of the Second International Conference
for Live Interfaces (Porto, Portugal, 2015), ISBN 978–989–746–060–9.
72 David Birnbaum, Rebecca Fiebrink, Joseph Malloch, and Marcelo M. Wanderley,
“Towards a Dimension Space for Musical Devices,” in Proceedings of the International Con-
ference on New Interfaces for Musical Expression (Vancouver, BC, Canada, 2005).
73 Marko Ciciliani and Zenon Mojzysz, “Evaluating a Method for the Analysis of Perfor-
mance Practices in Electronic Music,” in Proceedings of the Second International Conference
for Live Interfaces (Porto, Portugal, 2015), ISBN 978–989–746–060–9.
74 Steina Vasulka’s performance – www.youtube.com/watch?v=9mg1weOHWSs.
75 Alvin Lucier’s, Music for Solo Performer, http://daily.redbullmusicacademy.com/2017/05/
alvin-lucier-music-for-solo-performer.
76 Mick Grierson, “Noisescape: A 3d Audiovisual Multi-User Composition Environment,”
in J. Birringer, T. Dumke, K. Nicolai, eds., The World as Virtual Environment (Dresden:
Trans-Media-Akademie Hellerau, 2007), 160–168.
77 Adriana Sá, “Repurposing Video Game Software for Musical Expression: A Percep-
tual Approach,” in Proceedings of New Interfaces for Musical Expression (London, 2014),
331–334, http://nime.org/proceedings/2014/nime2014_343.pdf.
78 Atau Tanaka, “Back to the Cross-Modal Object: A Look Back at Early Audiovisual Per-
formance through the Lens of Objecthood,” Live Visuals for Performance, Gaming, Instal-
lation, and Electronic Environments, Leonardo Electronic Almanac, 19(3), MIT Press, 2013).
79 Adriana Sá, “How an Audio-Visual Instrument Can Foster the Sonic Experience,” Live
Visuals, Leonardo Electronic Almanac, 19(3), MIT Press, January 2013, 284–305.
80 Adriana Sá, A Perceptual Approach to Audio-Visual Instrument Design, Composition and
Performance. Research Online, Goldsmiths, University of London, http://research.gold.
ac.uk/19431/.
81 Merilyn Boltz, “The Cognitive Processing of Film and Musical Soundtracks,” in Mem-
ory & Cognition, 32, 2004, 1194–1205.
82 Charles Spence, “Explaining the Colavita Effect,” in N. Srinivasan, ed., Progress in Brain
Research, vol. 176 (Amsterdam: Elsevier, 2009), 245–258, DOI: 10.1016/S0079-6123(09)
17615-X.
83 Christopher Robinson, Wesley Barnhart, and Samuel Rivera, “Auditory Stimuli Slow
Down Responses and First Fixations: Support for Auditory Dominance in Adults,” in
37th Annual Meeting of the Cognitive Science Society (Pasadena, CA, 2015), DOI: 10.13140/
RG.2.1.1564.5286.
84 Adriana Sá, Baptiste Caramieux, and Atau Tanaka, “The Fungible Audio-Visual Map-
ping and Its Experience,” in Journal of Science and Technology of the Arts, 6(1), 2014, 85–96,
http://dx.doi.org/10.7559/citarj.v6i1.131.
85 Adriana Sá, “How an Audio-Visual Instrument Can Foster the Sonic Experience,” Live
Visuals for Performance, Gaming, Installation, and Electronic Environments, Leonardo Electronic
Almanac, 19(3), MIT Press, 2013.
86 Adriana Sá, A Perceptual Approach to Audio-Visual Instrument Design, Composition and
Performance. Research Online, Goldsmiths, University of London, http://research.gold.
ac.uk/19431/.
87 Audio-visual instrument developed by Adriana Sá -http://adrianasa.planetaclix.pt/
research/practiceOverview.htm.
266 Adriana Sá and Atau Tanaka

88 Adriana Sá, A Perceptual Approach to Audio-Visual Instrument Design, Composition and

Performance. Research Online, Goldsmiths, University of London, http://research.gold.
ac.uk/19431/.
89 Audio recordings from the instrument developed by Adriana Sá – http://adrianasa.
planetaclix.pt/research/ArpeggioDetuning.htm#music.
90 Adriana Sá, A Perceptual Approach to Audio-Visual Instrument Design, Composition and
Performance. Research Online, Goldsmiths, University of London, http://research.gold.
ac.uk/19431/.
11
PRESENCE AND LIVE VISUAL
PERFORMANCE
Donna Leishman

Introduction
This chapter will provide a contextual overview of current discourse in presence
studies and offer insight into the subjective experience of Live Visual perfor-
mance (LVP), predominately from an audience perspective. Presence affects both
the physiological and the psychological. Presence as a subject is simultaneously
located within measurement-based scientific investigation, subjective artistic
exploration and objective/subjective orientated technological studies. Given the
blend between the physical and digital interactions now intrinsic to much of
Western society, the study of presence is complex and increasingly important.
The effects of presence have implications for the ways in which humans work,
play and our wellbeing.
The extent to which our culture, senses and biological limitations affect pres-
ence (in which order and which magnitude) is a long-debated subject with a
lineage as far back as the 6th century, and not one we can resolve in this text.
Rather, the goal of this chapter is to explore what role presence has in LVP and
begin to explore, alongside its practical and commercial uses, what needs might
presence fulfil for the audience.
When exploring the nature of presence and LVP, we note that as creative
practice, LVP can straddle and be situated in different forms such as dance music
and/or the avant-garde performing arts, both of which can draw on formal
properties of further musical genres and artistic disciplines such as performance
art, video, cinema, media art and a host of design disciplines. For the purposes
of the discussion here, I will use a broad definition: contemporary LVP is medi-
ated experience fostered by the intermedial space of merged sound and image,
shared during a live, specific, time-bound performance that will eventually
come to an end.
DOI: 10.4324/9781003282396-14
268 Donna Leishman

Section 1: Key Definitions

Lombard and Ditton (1997) offered the inf luential definition of presence as “the
perceptual illusion of nonmediation.”1 The term “perceptual” refers to a con-
tinuous, real-time response of all the human sensory, cognitive and affective pro-
cessing systems that enables perception. They argue that for “presence” to occur,
we engage in the “illusion of nonmediation.” Put simply, when an experience
where mediation is occurring, we are no longer aware of its effect.
Slater and Usoh,2 channelling the poet Coleridge,3 also describe this as a
temporary “suspension of disbelief ” typically felt when the individual feels that
they are in a world other than where their real bodies are located. For Lombard
and Ditton other characteristics are also important to note; presence is not a
by-product of perception malfunction (say, through, psychoactive chemicals),
psychosis or a psychiatric hyper-focus: it also requires a facilitation by a human-
made medium/technology. These caveats separate out their definition of pres-
ence from the fields of sociology and psychology, which also discuss presence but
inclusive of a human-to-human paradigm.
While many scholars and technologists focus on the fully ‘open window’
model where any mediating factors have entirely disappeared in the audience’s
mental processing, within the LVP context this is rarely the desired intent; more
common is a type of presence where the mediating and merged technologies are
overt (i.e. large-format digital projection, screens, specialist lighting and sound
systems are salient rather than functioning as transparent mediations).

A Note on the Role of Human-Made Technology

Whilst this chapter will predominately work with Lombard and Ditton’s defini-
tion, it is also worth a quick note on some foundational principles. Philosophers
and psychologists point out that presence is a key part of human consciousness.
Platonic-Cartesian philosophy makes the distinction between the world as seen
by the eyes on one’s head and the world of the mind’s eye. The early work of
Plato (427–347 BC) and Descartes (1596–1650) showed that whilst they diverged
with regard to the role of the human ‘soul,’ together they helped to drive the
concept of “objective rationalism,” an epistemological view that reason is the
chief source (and test) of knowledge. This view endured through to the Enlight-
enment into today’s general culture: that our experiences are always mediated
by our sensory and perceptual systems. The human biological infrastructure is
naturalised (at least for those who can afford to take our bodies for granted) to
such an extent it no longer is foregrounded in our consciousness. Within LVP
the audience has this intrinsic biological mediation, and potentially in the other
mediums mediating the LVP environment, such as normal assistive technologies
like eyeglasses or hearing aids, or more enhanced material additions such as 3D
glasses (Figure 11.1). We may refer to these factors in relation to the discussion
but will primarily frame its explorations around the audience’s experience of
 Presence and Live Visual Performance 269

FIGURE 11.1 Novak Collective’s 3D Disco is an interesting example of the audience

being offered additional mediation via cardboard diffraction glasses to
inf luence their perception of the LVP.
Source: Novak Collective. Used by permission.

mediation and Lombard and Ditton’s foundational definition of “the perceptual

illusion of nonmediation” and the key role of additional human-made digital
technology within that experience.

Lombard and Ditton’s Categorisations

Alongside their inf luential definition of the notion of presence, Lombard and
Ditton also offer a comprehensive review of the common conceptualisations or
characteristics of presence. These are presence as a form of social richness, as real-
ism, as transportation, as immersion and as a social actor. They start with pres-
ence as ‘social richness’ whereby the experience of presence is normally a focus/
concern in large organisational contexts, this type of presence creates a sense of
satisfaction through interpersonal (person-to-person) intimacy and immediacy.
For example, a water cooler or kitchen kettle could be envisaged as mediums-
for-presence within a generic large office block context, as objects such as these
facilitate a change in human-to-human proximity, eye contact, intimacy of con-
versation topic, etc. Within LVP the coming together in the evening for a live
performance also symbolises a change of context in which interpersonal dynam-
ics and normal individuality dissolve through the crowd structure.
A second conceptualisation of presence is one that remains key in the enter-
tainment, science and military industries – presence as ‘realism.’ This con-
cerns how well the medium/technology can produce compellingly accurate
270 Donna Leishman

representations of objects, events and people. Reality and screen-mediated pres-

ence as a scholarly subject has its origins in what was termed the broadcast or
network era. From the 1950s onwards the prevalence of the household medium
of the television increased and in turn became a contested site for audiences to
access key public information, ideas and potentially ‘unreal’ and thus socially
harmful representations of reality. The concern about the insidious effects of
false realities is now writ large through the high levels of contemporary social
engagement through social media networks,4,5 a network set up to offer transpar-
ency through a shared insight into other peoples’ lives.
Marvin Minsky, co-founder of the Massachusetts Institute of Technology’s
artificial intelligence (AI) laboratory, helped drive and define televisual presence.
In his 1980s article “Telepresence” he denotes the quality of the illusion of real-
ism whilst introducing the term telepresence, which “emphasizes the importance
of high-quality sensory feedback and suggests future instruments that will feel
and work so much like our own hands that we won’t notice any significant dif-
ference.”6 Later studies showed that the scale of the screen and the definition of
the image and quality of the audio also inf luence the success of the “sensation
of realism.” 7 Recent developments in ultra-high-definition televisions also sit
alongside a growth in home cinema and projection technologies, all of which
further extend opportunities for improved sensations, allowing content mak-
ers (motion pictures, television, video games) to develop detailed computational
simulations and sophisticated material with the goal of increasing viewer immer-
sion at home. This will inevitably contribute further to concerns over the conf la-
tion of represented reality and how to parse that which is “of nature”8 and not an
artificial construction. This concern around the extent to which human-made
technologies become normalised or invisible in their inf luence has been an ongo-
ing concern of media theorists and scholars of embodiment and contemporary
aesthetics and culture. Outside of the continuous increase and quality of image
and sound in video conferencing enabled by better bandwidth, the COVID-19
global pandemic9 further accelerated the growth in web conferencing systems
and helped, through necessity, to foster forms of “virtual togetherness,”10 sug-
gesting that societies may be moving towards a point in time when the screen
eventually dissipates in our perception as we have over time similarly learned to
ignore the body as a mediating factor.11
There has also been renewed interest in the potential of presence in virtual
reality (VR), exemplified by the social media giant Facebook’s acquisition of
the developer Oculus VR in 2014. Facebook has mass consumer reach as well as
funding for research and development. The commercial promise of VR is built
on the aspirational ideology of “if you can dream it, VR can make it.”12 VR has
a similar trajectory to television, but unlike TV and its complicated relationship
to communicating/inf luencing social reality, VR is associated with a certain
utopian escapism, correlating with the belief that society wants to escape from
the chores of modernity. This can be charted as far back as the 1980s to Min-
sky’s belief that telepresence will free humans from “hazardous and unpleasant
 Presence and Live Visual Performance 271

tasks,”13 Haraway’s “dream of the hope for a monstrous world without gender”14
or Murray’s notion that computational narratives will better reshape “the spec-
trum of narrative expression.”15 Another interesting difference to note is that
TV screen technology has endured, whereas public interest and funding of VR
significantly shuttered in the mid-1990s, when the chasm between public expec-
tations for the technology and the reality of limitations were too wide (Figure
11.2). This was not helped by a public whose interests were also being diverted
by the emergence of the internet.
Alongside presence as social richness and as realism, Lombard and Ditton cite
another key conceptualisation of presence, one that has deep roots in human
culture and storytelling. This is the ability of presence to be ‘transportive,’ the
artificial sense of being ‘here’ and taken ‘there/somewhere.’ This has become
a common and desirable phenomenon and can be achieved in low-sensory/
bandwidth mediums such as in books or oral storytelling.16 Rheingold described
this kind of presence/telepresence as a “form of out-of-the-body experience.”17
For Lombard and Ditton transportive presence is related to human-made tech-
nology rather than through linguistics or language.

FIGURE 11.2 A page from a Virtuality gaming system marketing piece showing the
Visette and controller from 1994.
Source: Image permission is given. The image is sourced via Wikipedia, Creative Commons Share
Alike version 4.0.
272 Donna Leishman

In VR, the medium of putting on goggles, gloves and eyeglasses to simul-

taneously go nowhere yet be transported anywhere has become so pervasive
it’s intrinsic to the medium. However, outside of science and perhaps military
applications of VR, social realism is not normally part of the design. The user’s
expectations for imagined, fantastic and escapist content still endure from VR’s
formative years.
Lombard and Ditton also discuss presence as immersion, an experience when
the senses become led by or dominated by the technology to such an extent
that other senses are stopped or impaired. Csikszentmihalyi’s (1975) concept of
‘f low’18 is a state of being when one is completely absorbed in an activity, espe-
cially seen in activities which involves your creative abilities. Csikszentmihalyi
notes that often for an individual being in this state is a route to experiencing
satisfaction or happiness. Other scholars such as Palmer19 and Quarrick 20 have
characterised the psychological immersion as occurring when their users felt
‘absorbed’ and ‘involved’ in a given task.
As discussed previously in Chapter 10, within LVPs participants experience
a pull in terms of attention either between sound and visuals or the synthesis of
forms in an intermedial space. Biocca 21 uses the vocabulary “sensory bandwidth”
to describe the different cognitive demands different technologies or environ-
ments make on the subject. For each sensory channel, stimuli from the physical
environment compete with stimuli from the virtual environment for the atten-
tion of the individual. Biocca goes on to define this as sensory saturation, defined
as “the percentage of the sensory channel occupied by stimuli (information) from
the virtual as opposed to the physical environment.”22 With these statements
in mind, given the explicit marriage of audio and visual in a fixed durational
performance (that cannot be paused or rewound), it is reasonable to describe
the form as medium-high-bandwidth experience. This is also dependent on the
complexity of re-contextualising the audio and visual material as an additional
layer and if there is any requirement to create any formal ‘meaning’ or under-
standing from the material.
Presence can also be regarded as a ‘social actor’ within a medium. Lombard
and Ditton cite Horton and Wohl,23 who coined the term ‘para-social interac-
tion’ when describing television presenters and their ability to offer a compelling
illusion of real-time communicative interaction for the purposes of fostering
false intimacy with their viewers. In such para-social instances, users’ percep-
tions and the resulting psychological processes lead them to illogically overlook
the mediated or even artificial nature of an entity and attempt to humanise and
seek out interaction with the non-human. Digital ‘social actors’ (who are pro-
grammed to enable false intimacies) now come in many forms: automated retail
assistants, avatars, voice-only entities and smart toys for children. These sit upon
insights that many people also have come to humanise their computers or smart
phones, all of which speaks to changes in technology and human vulnerabili-
ties. Set within the contemporary context of increased mixing of realities, this
trend is also a concern for social psychologists, sociologists and human-computer
 Presence and Live Visual Performance 273

interaction scholars who note both positive effects but also deleterious impacts
of socialisation and technology, which increasingly now has a defining role in
our private lives.24
It is possible that the nature of the audio-visual content can partially feel like
a human ‘character’ within the performance. The content can be portrayed as a
‘being,’ images can be rendered as figurative and audio can be lyrical, which in
turn can help to foster feelings of para-social intimacy. Depending on the light-
ing, visual design and representational content, the performer may feel integrated
into the visual composition to such an extent that their living self can be captured
via a live camera feed and fed into the library of media being shared in the live
performance. Alternatively, the performer can feel separated and discrete, more
akin to an ‘entity’ rather than an ‘identity’ on stage (see front cover), a feel-
ing perhaps reinforced if there is also a volume of surrounding computational
hardware. If the performer herself or himself includes their voice/image via live
capture and are rescreened as part of the audio-visual performance, this could
encourage audience ‘empathic observation,’ which is the act of “watching facial
expressions and body language in human exchanges to figure out what is going
on.”25 McConachie and Hart point out that this is not the same as reading the
body as a sign. “It is a mode of cognitive engagement involving mirror neurons
in the mind/brain that allow spectators to replicate emotions of a performer’s
physical state without experiencing that physical state directly.”26
Furthermore, the artist could also engineer within her or his live perfor-
mance ‘autopoiesis,’27 a state in which the audience is invited to participate in a
feedback loop within the live performance. The audience’s data can be gathered
and then directly affect the proceedings of the performance. An example of
this approach can be seen Golan Levin et al.’s Dialtones (A Telesymphony), a per-
formance of choreographed dialling and ringing of the audiences’ own mobile
phones.28 Whilst not explicitly para-social, autopoiesis as a performance concept
could feel like one is participating with a living system along with other humans
(the performer, the audience). The orchestration of “emotional intensity”29 of
the audience by the performer’s skill in changing the audio-visual content in
response to how she or he reads the audience also functions as a connection, a
form of intimacy – as the real-time manipulation of their emotions could feel
like a direct, personal connection since the individual believes that performer is
communicating with them.

Section 2: Culture
We now will consider the key socio-cultural circumstances that also inf luence
human experience. Human culture is socially constructed in fast-changing vari-
able form, often described as the ‘software,’ and society being the ‘hardware,’
taking longer to update or change over.30 For Ratner31 culture and psychology
are two elements of a larger integrated and interdependent system, alongside
biology and personal experience. Capitalism both shapes cultural artifacts and
274 Donna Leishman

concepts and in turn requires these elements to be put into the service to sus-
tain its economy. Typical cultural socialisation processes promote success, status,
self-image and the need to develop a market-driven identity. Considering this
context, Butler posits the need for more personal agency,32 and Turkle points out
that digital/online culture ostensibly offers an ‘always on’ increased social con-
nectivity but has resulted in a deeper sense of individualisation.33 For many, con-
temporary existence in these settings creates a cognitive gap between ourselves
and our wellbeing. This distance or gap is often termed the wedge of alienation
or “false consciousness,”34 a wedge that can come to inhibit personal happiness
or fulfilment. However, within LVP contexts, the audience can experience a
specific type of interpersonal socialisation and intimacy (social richness), poten-
tially fulfilling a sense of connection missing within other lived experiences.
The ongoing saturation with screens in contemporary culture (and the medi-
ation of relationships through technology) and online/mobile content, increas-
ingly repositions the ‘live’ event as a diametric contrast to these behaviours.
The time-bound nature of live experiences makes them a rarefied phenomenon,
requiring the audience’s attention to focus ‘in the moment’ rather than common
distracted, multitasking inattention.
Theatre, dance and musical performances have traditionally been conceived
as primarily passive experiences,35 whilst the performers and audience may be
aware of each other and that ‘awareness’ may affect the emotional charge of the
proceedings, this is quite different within audio-visual live performance, which
ordinarily requires significantly more mental bandwidth to actively observe and
participate in what Lusch and Vargo refer to as the “co-creation of value.”36

FIGURE 11.3 An example of a complex audio-visual performance via space mapping,

Amon Tobin’s ISAM tour 2011, Roundhouse performance.
Source: Image credit: Valerio Berdini @liveon35mm. Used by permission.
 Presence and Live Visual Performance 275

This active participation is part of the knowledge that ‘live’ experiences are
different from the experience of being part of the audience for non-live arts. Lad-
bourne et al. describes the nature of ‘live-ness’ as a shared experience, a perfor-
mance that “is part of you and you are part of it,”37 and for many the irreversible
nature of Live Visual performance is one of the primary drivers in our attention/
focus – a form of positive attention, and a variation of Csikszentmihalyi’s concept
of ‘f low’ through immersion in creative activities.
The human experience of live performance can also create what Hirschman
and Holbrook termed a “hedonic response.”38 Santoro and Troilo,39 drawing on
the work of Lacher and Mizerski,40 define the hedonic response as “a combined
response from the emotions, senses, imagination, and intellect” (see Figure 11.4)
and argue that “consumers expect . . . [hedonic] products and services to create
an absorbing experience arousing their emotions, stimulating a physical reaction,
soliciting their memories and fantasies, and triggering their cognitive develop-
ment.” This concept taps directly into the importance of engaging an inner life, a
term typically understood as a non-technological yet virtualised set of intangible
experiences such as dreaming, fantasies, daydreaming or ‘reveries.’41
Individuals may choose to attend an LVP to experience emotions and have
their audio-visual intermedia senses stimulated; however, this definition of
hedonism is more nuanced than the common simplification of hedonism as a
reckless pleasure-seeking activity. Singer, channelling Nozick’s The Experience
Machine from 1974, says that “people that don’t aim at pleasure, but aim at some-
thing else, some activity that’s worthwhile in itself, and they get absorbed in
the moment of doing what they’re doing . . . they actually get enjoyment and

FIGURE 11.4 An example of audience hedonism in action within audio-visual culture.

TNGHT performing in San Francisco 20.4.2013.
Source: Image credit: Flickr YetiLegs Annika Kinsman CC by 2.0.
276 Donna Leishman

fulfilment out of it.”42 This need to be fulfilled in leisure time becomes more
vital when work lives become less satisfying within common isolating43 social
tendencies of ‘advanced capitalism.’44
Within LVP, activeness of the communicative intersection between per-
former, performance and the audience is a key defining characteristic. Interest-
ingly, Hirschman and Holbrook note that “if consumers know in advance that
hedonic consumption will require a certain level of imaginal participation and
emotional expenditure, they may choose to use (or to avoid) a certain product,”45
Hirschman and Holbrook f lag the importance of audience expectations and the
level of informed knowledge necessary before engaging with a hedonic product.
Radbourne et al. point out that the better the audience’s prior understanding
of the arts event is, the greater the appreciation will be if expectations are met;
however, within LVPs many performers will intentionally iterate or improvise
during their performance as part of their practice. This fosters a sense of trust or
risk taking on the part of the audience.
Hedonic ‘products’ have also been associated with luxury and the arts. Hagt-
vedt and Patrick have discussed a hedonic artwork’s ability to create luxury per-
ceptions by referring to the notion that “art is intrinsically tied to a heritage of
high culture, with connotations of exclusivity, luxury, and sophistication.”46 This
phenomenon, which in turn is based on Veblen’s “conspicuous consumption,”47
is a concept directly borne out of the opportunities manifested for some in the
industrialisation age. To enact conspicuous consumption, an individual will pur-
chase, primarily for display purposes, expensive and tasteful commodities.
The notion of commodity self (i.e. that ourselves are constructed in part
through our consumption and use of commodities) can also be an element in
LVP. Performances and experiences are sold as products, events that have evolved
from a free, experimental avant-garde practice (even illegal and underground)
into the daylight of mainstream commodity culture. Music clubs, VJs and DJs
have become contemporary inf luencers and tastemakers, and in some instances
even brand identities in their own right. Mainstream Live Visual performances
can be described as “designed design culture,”48 showing the power of design
(see Chapter 15) and how it moulds with societal issues such as taste and behav-
iour in a commercial world.
In developed, industrialised societies the audience experience of performative
visual arts has become more meaningful, primarily fostered by the prevalence of
digital distribution and piracy,49 which has had the consequence of demonetising
the ability of new music releases to create money for artists and labels. This in
turn has transformed the commercial value placed on the live performance sec-
tor, which is now an important way to create a profit. LVP, given its ‘live-ness,’
is a unique ephemeral event, thus in turn creating additional notions of value and
worthiness. This has resulted in more mainstream investment and active market-
ing of live, real-time events. Live tickets are now able to command high prices
because of a product’s symbolic value. The MUTEK arts festival (2000–present),
based in Montréal, Canada, regularly features VJs alongside experimental sound
 Presence and Live Visual Performance 277

art performances. Day passes typically start at £40 to £70 and often are pur-
chased as a full weekend event (£80 to 140). However, a single big ‘brand’ club
night such as Ministry of Sound, London, would be priced at £60+ per unit.
These tickets are a luxury product, approximately six times more than a cinema
ticket. It is also common for further costs to be incurred alongside participating
in an LVP: for example, travel, accommodation and refreshments during the
performance. The recent COVID-19 pandemic and the temporary sector closure
has thrown a spotlight on both the nature of the appetite to participate50 in LVPs
and the economic value of ‘after dark economy’51 of which LVP is a key part.
Radbourne (2007) argues that “the new arts consumer is on a quest for self-
actualisation where the creative or cultural experience is expected to fulfil a
spiritual need that has very little to do with the traditional marketing plan of
an arts organisation.”52 Authenticity is typically associated with reality, truth and
believability, yet these qualities mean different things to different individuals;
perception matters if an experience is understood to be staged or genuinely
authentic, and thus the sense of authenticity varies considerably in LVP attend-
ees. Wang identifies three types of authenticity – objective, constructive and
existential authenticity – each of which has implications for the study of value
within the performing arts. This last category is particularly pertinent to LVPs.53
Wang explains: “In common sense terms, existential authenticity denotes a spe-
cial state of being in which one is true to oneself, and acts as a counter dose to the
loss of true self in public roles and public spheres in modern Western society.”54
Participants express that their sense of individual freedom and/or acceptance55
with LVP in nightclubs is one of the primary attractors to these events. Value
within LVP can thus be considered both as personal value (inner life value to the
individual) and within a socialising framework, where the public demonstration
of the selection and financial investment to attend a live performance is also an
extension of self-identity and contributes to how ones wishes to be perceived by
others.

Social Presence
Social presence refers to the extent to which other beings also exist within the
live experience and are understood to affect or react to the audience’s individ-
ual presence. Whilst the individual sensory conditions, fused with culture and
genre-based expectations, are the foundations in which presence is created, one
of the most distinguishing extra features of LVPs is ‘being’ within a group. This
social presence is incredibly important when considering presence and all its
permutations.
Live Visual performances are often offered either as large of mass audience
gatherings (see Figure 11.5) or as medium or small conspicuous environments
between the artist and the crowd; yet both culturally are understood to be col-
lective experiences. Each type of social space inf luences the collective experi-
ence of non-verbal communication. The ability to exchange eye contact, smiles,
278 Donna Leishman

FIGURE 11.5 An example of a very large concert crowd. Vasco Ross’s 2017 show at
Enzo Ferrari Park, Modena, Italy.
Source: Image credit: Italian Interior Ministry from Wikipedia, Creative Commons 3.0.

vocalisations or being able to explicitly talk with strangers during or afterwards

provides audience members with value because it “allow[s] private feelings to be
jointly expressed and reinforce[s] the sense that we are not alone.”56
The field of social psychology states that crowd behaviour is heavily inf lu-
enced by the diminished responsibility of the individual and the impression
(another temporary illusion) of universality of behaviour, both of which increase
with crowd size.57 In terms of purposes for existence, LVP audiences can be
described as an expressive group – i.e. people gathering for an active purpose
and not typically aggressive.58 Sociologist Herbert Blumer also pointed out that
crowds should be considered as dynamic changing (over time) systems of “emo-
tional intensity.”
The nature of groups is also discussed by anthropologist Victor Turner who
pointed out atavistic similarities between the “leisure genres of art and entertain-
ment in complex industrial societies and the rituals and myths of archaic, tribal
and early agrarian cultures.”59 Turner coined the notion of ‘communitas’ which
is often cited as a useful remedy to experiences of alienation. Communitas is
defined as “unstructured or rudimentarily structure [with] a relatively undiffer-
entiated comitatus, community, or even communion of equal individuals who
submit together to the general authority of the ritual elders.”60 It could be said
that the performer(s) within LVP act/symbolise (albeit temporarily) this role of
an elder, especially in instances where extra prominence is afforded through a
celebrity status – popular culture’s deification of an individual. This also poten-
tially feeds back into a sense of rarefication, liveness and commercial value of
LVPs. For Turner being in a state of communitas fosters a liberation of human
capacities of cognition, affect, volition and creativity from the normative con-
straints incumbent upon occupying a particular social status.61 Wang (1999) also
 Presence and Live Visual Performance 279

discusses the value of space to contribute to the experience of existential authen-

ticity wherein “structures fall apart, and differences arising out of the institution-
alized socioeconomic and socio-political positions, roles, and status disappear.”62
On the other hand Turner’s communitas, as a temporary escape from cognition,
seems counter to the higher-bandwidth cognitive parsing of live audio-visual
stimuli. However, the notion of a communion of equal individuals who submit
together to the general authority of the artist will resonate with LVP audiences.
Furthermore, the particular demands on the sensorial realm in live audio-visuals
mean many also have focused attention to an extent that very little else can
intellectually enter one’s mind, thus engendering a temporary escape from one’s
identity and potentially a loosening of personal inhibitions.
Scholars of social psychology also cite Festinger, Pepitone and Newcomb’s
term “deindividuation”63 to describe the effect of a crowd or group on the
behaviour of an individual. Festinger et al. claimed that in large group con-
texts an individual becomes able to undertake forms of behaviour in which,
when acting as an individual, they would not through a lessening of per-
sonal controls (e.g. guilt, shame, self-evaluating behaviour). Deindividuation
is often discussed alongside the role of anonymity and can be a factor in both
positive and negative changes in behaviour – an innocuous example would
be loud clapping (too conspicuous to do as an individual), the less healthy
would become disinhibited enough to join others in the audience in throwing
glass bottles at a performer. These dynamics of identity perception and crowd
behaviour do not mean all members of a group will act in the same manner;
instead, it is understood that participants will exist on a continuum, differ-
ing in their ability to deviate from social norms. Within LVP there could be
a convergence of interest within the group, who would thus share in their
expectations of the event.

Environmental Presence
The last consideration in this section is the extent to which the environment
either inf luences the audience’s understanding of presence, and in turn their
‘mental model,’ or if the environment itself appears to know that you are there
and reacts to your involvement, thus increasing the subjective sense of ‘being’
somewhere.64
Starting with physical environments – concert halls, alongside exhibition
spaces, are part of the built architecture of urban planning; their aesthetics, con-
cepts and materials are carefully crafted as part of the ongoing fabric of high
culture. However, the most common spaces associated with LVP such as night-
clubs, and music venues are regarded as unprepossessing environments, at best
neutral ‘raw’ blanks over which stimulating LVPs are created (and experienced).
In short, any architectural qualities of such spaces should not distract or overly
stimulate, but rather recede into the background and are passive within the audi-
ence’s cognitive, mental model.
280 Donna Leishman

The interior design of the nightclubs in the 1950s would have a stage, orches-
tra and dance f loor, but successive decades would see the orchestra repositioned
or dropped for a performers’ booth, and then later these were enhanced with
the addition of multiple screen technologies. In contemporary LVP spaces, the
performing artist and their hardware are ordinarily semi-elevated and positioned
up front to allow for better audience line of sight and framing by the set lighting,
large visual projections and or screens. This represents an ongoing digitalisation of
experience in terms of increased sophistication of programmable ambient light-
ing, the quality and size of the projections and screens, alongside the dynamics of
the sound system. These all work together to direct audience attention towards
the main audio-visual stimuli. The audience itself is normally not lit or given spe-
cial consideration in the interior design; furthermore, this low-lit ambience allows
individual participants to dissolve into a more homogeneous crowd identity.
Within LVPs it is not common (unlike art installations such as 1024’s Vortex
responsive sculpture65) that the environment in which the audience find them-
selves is able to sense and react to their presence. However, the visual represen-
tations on the screen could be a simulated environment, which may play some
role, if absorbing enough, in offering a sensation of being transported into the
environment. For this effect to work, representation would have to bypass the
established pitfalls of live performance whereby the visual content is normally
fragmented, remixed and short in duration.
Whilst the majority of venues might be cognitively neutral (with regard to
commanding attention), there are some notable venues that do offer LVP whilst
being environmentally inf luential. Such an example would be Berlin’s Berghain,
which as a space is often referred to as the ‘temple to techno.’ Berghain is archi-
tecturally striking as a towering former power station,66 its inner structures are
cavernous and its main room is intentionally industrial, featuring stripped-back
exposed concrete and steelwork. Berghain as a site also comes with a specific
narrative context and heritage which can be built upon; this sense of place
and heritage would both inf luence the audience’s expectations and awareness
of ‘being’ there geographically whilst also participating in the specifics of an
LVP. Manchester’s Haçienda club (1982–2002) was also a visually distinctive
interior space fostered by Ben Kelly’s graphics and reappropriation of outdoor
safety furniture.67 The Haçienda also, like Berghain, was loaded with culture
and heritage – its role through the 1990s was as a creative caldron for emerging
British musicians. Demolished in 2002, the Haçienda can now only be experi-
enced as a 3D simulation using VR technology,68 as this space has been rebuilt
from Kelly’s original plans. As an aside, this project is an interesting contribution
to the design histories of LVP given how scant organised archival documentation
surrounding the artists, the interiors, the ephemeral visuals and/or the audience’s
experience currently is (also see The Flashback Project).69
Whilst the concept of a “virtual nightclub” does exist, this, like most VR,
is designed to be experienced individually, but has suffered the same fate as
other transmediations where the translation of the original medium’s distinctive
 Presence and Live Visual Performance 281

qualities fall short of the VR system’s capabilities. Furthermore, current exam-

ples of virtual nightclubs are also gamifications of the social interactions. Some
examples: Berghain has offered a playful VR training simulation wherein70 the
player is transported outside the club and invited to practice trying to ‘get past’
the club’s notorious strict door policy/doorman – this was regarded as a self-
aware piece of clever marketing. There is also Oculus’ Dance Central VR which
offers synchronous online multiplayer dance competitions with a side order of
in-game ‘selfies’ and a virtual lounge. These feel like related products offered to
build anticipation for the real LVP, or in the case of Dance Central to commer-
cially hybrid the club premise into gaming culture.
The contemporary experience of LVP is built on a complex ongoing relation-
ship with our interior/exterior architecture and should be considered important
when exploring presence (see Chapter 15 on design and Live Visuals for further
discussion).

Conclusion
We have encountered many types of presence and the complexities that come
together to foster the mediation at play within Live Visual performances: social
richness, as realism, as transportation, as immersion and as a social actor. We
discussed how social realism and realism through high-fidelity visual simula-
tions is not a key aesthetic concern within the practice; similarly it is rare within
LVP that place making and transporting to and from explicit destinations or
environments occur, partially because this would be difficult to achieve given
LVP’s native formal qualities. However, on closer inspection, LVP is well placed
to offer forms of para-social experience built onto the core knowledge that the
performer will be, by virtue of it being a live performance, already responding
to the presence of the audience; however, if the performer is furthermore figura-
tively represented within the LVP (see Figure 11.3), this could afford extra cog-
nitive engagement through the mirroring of the emotions of the performer by
the audience. Additionally, there is the potential of being invited into an explicit
creative feedback loop in the form of autopoiesis, which can also open up posi-
tive sensations of being an agent within a living system.
LVP functions as a locus for “socially rich presence,” as it facilitates temporary
and pleasurable changes in social behaviour. LVP does not normally require audi-
ence direct interpersonal communication, but instead offers a distinctive group
membership, a Turnerian loosening of self-identity that places more emphasis on
the intrapersonal to engage emotionally and intellectually with variety of audio-
visual stimuli.
Technology in LVP can often be innovative in its presentation of content (e.g.
space mapping, 3D graphics, live mixing), and the audience, depending on their
level of technical knowledge, can regard these as displays of virtuosity or feel past
these complex formalities and embrace the personalised emotions that the perfor-
mance, performer and crowd offers to them to in turn complete the performance.
282 Donna Leishman

Like users of virtual reality hardware, within LVP, we also witness a keen audi-
ence willingness, perhaps even a need, to enter into this temporary suspension of
disbelief. Together, this indicates a set of phenomena within presence, separate
from the actual audio-visual content, that point to an increasing need or pleasure
in “doubling down on the human” 71 and our humanising social interactions.
Changes in culture and society also reconfigure LVP. Digitally enabled peer-
to-peer exchange networks have upended and transformed both copyright and
the economic income models for audio-visual artists, whereas live events and the
relationship to presence building will play a vital role for artists and consumers
alike, and we see evidence of how this is being commodified.
The unique status of ‘live-ness’ in this art form requires a change and height-
ening of attention, in which participants physiological and the psychological
systems are invited to focus on the performance and being within an audience.
Nightclub culture, having moved from underground origins to professionalised
settings, now faces challenges of prohibitive costs through urban environmental
gentrification. Traditional venues may naturally evolve as practices situated in
hybrid multi-use spaces. This has the potential to further add to the intangible
characteristics in LVP, but as discussed earlier, there are some precedents about
the value of environmental heritage, especially venues as site of post-industrial
change (see Berlin’s Berghain). Without knowing what new trends will emerge
within LVP and the impacts from the COVID-19 pandemic, Statista (2021) con-
tinues to forecast global growth in the live music market.72
By considering LVPs as augmented “hedonic products” – the combination
of emotions, senses, imagination, and intellect – we also continue the trajec-
tory broadly witnessed within mainstream culture at a larger scale, of affording
immaterial experience the same status as traditional objects or products, and note
their ability to inf luence user’s expectations and desires.
Individuals are willing to pay significant amounts to access these time-bound
hedonic experiences. Across the spectrum of experimental to stadium-filling
LVPs, the act of engaging environmental crowded spaces is to access the dis-
sipation of ‘normal’ rules of society and identity, by being part of a larger more
amorphous social group, who usually converge to appreciate the performance and
access temporary cognitive escapism. The perceptual conditions of the interior
space – lighting, physical layout of the audience – help to create a sense of los-
ing oneself and provide positive conditions in which the spectators can actively
co-create a feeling of freedom, and for some even the feeling of transcen-
dence. These characteristics are welcome contributions in contemporary media,
enabling social interactions to foster sensations of wellbeing.

Notes
1 Matthew Lombard and Theresa Ditton, “At the Heart of It All: The Concept of Pres-
ence,” Journal of Computer-Mediated Communication, 3(2), 1997.
2 Mel Slater and Martin Usoh, “Representation Systems, Perceptual Position and Pres-
ence in Virtual Environments,” in Presence: Teleoperators and Virtual Environments (Cam-
bridge, MA: MIT Press, 1994).
 Presence and Live Visual Performance 283

3 Samuel Taylor Coleridge, Biographical Sketches of My Literary Life and Opinions (New
York: American Book Exchange, 1881).
4 Susan Greenfield, Mind Change: How Digital Technologies Are Leaving Their Mark on Our
Brains (New York: Random House, 2015).
5 Shirley Cramer and Becky Inkster, “#StatusOfMind: Social Media and Young Peoples
Mental Health and Well Being,” Royal Society for Public Health Vision Voice and Practice,
5(12), 2017.
6 Marvin Minsky, “Telepresence,” Omni Magazine, June 1980, 45–51.
7 Carrie Heeter, “Communication Research on Consumer VR,” in Frank Biocca and
Mark R. Levy, co-eds., Communication in the Age of Virtual Reality (Hillsdale, NJ: Law-
rence Erlbaum Associates, 1995), 191–218.
8 André Bazin and Susan Sontag both hold interesting discussions about the veracity/
authenticity afforded onto photography as a medium, see: Susan Sontag, On Photography
(New York: Farrar, Straus and Giroux, 1977), 175. Andre Bazin, “The Ontology of
the Photographic Image,” in Leo Braudy and Marshall Cohen, co-eds., Film Theory and
Criticism: Introductory Readings, 6th edition (New York: Oxford University Press, 2004),
166–170.
9 A new coronavirus disease (COVID-19) causing respiratory symptoms was first identi-
fied in December 2019 in China. The World Health Organization declared COVID-19
a pandemic on 11 March 2020, and this means COVID-19 has spread worldwide. One
societal impact was the adoption of work from home, as well as physical distancing prac-
tices which created a surge in demand and development of video-conferencing tools.
10 Janine Kacker, Jan Vom Broke, Joshua Harndali, Markus Otto, and Johannes Schnieder,
“Virtually in This together: How Web-Conferencing Systems Enabled a New Virtual
Togetherness during the COVID-19 Crisis,” European Journal of Information Systems,
29(5), Special Section: Orchestration in Contemporary Software Development Ecosys-
tems, 2020, 563–584.
11 Ivy Roberts, The Perceptual Illusion of Nonmediation, Online article, 2014, https://ivyrose
roberts.wordpress.com/2014/08/12/the-perceptual-illusion-of-nonmediation/.
12 Matthew Schnipper, Seeing Is Believing: The State of Virtual Reality, Online article, 2016,
www.theverge.com/a/virtual-reality/intro.
13 Marvin Minsky, “Telepresence,” Omni Magazine, June 1980, 45–51.
14 Originally published in 1985 in the Socialist Review. Donna Haraway, “A Cyborg Man-
ifesto: Science, Technology, and Socialist-Feminism in the Late Twentieth Century,” in
Donna Haraway, ed., Simians, Cyborgs, and Women, 1st editor (New York: Routledge,
1991), 149–181.
15 Janet Murray, Hamlet on the Holodeck: The Future of Narrative in Cyberspace (New York:
The Free Press, 1997).
16 Frank Biocca, “Cyborg’s Dilemma: Progressive Embodiment in Virtual Environments,”
Journal of Computer Mediated-Communication, 3(2), 1997, www.ascusc.org/jcmc/vol3/
issue2/biocca2.html and Frank Biocca, “Is This Body Really ‘Me’? Self Presence,
Body Schema, Self-Consciousness, and Identity,” The Cyborg’s Dilemma: Progressive
Embodiment in Virtual Environments, Journal of Computer Mediated-Communication,
3(2), 1997, 295–302.
17 Howard Rheingold, Virtual Reality (New York: Summit Books/Simon Schuster, 1991), 256.
18 Mihaly Csikszentmihalyi, Flow: The Psychology of Optimal Experience (New York: Harper &
Row, 1990).
19 Mark T. Palmer, “Interpersonal Communication and Virtual Reality: Mediating Inter-
personal Relationships,” in Frank Biocca and Mark R. Levy, co-eds., Communication in
the Age of Virtual Reality (Hillsdale, NJ: Lawrence Erlbaum, 1995), 277–302.
20 Gene Quarrick, Our Sweetest Hours: Recreation and the Mental State of Absorption ( Jeffer-
son, NC: McFarland, 1989).
21 Frank Biocca and Mark Levy (eds.), Communication in the Age of Virtual Reality (Hillsdale,
NJ: Lawrence Erlbaum, 1995), 277–302.
22 Frank Biocca and Mark Levy (eds.), Communication in the Age of Virtual Reality (Hillsdale,
NJ: Lawrence Erlbaum, 1995), 277–302.
284 Donna Leishman

23 Donald Horton and Richard Wohl, “Mass Communication and Para-Social Interac-
tion,” Psychiatry: Journal for the Study of Interpersonal Processes, 19, 1956, 215–229.
24 Sherry Turkle, Alone Together: Why We Expect More from Technology and Less from Each
Other (New York: Basic Books, 2011).
25 Bruce McConachie and Elizabeth Hart, Performance and Cognition: Theatre Studies and the
Cognitive Turn (New York: Routledge, 2010).
26 Bruce McConachie and Elizabeth Hart, Performance and Cognition: Theatre Studies and the
Cognitive Turn (New York: Routledge, 2010).
27 Erika Fischer-Lichte, The Transformative Power of Performance: A New Aesthetics (New
York: Routledge, 2008).
28 Golan Levin, Gregory Shakar, Scott Gibbons, Yasmin Sohrawardy, Joris Gruber,
Erich Semlak, Gunther Schmidl, Joerg Lehner, and Jonathan Feinberg, Dialtones (A
Telesymphony) (Premiered Ars Electronica, 2001), www.flong.com/archive/projects/
telesymphony/index.html.
29 Herbert Blumer, “Collective Behavior,” in Alfred M. Lee, ed., Principles of Sociology
(New York: Barnes and Noble, 1969), 165–221.
30 Jeanne Ballantine, Keith Roberts, and Kathleen Odell Korgen, Society and Culture: Hard-
ware and Software of Our Social World (Thousand Oaks, CA: Sage Publications, 2017).
31 Carl Ratner, Macro Cultural Psychology: A Political Philosophy of Mind (Oxford: Oxford
University Press, 2012).
32 Stephen Butler, “The Impact of Advanced Capitalism on Well-Being: An Evidence-
Informed Model,” Hu Arenas, 2, 2019, 200–227.
33 Sherry Turkle, Alone Together: Why We Expect More from Technology and Less from Each
Other (New York: Basic Books, 2011).
34 Terry Eagleton, Marx and Freedom (London: Phoenix, 1997), 37.
35 Britta Wheeler, “The Social Construction of an Art Field: How Audience Informed
the Institutionalization of Performance Art,” Journal of Arts Management, Law, and Society,
33(4), 2004, 336–350.
Miranda Boorsma, “A Strategic Logic for Arts Marketing: Integrating Customer
Value and Artistic Objectives,” International Journal of Cultural Policy, 12(l), 2006, 73–92.
36 Michael Etgar, “A Descriptive Model of the Consumer Co-Production Process,” Journal
of the Academy, 2008, 97.
37 Jennifer Radbourne, Katya Johanson, Hilary Glow, and Tabitha White, “The Audience
Experience: Measuring Quality in the Performing Arts,” International Journal of Arts Man-
agement, 11(3), 2009, 16–29.
38 Elizabeth Hirschman and Morris Holbrook, “Hedonic Consumption: Emerging Con-
cepts, Methods and Propositions,” Journal of Marketing, 46(3), 1982, 92–101.
39 Chiara Santoro and Gabriele Troilo, “The Drivers of Hedonic Consumption Experi-
ence: A Semiotic Analysis of Rock Concerts,” in Antonella Carù and Bernard Cova,
co-eds., Consuming Experience (London/New York: Routledge, 2007), 109.
40 Kathleen Lacher and Richard Mizerski, “An Exploratory Study of the Response and
Relationships Involved in the Evaluation of, and in the Intention to Purchase New
Rock Music,” Journal of Consumer Research, 21, 1994, 366–380.
41 Gaston Bachelard, The Poetics of Space (Boston, MA: Beacon Press (trans. 1964), 1994).
42 Peter Singer, Let’s Talk about Your Hedonism, 2012, Documented talk on YouTube, www.
youtube.com/watch?v=Vfkcg05_uUg.
Robert Nozick, The Experience Machine: Anarchy, State, and Utopia (New York: Basic
Books, 1974).
43 “Alienation” is a concept proposed by Karl Marx in 1844. Marx posited that capitalism
and class stratification through mass industrialisation would disenfranchise labour and
our lives would become characterised by a disorientating sense of exclusion and separa-
tion. For Marx, people are both world determined and world producing; consciousness
is dominated by the ideological super-structures with which we interact within. See:
Terry Eagleton, Marx and Freedom (London: Phoenix, 1997), 37.
44 Advanced capitalism is defined as an occurrence in society in which the capitalist model
has been integrated and developed deeply and extensively and for a prolonged period.
 Presence and Live Visual Performance 285

This term is commonly applied to countries with a high economic freedom score such
as Singapore, New Zealand and Switzerland. See mapping: https://worldpopulation
review.com/country-rankings/capitalist-countries.
45 Elizabeth Hirschman and Morris Holbrook, “Hedonic Consumption: Emerging Con-
cepts, Methods and Propositions,” Journal of Marketing, 46(3), 1982, 97.
46 Henrik Hagtvedt and Vanessa Patrick, “The Influence of Art Infusion on the Percep-
tion and Evaluation of Consumer Products,” in Angela Y. Lee and Dilip Soman co-eds.,
Advances in Consumer Research, Association for Consumer Research, vol. 35 (Chicago, IL,
2008), 795–796.
47 Thorstein Veblen, Theory of the Leisure Class: An Economic Study of Institutions (New York:
The Macmillan Company, 1899).
48 Guy Julier, Anders Munch, Mads Nygaard Folkmann, Hans Christian Jensen, and Niels
Peter Skou, Design Culture: Objects and Approaches (London: Bloomsbury, 2019).
49 Carolyn Guertin, Digital Prohibition: Piracy and Authorship in New Media Art (London:
Bloomsbury, 2010).
50 Laura Bettinson, “After a Year without Nightclubs, I Can’t Wait to Go Dancing Again,”
Guardian Online, March 2021, www.theguardian.com/commentisfree/2021/mar/05/
nightclubs-dancing-covid-live-music-djs-21-june.
51 Lanre Bakare, “We Lost the Love: UK Nightclubs Using Covid Crisis to Reassess Scene,”
Guardian Online, August 2020, www.theguardian.com/music/2020/aug/21/we-lost-the-
love-uk-nightclubs-using-covid-crisis-to-reassess-scene and Rob Davis, “UK Nightlife
Industry Facing Financial Armageddon,” Guardian Online, August 2021, www.theguardian.
com/business/2020/aug/20/uk-nightlife-industry-facing-financial-armageddon.
52 Jennifer Radbourne and Andrew Arthurs, “Adapting Musicology for Commercial Out-
comes,” in Manuel Cuadrado and Juan Montoro, co-eds., CD-ROM Proceedings: 9th
International Conference on Arts and Cultural Management (Valencia, Spain: University of
Valencia, 2007).
53 Ning Wang, “Rethinking Authenticity in Tourism Experience,” Annals of Tourism Research,
26(2), 1999, 353.
54 Ning Wang, “Rethinking Authenticity in Tourism Experience,” Annals of Tourism Research,
26(2), 1999, 358.
55 See Jeremy Deller (2018) documentary Everybody in the Place: An Incomplete History of
Britain 1984–1992, which charts the culture of dance music, and also Midnight Mass
(2017) capturing the religious experience of rave culture across three continents.
See Vice’s article: https://i-d.vice.com/en_us/article/wjdqdx/capturing-the-religious-
experience-of-rave-culture-across-three-continents.
56 Kevin McCarthy, Elizabeth Ondaatje, Laura Zakaras, and Arthur Brooks, Gifts of the
Muse: Reframing the Debates about the Benefits of the Arts (Cambridge: Rand Publications,
2004).
57 John M. Darley and Bibb Latané, “Bystander Intervention in Emergencies: Diffusion of
Responsibility,” Journal of Personality and Social Psychology, 8(4, Pt.1), 1968, 377–383.
58 Raymond Momboisse, Riots, Revolts, and Insurrections (Springfield, IL: C.C. Thomas,
1967).
59 Sally Moore and Barbara Myerhoff (eds.), “Variations on a Theme of Liminality,” in
Secular Ritual (Amsterdam: Van Gorcum, 1977), 43 and Victor Turner, The Forest of
Symbols: Aspects of Ndembu Ritual (Ithaca: Cornell University Press, 1967).
60 Victor Turner, The Ritual Process: Structure and Anti-Structure (Chicago: Aldine,
1969), 96.
61 Victor Turner, From Ritual to Theatre: The Human Seriousness of Play (New York: Per-
forming Arts Journal Publications, 1982), 44.
62 Ning Wang, “Rethinking Authenticity in Tourism Experience,” Annals of Tourism
Research, 26(2), 1999, 353.
63 Leon Festinger, Albert Pepitone, and Theodore Newcomb, “Some Consequences of
Deindividuation in a Group,” Journal of Social Psychology, 47, 1952, 382–389.
64 Mel Slater, Martin Usoh, and Anthony Steed, “Depth of Presence in Virtual Environ-
ments,” Presence-Teleoperators and Virtual Environments, 3(2), 1994, 130–144.
286 Donna Leishman

65 Vortex (2014) is a manually controllable music reactive light structure by the 1024 col-
lective. Available: www.1024architecture.net/?portfolio=vortex.
66 Tom Wilkinson, “Typology of a Nightclub,” Architectural Review, April 2020, www.
architectural-review.com/essays/typology/typology-nightclub.
67 Ben Kelly, interior designer of the Haçienda, a nightclub and music venue in Manchester,
northwest England. http://benkellydesign.com/hacienda/.
68 Ben Kelly, Brendan Mannion, and Justin Metz, Virtual Haçienda, 2020, http://benkelly
design.com/fac51-hacienda-vr/.
69 Jamie Holman and Alex Zawadzki, Flashback, 2019, www.acidhouseflashback.co.uk/.
70 Sansho Studio’s Berghain’s VR trainer, 2016, https://berghaintrainer.com/.
71 Tim Adams, “Interview with Jared Larnier,” Guardian Online, November 2017, www.
theguardian.com/technology/2017/nov/12/jaron-lanier-book-dawn-new-everything-
interview-virtual-reality.
72 Statista, Live Music Industry Revenue Worldwide from 2014 to 2024, January 2021, www.
statista.com/statistics/1096409/live-music-industry-revenue-worldwide-by-source/.
PART III

The Practice of Live Visuals

12
VJING, LIVE AUDIO-VISUALS
AND LIVE CINEMA
Steve Gibson and Stefan Arisona

Introduction
This chapter looks at VJing practices in the contemporary context, as well as
the work of related audio-visual performers and live cinema producers. Mov-
ing beyond the historical examples of Live Visuals covered in Part I, we look
here at performance scenarios, technology and the materials of present-day Live
Visuals, beginning with VJing and following with live audio-visual perfor-
mance and live cinema. We also present three case studies of contemporary Live
Visualists.
This chapter explores the emergence of projects from composition to per-
formance, both from technical and artistic viewpoints: What were the original
intentions behind the audio-visual correspondences? How was material collected
and assembled? What were the artistic choices of drawing the line between (non-
live) composition and performance? What were approaches to creating narratives
in a live context? In addition, we will look at the extended context such as the
relation to sound and music, both recorded and live.

A Brief History of VJing

As discussed in Chapter 4, VJing has its direct roots in ‘scratch video,’ as exem-
plified by the work of artists such as Emergency Broadcast Network. A template
for Live Visuals performance was established in the 1990s that is echoed in much
of the practice of contemporary VJing: short looped video clips, combined with
user-generated content (often in the form of motion graphics and/or abstract
geometric shapes), and processed in synch with the incoming audio using digital
signal processing (DSP). This template unites the practices of many VJs from the
most commercial to the most experimental.
DOI: 10.4324/9781003282396-16
290 Steve Gibson and Stefan Arisona

The break-through in computer processor speeds, as well as dramatically

enhanced hardware graphics capabilities, that occurred in the new millennium
meant that Live Visualists could also rely on their laptops to serve as the ultimate
“Database Cinema”1 device. Banks of clips could be stored, added into memory
and randomly accessed at any time in a performance. Effects could be applied in
real time with no noticeable lag, and the incoming audio signal could be used to
drive both the effects and the order and appearance of clips. The dream of Live
Visuals that began with the first colour organs could now be realised with rela-
tive ease (at least technologically).

Performance Scenarios in VJing

In VJ performance, we can observe a wide range of possible scenarios, depend-
ing on how the VJs collaborate with musicians and how much of the music is
known in advance or is improvised: on one side of the spectrum, there can be
a very loose or even no preliminary connection between the audio and visual
domains, and the visual performance usually is fully improvised following the
music, such as practised in the work of the Scheinwerfer collective. Or, as an oppo-
site extreme, the VJs and musicians work together beforehand and the audio-
visual performance is carefully orchestrated, which today is often the case of
touring DJs or musicians that consciously include Live Visuals as an integral part
of their shows. An example is Richie Hawtin’s PLASTIKMAN 2010 tour, in
which Hawtin collaborated with visual artist Ali Demirel as well as the software
company Derivative (see Chapter 18 for an interview with Greg Hermanovic,
the CEO of Derivative, for a further discussion on this).
The range chosen on this spectrum inf luences how VJs prepare their content:
In the former case, typically a large library of content needs to be prepared, and
elements from the library are selected as a choice by the VJ depending on the
music played. In conjunction with automatic audio analysis (such as beat sync-
ing), this allows the performer to respond to musical style, the atmosphere of
the venue and the mood of the audience. In the latter case, specific material is
prepared beforehand and then chosen to match the sequence of individual tracks
or musical elements; however, since musical material often varies during a tour
or a performance, the visuals cannot always be fully prepared, and typically a
visual artist does live mixing and improvisation (using such tools as Resolume,
Modul8, VDMX or TouchDesigner) as he or she would for fully improvised Live
Visuals performance.

The Technology of VJing

As mentioned earlier, the form of VJing is directly related to the scratch video
performance of the 1990s. Similarly, the materials and technologies used by VJs
can be directly linked to those used by scratch video artists. In place of videotape
loops, VJs use computer video loops; in place of analogue video mixers, VJs use
 VJing, Live Audio-Visuals and Live Cinema 291

the A-B (or more complex) mixing capabilities of Live Visuals software; in place
of video effects, such as those used by the Fairlight Computer Video Instru-
ment,2 VJs use an array of built-in software effects and plug-ins. In most cases
the comparison between the old form of scratch video and the current form of
VJing is quite direct: ‘A-B-ing’ a video loop on an analogue video mixer would
have involved the visual performer moving a crossfader. Similarly, a VJ normally
uses a MIDI control surface to manually create the same effect, or he or she can
use DSP to detect the incoming signal and switch between A and B depending
on volume level. While the technology of the computer makes ‘A-B-ing’ easier
for the VJ, the technique is not an original one, as it was employed extensively
by scratch video artists, as outlined previously in Chapter 4.
Similarly, many, if not most, of the effects employed by VJs in various appli-
cations are merely software versions of effects that have been employed in tradi-
tional film since the early 20th century. Compositing of different clips in order
to combine them together in various ways is one of the most obvious exam-
ples of an effect commonly used by VJs that has strong historical antecedents.
Arguably the first example of compositing comes from Thomas Edison’s 1903
film The Great Train Robbery. In the following example, the scene through the
“train door” showing passing scenery was inserted after the initial filming of
the sequence.3 The effect was achieved by ‘matting’ off the door area and super-
imposing moving scenery into the rectangle encompassed by the door. Matting
effects are common in both off-line digital video editing system such as Adobe
Premiere and also can be achieved in real time using projection mapping soft-
ware such as MadMapper.
Compositing in general is used almost universally by VJs to combine clips
together in various way. Using effects such as ‘Luma Keying,’ a VJ can allow por-
tions of a video to be effectively erased, and therefore bleed through to another
video channel. Other effects such as additive colouring, scaling, blurring, and
saturation all have roots in traditional analogue film. The screen shot of the top
section of GarageCube’s Modul8 software interface in Figure 12.1 illustrates
how these effects can be easily accessed and changed in real time by the use of
sliders and knobs.
If the effects used by VJs are in many ways the same as those used by tradi-
tional film artists, then what is the unique aspect or quality of VJing that makes
it distinct? While it may seem obvious, it bears repeating here: these effects are
achieved in real time. It is difficult to overstate the importance of this distinc-
tion. While there are examples of visuals being employed in a live context in ear-
lier eras, as alluded to in the discussion of liquid visuals in Chapter 3 and scratch
video in Chapter 4, the introduction of non-linear video processing radically
enhanced the possibilities of mixing and matching sources and adding a variety
of effects in real time, often with little need for any pre-production.
In addition, the employment of DSP to analyse the incoming audio signal and
produce effects related to that audio is in fact a unique feature of VJing distinct
from earlier forms and can produce subtle effects that link the formal and material
292 Steve Gibson and Stefan Arisona

FIGURE 12.1 GarageCube’s Modul8 interface, effects section. ©GarageCube SA.

Used by permission.
Source: www.garagecube.com/contact/

properties of the visual stream with the audio source. It should be said that the use
of DSP has also been problematic for VJing in less competent hands. The cliché of
a throbbing image bouncing to a four-on-the-f loor kick drum has been repeated
so many times, it has become evidence of the aesthetic entropy within VJ culture.
On the other hand, there are many VJs who eschew standard VJ software and cre-
ate their own tools, either using packages such as Max MSP/Jitter or by program-
ming their own software. An example of this is Scheinwerfer’s Soundium package,
which employs a node-like structure and complex prediction algorithms to create
unique effects that are unobtainable with off-the shelf software.
Any discussion of the technology of VJing would be incomplete without con-
sidering the controllers used by VJs in live performance. Many, if not most, VJs use
some sort of control surface or other MIDI interface to enable hands-on manipu-
lation of video parameters and effects. These range from simple slider and knob-
based systems to tactile systems based on gesture using tools such as the Microsoft
Kinect or the Leap Motion. These tools allow the VJ, the audio-visual performer
and the live cinema artist to have direct real-time control of multiple visual and
audio-visual parameters. These devices can produce a rich experience of Live
Visuals in which the performance of the visual material is directly followed by the
audience, while at the same time the technique of audio-visual matching produces
an awareness of the virtuosity of these tools in the hands of the Live Visualist.
 VJing, Live Audio-Visuals and Live Cinema 293

The Material of VJing

The concept of the performed image is hardly new or unique to VJing or any
form of contemporary Live Visuals. As outlined in Chapters 1 and 2, the per-
formed image has been possible at least since the late 19th century, with the
advent of Rimington’s colour organ, and arguably as far back as Louis-Bertrand
Castel’s ocular clavichord of 1734. The long history of Live Visuals since then
has provided abundant evidence that VJing is a continuation of a practice (rather
than a new practice) that has moved through light organ performance, to hand-
drawn film, to liquid light shows, to expanded cinema, to scratch video and
finally to the computer-based Live Visual performance of the present. Each of
these earlier forms has in some way informed the practice of VJing, even if the
VJs themselves are unaware of these historical antecedents.
In terms of a performance medium, the light organ is a clear reference point
for early VJ applications, such as early versions of ArKaos which allowed the art-
ist to place images and effects on keys of a virtual keyboard, which then could
be driven by a MIDI keyboard. It can be easily argued that early software such
as ArKaos4 was merely a digital representation of a light or colour organ, with
enhanced possibilities for effects, as well as real-time access and configurability.
Similarly, as a medium of abstract and semi-abstract moving images, the materi-
als of the VJ directly reference (consciously or not) with the hand-drawn films
of Fischinger and McLaren, the structural-materialist films of Stan Brakhage,
Michael Snow and others, the liquid light shows of The Joshua Light Show and
Single Wing Turquoise Bird, the computer animations and graphics of John
Whitney and others and the scratch video of Emergency Broadcast Network
and others. Taken side-by-side, much of the visual material of the vast majority
of contemporary VJ culture is virtually indistinguishable visually from one or
more of these forms.

The Performance of VJing

As stated previously, the genuine innovation of contemporary Live Visuals cul-
ture is its real-time performance aspect. Using the previously mentioned con-
trol interfaces, a visual performance by VJs now very closely resembles the live
performance of electronic music. One of the key substantiations of this claim is
that it is possible to have errors in Live Visuals performances in a similar manner
as in music performance, in which a musician can play wrong notes, though as
noted by the Scheinwerfer VJ collective: “having errors in a visuals performance is
still much more forgiving than in a music performance.”5 In a straight playback
medium such as much of the work mentioned earlier, barring an error on the
playback device or storage medium, a linear video will always be a perfect replica
of its original conception. In genuine Live Visuals work that is performed in real-
time (even if there is DSP and other automatic processing involved), the hand of
the performer is key to the experience.
294 Steve Gibson and Stefan Arisona

In this regard, connections can be made between VJing and improvised jazz.
In improvised jazz a context is provided, which might include a tempo, a key
or mode, and a collection of riffs commonly is used, but the performers then
respond to each other and the mood of the audience when deciding how to pro-
ceed in time. Similarly, a VJ might have a context, which might include a tempo
range, a style of music and a collection of commonly used visual material, but
he or she will respond to the music and the mood of the audience and vary that
performance in real time. Unpredictability in performance has gradations in the
practice of VJing, but even in the most tightly controlled Live Visuals perfor-
mance in which specific images are chosen for specific music tracks or sounds,
there will almost always be some element of improvisation that comes into play.

Case Study 1: Primarily Improvised Live Visuals Performance:

Scheinwerfer VJ Collective
Scheinwerfer (translated variously as spotlight, headlight or headlamp) is a Swiss
VJ collective that has been operating since the late 1990s. Their work has been
varied, encompassing one-off club shows, residencies at various clubs and more
complex audio-visual installations (often under the name of their parent orga-
nization, Corebounce). Their members are trained as computer scientists, and
therefore they have built their own VJ platform, Soundium, that they use exclu-
sively for their Live Visuals events. They have supported acts as diverse as Jeff
Mills, Miss Kittin, Dave Clarke and Josh Wink, and as a rule use a primarily
improvised approach to their VJ performances. This is a familiar scenario to the
club VJ. When we generally consider VJ performance, a basic model of impro-
vised visuals, with DSP applied to the incoming audio, is the most recognisable
form that it takes.
Scheinwerfer describe their approach to VJing as very much related to the devel-
opmental process of DJing: “Browsing record stores for new releases, checking
out DJ charts, listening to radio shows, and trying to get hold of recorded live
sets.”6 The rise of the VJ as a genuine live visual performer, as outlined in Chap-
ters 4 and 5, roughly follows the rise of the DJ as a genuine sound performer
(distinct from the previous understanding of a DJ as a radio presenter or a club
DJ playing requests). Both the VJ and the DJ have been unfairly derided by tra-
ditional musicians and visual artists as ‘not music’ or ‘not art.’ The arguments to
the contrary are many, but the key claims are as follows: the DJ is involved in
a reconstruction of many sources of music, often leading to a radically altered
versions of original tracks, in some cases with multiple referents producing a
distinctly new composition; similarly, the VJ often takes multiple found sources,
mixes those with original footage, abstract imagery and visual effects, to create
a new variation on the source material. Put both the DJ and the VJ together
in this context, and a form is created that can be best described as audio-visual
remix art (see Figure 12.2). In terms of concept and realisation, this form is not
 VJing, Live Audio-Visuals and Live Cinema 295

FIGURE 12.2 Scheinwerfer live at Digital Art Weeks 2007, Zurich. Photo by Ruedi
Kuchen. Used by permission.
Source: [email protected]

indistinct from the work of artists now considered to be key figures in modern-
ist and postmodernist musical and visual cultures, from musique concrète to
structuralist-materialist film.
For the members of Scheinwerfer, despite the improvised nature of much of their
work, it is absolutely vital that the Live Visuals they use not fade into wallpaper:

It started with our disappointment about the light shows we experienced at

electronic music events in the early 90s. Clearly, there are notable excep-
tions, but the average electronic music club did not pay much attention
to visuals back then, and the music was often awkwardly accompanied by
some blinking lights often out-of-sync with the beat. This was in harsh
contrast to the holistic, synaesthetic experience we were seeking. So, we
set out to gather some background in film language and film music theory
and tried to put as much of it as possible into a computer system.7

The key to achieving a genuinely engaging and meaningful audio-visual per-

formance in an improvised context is by addressing “how to deal with time
and alignment of a ‘musical score’ to visual action points.”8 Since they have
built their own VJ software, there are various ways of achieving this alignment,
from the lowest, beat level, to a more macro, long-term level. As with most VJs,
296 Steve Gibson and Stefan Arisona

Scheinwerfer use signal processing to provide rhythmic connections between the

incoming sound and the visual results:

Real-time audio signal processing for beat and frequency tracking clearly
helps a lot creating perceptible audio-visual correspondence in a very
direct manner (for example visual distortions linked to frequency levels or
geometric transformations coupled to beat triggers). However, one needs
to pay attention that this connection does not become too rigid, thus our
audio engine can be manipulated during performance, which opens room
for improvisation.9

On a subtler level, Scheinwerfer also work on a medium-level time scale to create

“scenes” that change as the music is varied. This idea is indebted to both Eisen-
stein’s general theories of montage, as well as sophisticated music videos, such as
those by Michel Gondry and Chris Cunningham. At a higher level, they employ
a much looser approach, not relying on any formal process or automatic audio-
visual mapping to change the visual world, but instead they “rotate [performers]
at the visuals console (usually four performers in total, either one performer at a
time, or two performers collaborating). Often this is aligned with set changes,
which helps in creating an audio-visual correspondence in terms of individual
visual style.”10
Scheinwerfer’s video documentation of their past performances can be sampled
at www.youtube.com/channel/UCXrWYksXzyCbxMGBc5u1m5w.

A Brief History of Live Audio-Visual Performance

Live audio-visual performance has a comparatively long history, particularly if
we consider earlier mixed-media forms such as opera as part of the long history
of moving images becoming alive. For the purposes of this volume, we are pri-
marily concerned with the performance of audio-visuals in real time as assisted
by analogue and digital technologies. As outlined in Chapters 1 and 2, examples
of technologically enabled live audio-visuals can be traced back to Riming-
ton’s colour-organ and Scriabin’s chromola (or luce) used in his Prometheus: Poem
of Fire. These interfaces were meant to be used as visual performance devices
within the context of a live music performance and therefore can be seen as logi-
cal antecedents to the interface technologies used in contemporary audio-visual
performance.
Similarly, contemporary audio-visual performance is inf luenced broadly by
the various avant-garde movements described in Chapters 2 and 3. A direct
line can be drawn between the audio-visual experiments of McLaren and
Fischinger to the work of contemporary live audio-visual performers such as
alva noto and Robert Henke. These figures are all united by an overriding
preoccupation with the precision of audio-visual correspondences, the former
 VJing, Live Audio-Visuals and Live Cinema 297

by utilising the medium of the film strip as a canvas for precise audio-visual
synchronization, and the latter by using digital interfaces to precise map the
connections between the audio and visual spheres for use in a live performance
environment.

Performance Scenarios Unique to Audio-Visual Performance

Contemporary audio-visual performance is often completed by one solo per-
former controlling both the audio and visual materials simultaneously. This
approach can be seen in the work of alva noto described in the case study that
follows. In this scenario the audio-visual mappings tend to be quite fixed and
formally applied. One could argue that while there has yet to emerge a unified
system for such mappings, the concerns of many audio-visual performers are
similar to formal concerns explored by modernist artists. This is particularly
evident in the use of mathematical, rhythmic matching of audio and visual
objects (arguably similar to Eisenstein’s concept of ‘metric montage’) or in the
use of precise charts to describe the audio-visual tonal mappings to be used
(arguably related to Scriabin’s charts for his luce, as discussed in Chapter 2 – see
Figure 2.2).
Other performance scenarios in live audio-visual performance include mul-
tiple performers, some controlling audio and some controlling visual domains.
This can be seen in the work of Sensors, Sonics, Sights, in which one performer
controls the visual domain with a gestural controller and two musicians perform
with a Theremin and a purpose-built muscle-sensor control system, respectively.
In this scenario the relationship between the audio and visual realms tends to be
more f luid and improvisatory, not unlike the relationship between DJ and VJ
described at the start of this chapter.

The Technology of Live Audio-Visual Performance

The visual technology used in live audio-visual performance tends to be similar
to that used by VJs, focusing on either precisely mapped visuals that are con-
trolled by some sort of MIDI controller or sensor interface. The audio tech-
nologies, on the other hand, can be much more varied than in the tradition DJ/
VJ scenario. Given that the music is performed live, the use of specialist audio
software from simple sequencers (i.e. Ableton Live) to complex object-oriented
programming systems (i.e. Max MSP) are used both to process and/or gener-
ate a performance in real time. It is common for a variety of different musical
elements, both analogue, digital and traditional, to co-exist with the visual ele-
ment. This is also evident in the work of live cinema artists such as The Light
Surgeons, as discussed later in this chapter. In this regard it is much harder to
precisely quantify the technological norms of audio-visual performance, as the
roles are much less fixed than in DJ/VJ performance.
298 Steve Gibson and Stefan Arisona

The Material of Live Audio-Visual Performance

The visual material of live audio-visual performance tends to be relatively simi-
lar to that of VJing, particularly in its general use of abstract imagery in synchro-
nisation with incoming audio input. There are, of course, exceptions to this,
such as Adriana Sá’s performances with the semi-realistic 3D environments that
look like moving paintings, but as a rule audio-visual performance tends toward
the abstract. The audio element of live audio-visual performance is much more
varied than in DJ/VJ performance. The latter is dominated by club-based dance
music, while the former may range from wildly experimental to comparatively
dance-based. It is therefore extremely difficult to generalise about the audio
material of live audio-visual performance, other than to say a drastic range of
forms, genres and styles is employed, often by the same artist.

The Performance of Live-Audio Visuals

While live audio-visual performance has a clear connection with earlier forms,
such as the audio-visual handwork of Norman McLaren or the performances of
Andy Warhol and the Velvet Underground, its key innovation is in the precision
of the audio-visual relationship as performed live and in real time. In this way it
is similar to VJing, but with a performed audio element added into the mix, live
audio-visual performance tends to be more structural, often relying on complex
multimodal mappings rather than DSP. In this way live audio-visual perfor-
mance points to possible formal solutions for audio-visual relationships that go
beyond the simple mapping of audio input often used in VJing (e.g. the spike of a
kick drum) to visual result (dramatically increased size of visual object). As out-
lined in the Case Study that follows, this usually involves very carefully planning
between programmer and artist and yields results that are more repeatable and
ultimately more satisfying aesthetically.

Case Study 2: Primarily Pre-planned Live Audio-Visual

Performance: Markus Heckmann
Markus Heckmann is a software developer, programmer and technical director
who works for Derivative, makers of the TouchDesigner VJ software. He has col-
laborated with alva noto, Matt Thibideau, Whitevoid and Robert Henke, amongst
many others. He also works as a live visualist in his own right and has produced a
number of installations and Live Visuals performances that have been shown across
North America and Europe. In his work with major artists such as alva noto the
audio-visual interaction is highly pre-planned, thus creating a situation where the
interface “was as minimally intrusive to the artist’s usual workf low as possible.”11
Since 2006 Markus has worked with Derivative in Toronto, producing inter-
faces and audio-visual programming for artists who are internationally known
for complex and engaging audio-visual performances (see Figure 12.3). This
 VJing, Live Audio-Visuals and Live Cinema 299

FIGURE 12.3 Matt Thibideau and Markus Heckman, Reclusion at MUTEK, 2014.
Photo by Caroline Hayeur. Used by permission.
Source: Mail conversation with artist available on request.

distinction is vital here, as much of the Live Visuals world is dominated by VJing
in response to DJ sets that may or may not be pre-planned. In an audio-visual
performance such as alva noto’s univrs the audio and visual material is performed
and controlled live with a minimum of improvisation (both audio and visual)
and the relationship between the two elements is therefore relatively fixed.
Markus describes his approach to this and other audio-visual performances
as follows:

In the case of Alva Noto’s univrs and Christopher Bauder’s and Robert
Henke’s Grid, designing a performance interface was not necessary as
Ableton Live, the audio software used in both cases, provided the art-
ists with sufficiently precise control over the triggering and parametric
changes occurring during these shows (OSC and MIDI).12

Due to the fact that in most audio-visual performance both domains may be
controlled by one (or more) performer(s) simultaneously, the degree of visual
improvisation does not lie in the hands of an independent live visualist or VJ,
but rather “possibilities for improvisation are limited to the musician as the show
runs autonomous from the developer. In alva noto’s case, the show was a collec-
tion of direct audio visualizations and MIDI triggered effects, variations in audio
would see a direct response in the visual output.”13
300 Steve Gibson and Stefan Arisona

The situation described earlier necessitates a highly pre-planned relationship

between the audio materials and the visual results. This is quite distinct from
traditional VJ performance, in which the visualist may have to respond quickly
to unknown changes in the DJ’s selection of music mixes. Markus describes the
process of pre-planning univrs as follows:

univrs is a collection of modules demonstrating different methods of audio

visualization partially derived and inspired by previous works of Carsten
Nicolai [aka alva noto]. In creating the installation, one point of reference
was the signal f low of the audio waveform through the various visualizers.
Further inspiration came directly from working alongside Carsten in prepa-
ration of the show – very much a listen, discuss, create, revise process where
you look at the audio waveform from a technical standpoint and imagine
what visual representation best matches shape, energy and musicality.14

For Markus the key instrument for developing the kind of audio-visual rela-
tionships that work in this context is TouchDesigner itself:

In the most ideal configuration my interface is the UI of the TouchDe-

signer, in an attempt to have as much f lexibility as possible and not rule
out a creative impulse by being tied to a fairly rigid classic controller layout
with its buttons, sliders and presets. For [audio-visual] correspondence, I
would try to keep a rhythmic element going, which can be driven by a
manual beat or audio analysis, while working on additions or derivations
of the current look which is then gradually introduced.15

See Chapter 19 for a longer interview with Markus. His personal and profes-
sional work can be viewed at https://project1.net.

A Brief History of Live Cinema

As alluded to in Chapter 4, live cinema came out of the Live Visuals revolution of
the 1990s and was led in the early 2000s by filmmakers such as Peter Greenaway
and Mike Figgis, as well as Live Visualists such as The Light Surgeons. Greenaway
famously described cinema as dead in 2007: “Cinema’s death date was 31 Septem-
ber 1983, when the remote-control zapper was introduced to the living room,
because now cinema has to be interactive, multi-media art.”16 Greenaway’s films
such as 1985’s A Zed and Two Noughts already pointed to a type of database cin-
ema, with its alphabetised categorisations and effects-driven sequences. Similarly,
1991’s Prospero’s Books had long sections that could easily be described a proto-
audio-visual performance, with Michael Nyman’s music accompanied by com-
plex montages of both semi-abstract and realistic (but highly formal) imagery.
Slightly different in both tone and quality, but no less inf luential, was Mike
Figgis’s 2000 film Timecode, which consisted of four different stories playing
 VJing, Live Audio-Visuals and Live Cinema 301

simultaneously in quarter screen. While the film was considered a mixed suc-
cess at the time, it clearly pointed the way to a non-linear concept of cinema.
A connection between the expanded cinema of the 1960s and 1970s could also
be made to live cinema, though this is primarily because ultimately both are
meant to be a live, performed medium, rather than a simple playback medium.
The means and materials of expanded cinema are necessarily simpler than live
cinema: expanded cinema, as evidenced in the works of Tony Hill, Guy Sherwin
and others, usually (though not always) consists of a single performer interacting
in some manner with projected film.
As with VJing, the possibilities for live cinema performance were enabled by
the personal computer revolution of the late 1990s and early 2000s. The increase
in speed and processing power of computers, as well as the development of con-
trollers for manipulating visual material in real time, enabled the development
of complex non-cinematic performances such a Peter Greenaway’s multilayered,
multipronged Tulse-Luper Suitcases from 2003 (see NOTV’s performance with
Peter Greenaway in Figure 12.4).
It is worth reiterating that much of the work that can be described as live cin-
ema is not necessarily created by former or current filmmakers per se. The Light
Surgeons are an excellent example of a group who evolved from the Live Visuals
performance world of the 1990s into a group which now deals with large cin-
ematic structures in real time. Chris Allen from The Light Surgeons describes their
work as follows:

I have always felt that the term live cinema was a better ref lection of our
performances because we are using a lot of narrative and filmic language

FIGURE 12.4 NOTV, Peter Greenaway Tulse Luper VJ Performance, 02–09–2009

Mantova Ital. Photos by NOTV.COM. Used by permission.
Source: www.notv.com/index.php/contact-us
302 Steve Gibson and Stefan Arisona

in our shows. We are musicians and film makers who explore our material
live I would say, editing and compositing our material in a more expressive
musical way in real-time as opposed to a totally pre-rendered film.17

Performance Scenarios Unique to Live Cinema

The Live Cinema Network in the UK categorises live cinema as follows: “Live cin-
ema events can be typified as three categories: enhanced, augmented and participa-
tory.”18 Enhanced events can be broadly described as projected cinema screenings
with live musical accompaniment, which strictly speaking is not “live” cinema, but
rather traditional cinema with a live music element. Augmented live cinema is more
typical of genuine live cinema in which the visual material is performed live in some
manner and is accompanied by live music. Participatory events are live cinema with
some audience interaction. For the purposes of our discussion of Live Visuals, only
augmented and participatory live cinema can be authentically described as “live.”
While live cinema has recently been classified as a distinct medium (the Live
Cinema Network would seem to confirm this), the distinction between VJing
and live cinema is occasionally blurred. Despite this blurring there are character-
istics to live cinema that are unique:

1 Live cinema usually makes use of longer, more realistic shots (which may or
may not be abstracted through visual effects).
2 Live cinema at times makes use of documentary as well as fictional visual
scenarios, thus creating a more narrative form than VJing.
3 In most live cinema performances, the music is performed as a score with
the visuals. In some cases, the control of both is done by the same performer,
thus making it closer to audio-visual performance than VJing.

The use of longer-form narratives in live cinema is exemplified by The Light

Surgeons’ SuperEverything* project. This project uses footage shot in Malaysia
made over the course of a year (commissioned by the British Council in Malay-
sia) to tell quasi-documentary stories about various cultures in the country. This
footage is re-assembled in real time and performed with a group of Western
and Malay musicians. The focus on this larger narrative makes most live cinema
projects more complex to organize than VJ performances, which usually rely on
shorter clips, often of an abstract quality. Live cinema performances therefore
require more careful planning and less automatic processing of visual material
in response to incoming audio data. How material is assembled for performance
and what strategies are used to balance improvisational and orchestrated elements
are key aspects to consider in relation to live cinema practice.

The Technology of Live Cinema

While the technologies used by live cinema artists are similar to those used by
VJs (and live audio-visual performers), there are some distinct differences in both
 VJing, Live Audio-Visuals and Live Cinema 303

the type of technologies used and the qualities of the visual material employed.
VJs, particularly in the early days when computer speeds were slower and mem-
ory was limited, tend to accept low-quality, shorter, highly compressed clips.
This is due to the fact that VJ performances are often long (particularly club-
based VJ performances), and if the music selection is unknown prior to the event,
then a very large database of videos may be required. For this reason, video clips
for VJs are generally seen as short loops that can be repeated and processed in
synch with the music. The quality of the image is less vital than the potential for
the image to be loaded and to respond to the incoming DSP in relative real time.
Because live cinema artists tend to be focused on more narrative imagery, image
quality is therefore much more important for them than it is for VJs. In addition,
as live cinema is involved with longer narratives, it tends to be less concerned
with short loops, instead preferring longer shots that can be overlaid with other
images, abstract forms or text.
For these reasons, most live cinema rigs are purpose-built and use specially
designed controllers to interact with the visual material in real time. A good
example of this is shown in the short video documentary of Peter Greenaway’s
Tulse Luper.19 Live cinema also tends to make use of more devices and materials
than VJ performance. VJs are often content to use a simple control surface and a
laptop for their performances. Live cinema often adds analogue equipment such
as film and slide projectors, multiple control devices, as well as perhaps props
and theatrical elements. Importantly live cinema almost always employs some
sort of live music element (electronic and/or acoustic), and the audio element is
generally also incorporated into the technological setup. See the Light Surgeon’s
SuperEverything* (see Figure 12.5 or Figure 17.5 in Chapter 17) for a good over-
view of the stage complexity of a live cinema performance.

The Material of Live Cinema

The materials of live cinema tend to be broader than those used in VJing and live
audio-visual performance. Similar to the materials used by VJs and live audio-
visual performers, they cannot claim to be entirely unique or original, given
the long history of performed visuals. Similar reference points to the structural-
materialist films of the 1960s and the scratch video of the 1990s can also be found
in many live cinema performances. The key difference in live cinema is the focus
on narrative and/or documentary elements as part of the live experience, as well
as the inclusion of live music as a key part of the experience. Chris Allen from
The Light Surgeons describes this key difference as follows:

Narrative has certainly played a large role in the development of our live
performances and we explore this in a number of ways. There is a progres-
sion that forms a visual narrative that might be quite subtle and can some-
times employ abstract visual imagery not unlike what you might see in a
VJ set. However, we always apply a structure to this material and place it in
an order of some kind that plays off or juxtaposes a more literal narrative.20
304 Steve Gibson and Stefan Arisona

Live cinema tends to be a more complex medium to both compose and

perform, but the results are often more visually and aurally spectacular and
genuinely immersive in a way that VJing rarely achieves. In addition, the pre-
composed aspect of much live cinema lends itself to repeat performances that
may be roughly the same as previously performed versions. This is rarely the case
for VJ performances. While VJs may use certain loops and effects as tropes for
certain types of music, the generally transient nature of much VJ performance
means that spontaneity is a necessary value, rather than formal coherence and
narrative logic.

The Performance of Live Cinema

As stated previously, the genuine innovation of contemporary Live Visuals cul-
ture is its real-time performance aspect. Expanding on the visual performances
done by VJs, live cinema arguably very closely resembles Wagner’s concept of
the Gesamtkunstwerk. While earlier multimedia artists such as Laurie Anderson
produced a kind of total artwork in the 1980s, these pieces often had elements
that involved only playback (such as the films for Anderson’s United States Live).
Live cinema proposes to go further, with all elements live and responsive to
each other in ways that can only be achieved by the use of computers and digital
technology.
At its most complex, such as in the aforementioned SuperEverything* project,
the number of mediums that contribute to the whole are formidable: live acous-
tic music (both Western and Asian), live electronic music and live cinematic
imagery overlaid with visual effects and abstract imagery. The real-time control
of this complex a mix requires a considerable amount of pre-planning to keep
a logical temporal and spatial f low in place. Chris Allen explains this process as
follows:

For SuperEverything* and True Fictions we gathered a lot of musical and

audio-visual material by recording and filming the musicians who we had
invited to collaborate on the projects. . . . Some of this material is cut and
sampled to be triggered live, some of it is baked into our backing tracks.
When we perform live we generally trigger a backing track with a fixed
narrative at the start of each track which might also have a “B” mix run-
ning under it that we mix to live. Then we will have banks of other audio
and visual clips that are triggered live using a drum machine to transform
them and add audio and visual effects. These are run in the VDMX soft-
ware we use and their audio is sent via a sound card to a sound mixer
that has MIDI and effects so that the audio can be effected as well as the
visual.21

The performance of this type of live cinema therefore requires a much more
thorough consideration of the audio and video signal path (in a VJ performance,
 VJing, Live Audio-Visuals and Live Cinema 305

generally the VJ simply takes the audio feed from the master mix and applies
DSP to that). The live cinema event at its best promises a much more integrated
media experience in which all elements are ‘speaking’ to each other, in a sense
creating a synesthetic experience for the audience, in which all mediums mutu-
ally reinforce one another.

Case Study 3: Live Cinema Performance: Chris Allen,

The Light Surgeons
The Light Surgeons have operated for 20 years as a creative studio creating a
variety of multimedia projects and are one of the international pioneers of live
cinema. Beginning as Live Visualists in the 1990s they evolved to incorporate
longer forms and genuinely multimedia elements in their work. They are well-
known for their recent live cinema work, as well as their involvement in the
development of VJing and the practice of creating Live Visuals for various music
acts like ZERO7, U.N.K.L.E., Sneaker Pimps and U2.
The Light Surgeons are recognized as perhaps the leading purveyors of live
cinema worldwide. They have a long trajectory of working with Live Visuals gen-
erally and a growing body of work that now incorporates narrative and non-nar-
rative film into their live performance work. They describe live cinema as follows:

The concept of live cinema is still a fairly new and developing genre within
media art that brings together experimental approaches to narrative and
non-narrative film making, with live music and the performing arts.
Rather than screening a traditional, linear edited film, a live cinema per-
formance allows artists the freedom to experiment and improvise within
a selection of different material, prepared video clips, audio visual samples
or more generative code-based plugins that can be run in VJ software such
as VDMX.22

Their landmark work SuperEverything* (see Figure 12.5) exemplifies the use of
both narrative and non-narrative filmic content in a performance context. They
have deliberately decided to pursue a more filmic approach to Live Visuals work,
in direct reaction to previous approaches in which they worked with primar-
ily abstract imagery. As has been outlined previously, most VJ performance, as
well as much audio-visual performance work, involves the use of abstract imag-
ery overlaid with effects, with any figurative or narrative visual material often
added as an aside. In opposition to this tendency, Chris Allen from The Light
Surgeons describes their intention “to create more structured work that could
convey a narrative and work in the more formal, seating situation of a cinema or
theatre.”23 This distinction is important when talking about live cinema work:
whereas most VJ performance is concentrated in the club environment, in which
the music is foregrounded and the audience is generally dancing, live cinema
is normally presented in a more theatrical setting, in which the visual element
306 Steve Gibson and Stefan Arisona

FIGURE 12.5 SuperEverything* live cinema performance by The Light Surgeons.

Photo by Nik Eagland. SuperEverything* supported by Arts Council
England. ©2013 by The Light Surgeons. Used by permission.
Source: Mail conversation with artist available on request.

is foregrounded and the audience is immersed in a large-screen or multiscreen

audio-visual environment.
The experience of live cinema therefore generally is more focused on active
viewing and listening, as opposed to the club environment, in which the visuals
may only grab the f leeting attention of the audience. Chris Allen describes the
process of using narrative strategies in live cinema to maintain audience attention
as akin to journalism:

The literal narratives in our live cinema work are normally gathered dur-
ing our production via interviews with different people. We approach this
in the same way you would with a piece of journalism, we explore differ-
ent themes through a set of questions and more open conversations with
our subjects and then edit these down before weaving them together to
form distinct sections in our performances.24

For a longer interview with Chris Allen from The Light Surgeons see Chap-
ter 17. The Light Surgeons’ work can be viewed at www.lightsurgeons.com/

Conclusion
Live cinema, as practiced by The Light Surgeons and others, points the way to
a simultaneously more focused, immersive and formally coherent model of Live
 VJing, Live Audio-Visuals and Live Cinema 307

Visuals. Similarly, audio-visual performance, such as the work done by Markus

Heckman with alva noto, illustrates a temporal logic and precise audio-visual
synchronicity that is missing from conventional VJ performance. In the next
chapter on immersive environments we will look how the inclusion of user
interaction with live audio-visuals presents yet another pathway for Live Visuals
to progress into a distinct medium.

Notes
1 Lev Manovich, The Language of New Media (Cambridge, MA: MIT Press, 2001).
2 VJzoo, Fairlight Computer Video Instrument “About” page, http://fairlightcvi.com/
wordpress/?page_id=2.
3 Thomas Edison, The Great Train Robbery (excerpt), 1903. www.youtube.com/watch?v=
chWXOwT5RkM.
4 Arkaos VJDJ page, Arkaos Officiall website, https://vj.arkaos.com/arkaos-vjdj.
5 Pascal Müller, Stefan Müller Arisona, Simon Schubiger-Banz, and Matthias Specht,
“Interactive Editing of Live Visuals,” in José Braz, Alpesh Ranchordas, Helder Araújo
and Joaquim Jorge, eds., Advances in Computer Graphics and Computer Vision, Vol. 4
(Berlin, Heidelberg: Communications in Computer and Information Science, Springer,
2007), 169–184.
6 Scheinwerfer, Interview with Steve Gibson, 13 June, 2017.
7 Scheinwerfer, Interview with Steve Gibson, 13 June, 2017.
8 Scheinwerfer, Interview with Steve Gibson, 13 June, 2017.
9 Scheinwerfer, Interview with Steve Gibson, 13 June, 2017.
10 Scheinwerfer, Interview with Steve Gibson, 13 June, 2017.
11 Markus Heckmann, Interview with Steve Gibson, 27 June, 2017.
12 Markus Heckmann, Interview with Steve Gibson, 27 June, 2017.
13 Markus Heckmann, Interview with Steve Gibson, 27 June, 2017.
14 Markus Heckmann, Interview with Steve Gibson, 27 June, 2017.
15 Markus Heckmann, Interview with Steve Gibson, 27 June, 2017.
16 Peter Greenaway, quoted by Clifford Coonan, in “Greenaway Announces the Death
of Cinema: And Blames the Remote-Control Zapper,” The Independent, October 9,
2007, www.independent.co.uk/news/world/asia/greenaway-announces-the-death-of-
cinema-and-blames-the-remote-control-zapper-394546.html.
17 Christopher Allen, Interview with Steve Gibson, 26 July, 2017.
18 “Live Cinema in the UK Report,” Live Cinema Official website, 2016, http://livecinema.
org.uk/live-cinema-in-the-uk-report/.
19 Peter Greenaway VJ Performance, notvisualmusic youtube page, 2017, www.youtube.
com/watch?v=gg6Rmvb7EPM.
20 Christopher Allen, Interview with Steve Gibson, 26 July, 2017.
21 Christopher Allen, Interview with Steve Gibson, 26 July, 2017.
22 The Light Surgeons, “What Is Live Cinema,” The Light Surgeons official website, 2017,
http://supereverything.net/what-is-live-cinema/.
23 Christopher Allen, Interview with Steve Gibson, 26 July, 2017.
24 Christopher Allen, Interview with Steve Gibson, 26 July, 2017.
13
IMMERSIVE ENVIRONMENTS
AND LIVE VISUALS
Steve Gibson

Introduction
This chapter looks at contemporary examples of immersive environments in
interactive media art that explore Live Visuals practices. Concentrating on the
works of established artists such as Jeffrey Shaw, Rafael Lozano-Hemmer, Luc
Courchesne, Char Davies and Don Ritter; newcomers such as Alan Kwan and
Shezad Dawood; and the author’s own practice, this chapter illustrates some con-
ceptual, narrative and formal models for the use of live audio-visuals in a trans-
media installation context.
With the advance of display technologies, including multiscreen projection
walls, 360-degree projectors and super-bright projectors for outdoor public use,
there has been an extension of Live Visuals practice into large public installation,
often involving the public as agents of interaction. The parallel development of
software tools for easily controlling transmedia data has made it much easier for
artists to control all media parameters in real time, thereby effectively realising
Wagner’s concept of the Gesamtkunstwerk.
Simultaneous to these technical advancements, artists have posited theories
of interaction for large-scale publics, including Lozano-Hemmer’s concepts of
‘relational architecture’ (with a nod to the Situationists) and the author’s idea of
‘synesthetic simulation’ (extending from Scriaban’s theories). These conceptual/
formal models seek to define strategies for defining and realising public interac-
tion with complex transmedia systems. Lozano-Hemmer uses motion-tracking
technology in his works such as Body Movies to enable the public to change the
face of architectural façades, thus (temporarily) rendering buildings f luid. The
author’s work is more strictly formal and relies on new scientific explanations of
the verity of synesthetic experience to inform models of audio-visual-light map-
pings, based on the matching of simultaneous media data in these realms.
DOI: 10.4324/9781003282396-17
 Immersive Environments and Live Visuals 309

Immersion as a more general theme will also be explored in relation to the

work of various artists. Immersion as a topic has broad implications and has been
discussed exhaustively in relation to both traditional and digital arts by Oliver
Grau, amongst others. Considering immersion as the “psychological perception
of essence manifested as a sensorial experience in the observer,”1 this chapter
looks at works which use various conceptual and technical strategies to immerse
the individual or audience in an artificial experience. Immersion can involve
being suffused in a technology such as Luc Courchesne 360-degree projection
systems, in which a user is surrounded by visual projections in a circular dome,
and therefore experiences the sense of being within an artificial environment. It
can also involve personal immersion in a situation, as explored in Char Davies
Osmose, in which the user experiences a solitary journey through a virtual envi-
ronment that is not dissimilar from the experience of scuba diving, or Don Rit-
ter’s Vested, in which users wear a vest that can trigger the destruction of a public
building on a large screen.
In these systems ‘immersion’ comprises the technology, the form and the
conceptual method for public interaction. This chapter ultimately argues for the
unified approach to immersion, where the form, the live audio-visual content
and the user interaction models are developed with a holistic view of transmedia.

Immersion, Virtual Reality, and Artificial Reality in

(Historical) Context
In many quarters, virtual reality is viewed as a totally new phenomenon. However, . . .
the idea of installing an observer in a hermetically closed-off image space of illu-
sion did not make its first appearance with the technical invention of computer-
aided virtual realities. On the contrary, virtual reality forms part of the core of
the relationship of humans to images. It is grounded in art traditions, which have
received scant attention up to now, that, in the course of history, suffered ruptures
and discontinuities, were subject to the specific media of their epoch, and used to
transport content of a highly disparate nature. Yet the idea goes back at least as far
as the classical world, and it now reappears in the immersion strategies of present-
day virtual art.2

As Oliver Grau forcefully argues here, the notion of immersion didn’t begin
with the development of virtual reality as a technical medium in the 1960s, or
with the beginnings of interactivity in art in the 1960s and 1970s. Instead, it has
a long history that stretches back to the Renaissance and perhaps earlier. While
a history of immersion is beyond the scope of this book, some key events are
worth outlining before moving to a discussion of the present state of immersion
Live Visuals.

Leon Battista Alberti was an early fifteenth-century art theorist who is

widely credited with formulating an inf luential method of constructing
310 Steve Gibson

images using perspective. At the risk of considerable oversimplification,

we may say that Alberti’s method established the ground for a whole tra-
dition of pictorial representation, the subsequent history of Western art,
which eventually leads to the photographic camera.3

Put simply, Alberti’s theory of vanishing point perspective places (immerses) the
viewer in the space of the scene of an artwork, “with the space inside the picture
as ‘virtual space,’”4 Alberti’s theory of perspective had a broad and demonstrable
inf luence on the visual representation of the 15th century and beyond, including
the work of painters Andrea Mantegna and Masaccio, both of whom explored
perspective as a means of immersing spectators in a scene. Masaccio “connects
the virtual spaces of his frescoes to the physical space of the chapel.”5 This work
is continued throughout the 15th and 16th centuries, particularly in architectural
attempts to use ever more elaborate schemes to immerse the audience in interior
(usually aristocratic) spaces. In essence, the Cartesian grid system that dominated
Western art from the Renaissance to the present (including in virtual environ-
ments) is a product of Alberti’s theories and their application in the 15th century
by Mantegna, Masaccio and others.
Another key development that is important for the history of immersion was
the development of 360-degree panoramas at the end of the 18th century. These
were important, as they were intended for broader public consumption:

Static and mobile touring Panoramas proliferated across Europe and North
America as a form of spectacular entertainment. The painted illusions of
the fifteenth and sixteenth centuries now left the palaces and private villas
of aristocrats and entered the public sphere as an early kind of mass enter-
tainment. . . . The spectator was positioned in the centre of the Panorama,
surrounded completely by a seamless, illusionistic painting of a landscape,
a historical event, or battle.6

The idea of 360-degree immersion in a panoramic scene has obvious resonances

with contemporary virtual environments and immersive interactive arts, given
that both use the full visual field to represent a virtual space for the viewer
to be immersed in. In addition to the panorama, later devices such as the ste-
reoscope (see Figure 13.1) represent an attempt to create the illusion of visual
immersion by mechanical means: “In the early nineteenth Century one of the
most popular ways of viewing the new ‘photographs’ was the stereoscope.” 7 A
number of stereoscopes appeared in the early 19th century, some using mirrors,
such as the Wheatstone stereoscope shown in Figure 13.1, others using two
lenses to view images in 3D by using two images shot from different angles and
presented to each eye in order to create the illusion of a third dimension. The
stereoscope was ubiquitous in the mid-to-late 19th century, and newer versions
were developed in the 20th century. Unmistakably, the immersion created by
the illusion of the third dimension has deep resonances with the experience of
 Immersive Environments and Live Visuals 311

FIGURE 13.1 Charles Wheatstone, Mirror stereoscope, 19th century, public domain.
Source: https://upload.wikimedia.org/wikipedia/commons/1/1f/Charles_Wheatstone-mirror_stereo
scope_XIXc.jpg

virtual reality and certainly informed the design of the first virtual reality (VR)
devices, such as Ivan Sutherland’s ‘head-mounted display’ from 1966 to 1968
(see Figure 13.2).
In short, “the immersive virtual realities of the late twentieth and early
twenty-first centuries are part of a continuum of technological development,
rather than an absolute and revolutionary break with earlier image forms.”8 Vir-
tual and immersive environments may appear to be a radical break with what
came before, and technologically, that is at least in part the case, but there are
clear precedents in the history of Western visual culture that illustrate the con-
tinuum from Renaissance-era painting to the VR helmet.
The full history of VR is beyond the scope of this book, but some important
developments are also key to understanding how we have arrived at virtual real-
ity 2.0 in the early 21st century. Demonstrably significant to the development of
immersive VR was the development of the aforementioned ultimate display by
Ivan Sutherland. Sutherland was not only responsible for the physical develop-
ment of a stereoscopic head-mounted device (again, see Figure 13.2), but he also
laid down some key concepts for the aims of virtual environments which, while
perhaps somewhat hyperbolic, point of the radical sense of immersion he was
proposing for immersive virtual reality:

The ultimate display would, of course, be a room within which the com-
puter can control the existence of matter. A chair displayed in such a room
would be good enough to sit in. Handcuffs displayed in such a room would
be confining, and a bullet displayed in such a room would be fatal.9
312 Steve Gibson

FIGURE 13.2 Ivan Sutherland, head-mounted display probes at Harvard University,

1968. Used by permission.
Source: [email protected]

Sutherland proposed a virtuality that emulates the physical behaviours of the real
world, that has physical laws and actions and results that are internally logical.
This system of belief is maintained in much of the work of present-day VR artists
(as well as in VR games and scientific simulations), though naturally the brutal
effects proposed by Sutherland are, as yet, unrealised.
As outlined in detail in Chapter 3, Myron Krueger was also a key figure
in the 1970s for the development of technologies of interaction and immer-
sion. Krueger was not only concerned with technological development, but he
also proposed an entire system of artificial reality in which systems would have
defined behaviours that were internally consistent (but in a distinction from
Sutherland, were not necessarily in keeping with physical laws). This is evidenced
in his previously-discussed Videoplace (1974) installation, in which a user’s outline
is transported into an “artificial reality” of creatures and objects that interact
with the user’s outline figure in consistent, but unusual ways. For example, the
video excerpts provided by Aneddotica Magazine10 illustrate different behaviours
by the on-screen creatures and objects, including creatures that defy gravity by
moving up on the arms of the user, while at the same time obey an established
rule of magnetism (they stick to the arms, regardless of their orientation). This
 Immersive Environments and Live Visuals 313

playful interpretation of artificial reality is also key to understanding the concep-

tual concerns of virtual and immersive artists, who may seek to present fantastic
versions of reality, with behaviours that are internally consistent to the rules of an
artificial world, but are not necessarily compatible with the real world.
Equally importantly to this, the first commercial VR products were created
by Jaron Lanier11 at VPL Research in the 1980s: “Among these products were a
glove device for interaction with virtual worlds (1984), head-mounted displays
that enabled users to enter 3D worlds (1987) and a networked virtual world sys-
tem (1989).”12 The devices created by VPL enabled the wave of first-generation
projects developed in the 1990s, including those such as Char Davies’s Osmose
(1995), discussed later in this chapter. While the VR devices invented by VPL
were eventually abandoned at the end of the 1990s, their form and functionality
remained largely intact for the second generation of VR devices that appeared in
the mid-2010s, including the Oculus Quest and HTC Vive.

Interactive and Digital Art in (Historical) Context

Interactive art owes its origins to experiments by Krueger and others in the
1970s, though there were self-evidently earlier developments that foreshadowed
interactivity in art (such as the audience participation pioneered by the Fluxus
artists in the 1960s and the electronic art of the late 1960s). It is worth noting that
the intersection of art, technology and interaction is represented by a confusing
number of terms used to describe the medium, including but not limited to elec-
tronic art, media art, new media art, digital art and interactive art. Effectively all
of these terms are used interchangeably, though naturally art can be digital with-
out necessarily being interactive, so here those two terms are used separately,
with digital art being all art made primarily through the use of digital devices,
and interactive art being a subset of digital art that also allows for user and audi-
ence input into the outcome of the work. In general, this discussion focusses on
interactive art used to immerse the audience in an audio-visual environment in
which they may have some agency (no matter how limited that might be).
The development of digital and interactive art accelerated in the 1980s and
1990s and included significant works such as Jeffrey Shaw’s The Legible City
(1989 – see Figure 13.3) in which the user rides on a sensor-equipped stationary
bike through a 3D city-scape of words in the shape of either Manhattan, Amster-
dam or Karlsruhe (Germany):

The Legible City completely replaces the existing architecture of these cities
with text formations written and compiled by Dirk Groeneveld. . . . The
handlebars and pedals of the bicycle interface give the viewer interactive
control over direction and speed of travel. The physical effort of cycling
in the real world is gratuitously transposed into the virtual environment,
creating a kinesthetic conjunction of the active body in the virtual domain.
A video projector projects the computer-generated image onto a large
314 Steve Gibson

FIGURE 13.3 Jeffrey Shaw and Dirk Groeneveld, Legible City, L’Immagine Elettronica,
Chiesa di San Romano, Ferrara, Italy, 1990. Used by permission.
Source: www.jeffreyshawcompendium.com/contact/

screen, and a small LCD monitor in front of the bicycle shows a simple
ground plan of each city and the immediate position of the cyclist there.13

In addition to being one of the first interactive art works that used a repurposed
device for physical interaction with a digital system, The Legible City establishes
some key techniques and concepts for the use of interaction within a digital
installation, including:

1 The use of a physical device from the real world as a means of user interac-
tion and control (in the case of The Legible City, a stationary bicycle). This
type of physical interaction (using a variety of repurposed devices) is broadly
used in a number of later interactive digital art projects.
2 The use of a large screen placed in proximity to the user in order to immerse
them in the visual field of the installation. This mode of presentation fore-
shadows the use of ever-increasingly complex projection screen setups in
live audio-visuals and installation, including 360-degree systems.
3 The use of a natural interface, wherein user actions have logical and pre-
dictable results in the system. This is also broadly (though not exclusively)
followed up in a number of later interactive installations.
 Immersive Environments and Live Visuals 315

Christiane Paul describes The Legible City as follows:

The Legible City establishes a direct connection between the physical and
virtual realm by allowing users to control the speed and direction of navi-
gation by using the bicycle’s pedals and steering handle, which are con-
nected to a computer that translates the physical actions into changes of the
landscape on the screen.14

The Legible City laid the foundations for interactive digital art’s use of immersion
within a screen-based installation, as controlled by the physical actions of the
user. The 3D textual images, while pre-rendered, are also accessed in real time,
and therefore the experience of the installation is varied depending on the twists
and turns the user decides to take on the bike interface. It is worth mentioning
that The Legible City prefigures the connection between interactive digital art
and architecture, which is born out in succeeding works such as Rafael Lozano-
Hemmer’s Relational Architecture series, discussed later in this chapter, as well as
the architectural projections of the 21st century, discussed in Chapter 14.
Shaw continued working in interactive installation and expanded into more
immersive environments in projects such as EVE (1993) which “consists of a
large inf latable dome with two video projectors on a robotic arm at its cen-
tre, which can f luidly move projected images over the inside of the dome.
The images are presented as a stereo pair, so that viewers – wearing polarizing
glasses – encounter a three-dimensional world.”15 EVE has obviously connec-
tions to the aforementioned 360-degree panoramas of the 18th and 19th cen-
turies, as well as the various stereoscopes of the 19th and early 20th centuries.
In its use of panoramic video within an interactive digital art context, it also
points to the later immersive work of artists such as Luc Courchesne and others.
Another seminal work from the 1990s that established some of conceptual and
parameters for the use of Live Visuals within the interaction digital artwork was
Rafael Lozano-Hemmer and Will Bauer’s Displaced Emperors. This project is dis-
cussed in detail in relation to its architecture in Chapter 14 (also see Figure 14.3
in that chapter), but it is worth mentioning a number of key features in relation
to its use of user interaction with Live Visuals, as well the conceptual concerns
of much interactive art:

1 The project uses a hand-based tracking system to allow users to control the
placement of projected images on the façade of a building.16 Gestural control
of images is now a fairly common technique in interactive work in general.
2 The project allows for the user to control a limited mix of pre-prepared
images in real time in a predictable manner using the tracking system. In a
sense, this use of an image database prefigures the use of similar video data-
bases and banks in VJing and audio-visual performance.
3 The project subtly references the colonial history of Austria in its superim-
position of a Mexican Hapsburg palace over its original in Linz. Interactive
316 Steve Gibson

art is generally (though not exclusively) concerned with the social and/or
political means of its production and consumption and often ref lects on this
directly in both its means of production and the way it deals with content.

The work of Shaw and Lozano-Hemmer (and others) in the early and mid-1990s
laid the technical and conceptual foundations for the control of image in real
time via user and audience input, broadly establishing the terms for the use of
immersion in a live audio-visual installation.

Relational Architecture: Rafael Lozano-Hemmer

The possibility of a relational art (an art taking as its theoretical horizon the
realm of human interactions and its social context, rather than the assertion
of an independent and private symbolic space), points to a radical upheaval
of the aesthetic, cultural and political goals introduced by modern art. . . .
What is collapsing before our very eyes is nothing other than this falsely
aristocratic conception of the arrangement of works of art, associated with
the feeling of territorial acquisition. In other words, it is no longer possible
to regard the contemporary work as a space to be walked through. . . . It is
henceforth presented as a period of time to be lived through, like an open-
ing to unlimited discussion.17

This quote from Nicolas Bourriaud’s Relational Aesthetics provides a useful

starting point for discussing Rafael Lozano-Hemmer’s Relational Architecture
series of public digital interactive art works (the number of Relational Architec-
ture pieces by Lozano-Hemmer now stands at 21).18 Bourriaud’s proposal for a
new “relational” artwork extends the artwork into the public space of the city,
eschewing private ownership of the artwork, and instead positing a new form
of art that exists in public space and is not “owned” in the traditional man-
ner. If we view Lozano-Hemmer’s various outputs in the Relational Architecture
series, including the aforementioned Displaced Emperors, we can observe that
they are almost exclusively concerned with social interaction in public space,
wherein a group of users interacts with a large-scale visual or audio-visual
environment. In Relational Architecture the visuals may be pre-recorded and/
or pre-prepared, but the interactions with them are demonstrably “live.” In
general, these environments are installed in urban locations, often with some
built-in political narrative baked into their usage (this is clearly observable in
the chosen locale for Displaced Emperors, as well as later pieces in the series such
as Voz Alta, described later).
Lozano-Hemmer describes relational architecture

as the technological actualization of buildings and public spaces with

alien memory. Relational architecture disorganizes the master narratives
of a building by adding and subtracting audiovisual elements. . . . Rela-
tional buildings have audience-activated hyperlinks to predetermined
 Immersive Environments and Live Visuals 317

spatio-temporal settings that may include other buildings, other political

or aesthetic contexts, other histories, or other physics.19

This account extends Bourriaud’s relational concept to also embody notions of

reconfiguration, recontextualisation and “making alien.” Within the Relational
Architecture series Lozano-Hemmer uses this as the theoretical framework from
which to create the individual works. This is evident in Displaced Emperors in
which two related buildings are merged, with the colonial aspects of the one
revealed in the other. It is also an important aspect of other works in the series
such as Voz Alta, in which the radio and live statements by survivors of and wit-
nesses to the 1968 police/military massacre in Mexico City are converted into
powerful f lashes of light that are pointed to key political and cultural buildings
in the city, thus making their appearance alien, but also revealing the cultural
and political relationship between these buildings and the historical event that is
being documented.20
It is also worth drawing the obvious connection between the Relational
Architecture series and the projection mapping of the early 21st century. Lozano-
Hemmer’s work in the 1990s and early 2000s paved the way for the concept of
projection mapping onto buildings, which has now become ubiquitous. Lozano-
Hemmer’s early works provide the technical means and conceptual groundwork
for projection mapping onto buildings and/or public objects, from the very early
still images used in Displaced Emperors, to the complex multiscreen projection
used in Solar Equation (2010). For the most part, contemporary projection map-
ping projects are more technically sophisticated than Lozano-Hemmer’s early
projects, often using complex 3D effects to make buildings appear radically dif-
ferent than their actual appearance; however, Lozano-Hemmer’s work is argu-
ably much more conceptually and theoretically significant than the majority
(though not all) of contemporary projection mapping work, much of which
exists only to demonstrate a visual effect and is devoid of conceptual depth. See
Chapter 14 for a further discussion of the intersection between architecture, pro-
jection mapping and Live Visuals.
One of the most significant projects from the Relational Architecture series is Body
Movies, originally shown in Rotterdam in 2001, but also mounted in Liverpool
(2002), Linz (2002), Hong Kong (2006), Québec City (2008) and other locales.

“Body Movies” transforms public space with interactive projections mea-

suring between 400 and 1,800 square metres. Thousands of photographic
portraits, previously taken on the streets of the host city, are shown using
robotically controlled projectors. However the portraits only appear inside
the projected shadows of the passers-by, whose silhouettes can measure
between two and twenty-five metres depending on how close or far away
they are from the powerful light sources positioned on the ground. A video
surveillance tracking system triggers new portraits when all the existing
ones have been revealed, inviting the public to occupy new narratives of
representation.21
318 Steve Gibson

FIGURE 13.4 Rafael Lozano-Hemmer, Body Movies, Relational Architecture 6,

Schouwburgplein, V2 Cultural Capital of Europe, Rotterdam, The
Netherlands, 2001. Photo by: Jan Sprij. Image © Rafael Lozano-
Hemmer. Used by permission.
Source: Mail conversation with artist available on request.

Key to the concept of Body Movies is the audience’s ability to embody pre-created
images in real-time, to tangibly perform with those images and each other pub-
licly and to cooperate or compete with other audience members in revealing new
portraits or purposely avoiding a switch of scenes.22 Figure 13.4 illustrates the
social nature of the interaction, with some audience members precisely embody-
ing the projected figures, while others play visual games with the images, and
still others concentrate on using their shadows to create new visual plays. Even
though the images are still, the ‘liveness’ of the visual world is emphasised by
the fundamentally different experience of Body Movies in different locales. Critic
and curator Beryl Graham describes the audience experience as follows: “Mock
violence, f lirting and cheerful obscenity were obvious popular themes, as were
quite elaborate mimes and props – pouring water into the mouths of smaller
shadows, making combined body shapers, children towering over parents, or
acting out stories.”23 Body Movies employs a limited interactive rule (as described
earlier) in order to generate maximum visual interest and intense public engage-
ment. Its visual world is demonstrably vibrant, f luid and rich in both its look and
its public appeal. As such it represents an early highpoint in the use of interactive
Live Visuals in an immersive digital installation, even though the actual visual
images are limited in form, consisting solely of still image portraits.
 Immersive Environments and Live Visuals 319

Another Relational Architecture project that expands on Lozano-Hemmer’s

concepts of public interaction with real-time visual worlds is his masterful instal-
lation Solar Equation from 2010. Solar Equation “consists of a faithful simulation of
the Sun, 100 million times smaller than the real thing.”24 Solar Equation present a
fascinating study of the collision between digital art, science and interactivity, in
that users can interact with an artificial sun that consists of a balloon onto which
is projected actual simulations of solar behaviours. Lozano-Hemmer describes it
as follows:

The piece features the world’s largest spherical balloon, custom-manufactured

for the project, which is tethered over Federation Square and animated
using five projectors. The solar animation on the balloon is generated by
live mathematical equations that simulate the turbulence, f lares and sun-
spots that can be seen on the surface of the Sun. . . . The project uses the
latest SOHO and SDO solar observatory imaging available from NASA,
overlaid with live animations derived from Navier-Stokes, reaction diffu-
sion, perlin, particle systems and fractal f lame equations. Using an iPhone,
iPod touch or iPad, people may disturb the animations in real-time and
select different f luid dynamic visualizations.25

The legitimate innovation of Solar Equation is in its use live imaging and anima-
tion (based on actual scientific data) in order to allow users to control aspects of
an artificial sun in real time. Continuing a common theme in Lozano-Hemmer’s
Relational Architecture series, Solar Equation is also a fascinating study in the use of
Live Visuals in a social, public context. The playfulness and wonder of the audio-
visual interaction with the sun through the bespoke app is very evident in the
various video documents of the project.26
Solar Equation also provides an important link back to some of the original
ideas of Sutherland and Krueger in its use of scientific insights in a public art
context. At the same time, it points to an exciting possible future for Live Visu-
als in which the representation of visual and physical data is made visible in real
time in a public environment, often (though not exclusively) by the actions of
users. This general approach is followed up in a broad number of subsequent
public interactive projects, including such diverse offerings as Squidsoup’s Aurora
Imaginaris (2019) – an evocation/simulation of the aurora, amongst other “icy”
objects27; Random International’s Rain Room – in which user movement con-
trols the behaviour of indoor “rain”28; and Lozano-Hemmer’s own Cloud Display
(2019) in which a user’s recorded speech is transformed into words formed of
water vapour.29
Lozano-Hemmer’s body of work is one of the richest examples of the inter-
relationship between public user interaction, immersion and live-audio visual
systems of various shapes. As an exemplar, it illustrates the positive conceptual
power of real-time interactive digital art. It simultaneously connects art and sci-
ence in very direct and obvious ways, bridging a gap that has been somewhat
320 Steve Gibson

defined since the Renaissance. Finally, in its use of live visual elements via
defined audience interaction methods, Lozano-Hemmer’s work present a direct
model for how to conceptualise the relation between user action and visual result
in public digital systems.

360-Degree Projection Systems and Immersion:

Luc Courchesne
Luc Courchesne is a French-Canadian digital media artist who has built a career
focussed primarily (though not exclusively) on the use of 360-degree projec-
tion systems as sites for total visual immersion in Live Visuals and audio-visuals.
Courchesne’s work provides a strong tie to the past history of immersion that
was discussed in the beginning of this chapter, including most noticeably 18th-
century panoramas and 19th-century stereoscopes. Initially experimenting
within surround multiscreen display, as in Landscape 1 from 1997, Courchesne
combined multiscreen video projection in order to totally immerse the viewer
in a scene (in the earlier case, Montreal’s Mont-Royal Park).30 As shown in the
video documentation, using either a touch interface or voice activation, viewers
can interact with various characters in the panorama, changing the scene por-
trayed or playfully chasing these characters.31 While the video footage is obvi-
ously pre-recorded, the interaction with the viewers is clearly in real time and
presents an unpredictable immersive audio-visual narrative which would rarely
(if ever) be played out the same way twice.
In order to more fully immerse the viewer, Courchesne moved beyond mul-
tiscreen projection and conceived of the Panoscope 360° in 2000.

The Panoscope 360° is an immersive projection system capable of display-

ing a full 360° × 180° hemispheric stereoscopic image from a single data
channel. A hemispheric projector is placed above a downwardly f laring
hemispheric screen; it projects an anamorphic disc image composed so the
full horizon is placed at about 4/5th of the image radius. From within the
installation, visitors see the horizon at eye level all around them, and are
immersed in a distortion free projected space. . . . The Panoscope 360° has
been developed by Luc Courchesne (US patent: 6,905,218 B2).32

The Panoscope 360° represents the most direct attempt to fully immerse a
viewer in a real-time visual field without the user of a helmet or other encum-
bered visual interface. Courchesne has used a variation of the device in a number
of projects, including You Are Here from 2011 (see Figure 13.5). As shown in the
video documentation,33 a single user is immersed in a 360-degree projection
system that totally surrounds him or her. The work situates the user within the
Bank of Montreal building in Toronto and allows them to navigate through the
visual world using an iPhone app.34
The viewer can also drop (outside the building) from the 68th f loor gallery
to the ground, an effect which is acutely immersive given the total surrounding
 Immersive Environments and Live Visuals 321

FIGURE 13.5 Luc Courchesne, left: mock-up for the Panoscope 360°, 2000; right:
Panoscope 360° as used in You Are Here, 2011. Used by permission.
Source: http://courchel.net/

visual projection system. You can also “pass” into a number of the art pieces
in the aforementioned gallery, often coming out in a different locale that has
something in connection with the object shown in the gallery. For example,
moving into the unspecified “heraldic” object at 3:25 of the video transports
the user to a panorama of one of the first branches of the Bank of Montreal in
Montreal. The user can also transport into Courchesne’s head discovering “a
repository of his own work.”35 Again the work of Courchesne here ties back to
our original discussion of historical immersion, as described by the artist state-
ment on the piece:

The experience of visual immersion developed at the end of the 18th cen-
tury with the first panoramas. . . . The irresistible appeal of cinema at the
turn of the 20th century destroyed the business of still panoramas. The
situation has changed during the 20th century with more sophisticated
projection techniques, and more so recently with computer imaging tech-
nologies. These now allow for immersive and interactive experiences that
go far beyond the experience of cinema and of the original panorama. . . .
Courchesne’s project participates in this media development in attempting
to materialize the concept of an expanded reality where the physical world
can be augmented to the point where it becomes virtual.36

The Panoscope 360° presents one of the compelling examples of single-user

immersion within an interactive live audio-visual environment. Its particular
brilliance lies in the use of a single 360-degree projection system that envelops
the user in an artificial reality, allowing them considerable freedom in navigating
through space and objects without the need for the encumbrance of a helmet or
head-mounted display of any sort. It represents one of the fullest and convinc-
ing realisations of the original concept of immersion as conceptualised by artists
from the Renaissance to the present.
322 Steve Gibson

Virtual Reality 1.0 and Immersion: Char Davies’s Osmose

While VR as a technical medium had its origins in Ivan Sutherland’s work in
the 1960s and its commercial origins in VPL’s headsets in the 1980s, as an artis-
tic and production medium, it didn’t really take off until the 1990s. This was
partly technological: rendering a 3D visual world for 360-degree headset-based
surround-viewing demonstrably required considerable computing power, some-
thing most individuals and organisations could not even dream of in the 1980s
and 1990s. It took a concerted effort from institutes such as the Banff Centre
for the Arts with their “Art and Virtual Environments” programme to get VR
projects built. The Banff Centre used a Silicon Graphics Onyx Computer in
the mid-1990s to produce a number of VR projects. These were primarily built
using Montreal’s Softimage 3D software.37
One of the co-founders of Softimage was Canadian digital artist Char Davies:

Originally a painter, Davies transitioned to digital media in the mid-1980s when

she began exploring 3D computer imaging as a means of going “beyond” the
2D picture plane. In 1987 she became a founding director of the 3D software
company Softimage, whose intuitive design philosophy arguably reconfigured
the computer graphics industry. . . . During her ten years at the company . . .
Davies began adapting its software for her own artistic purposes.38

Davis’ Osmose from 1995 (see Figure 13.6) was one of the first successful VR
installations, and to this day it endures as one of the few genuinely classic

FIGURE 13.6 Left: Char Davies. Tree Pond, Osmose (1995). Digital still captured
in real time through head-mounted display (HMD) during a live
performance of immersive virtual environment Osmose. Right: Char
Davies. Immersant wearing a stereoscopic HMD and breathing/
balance interface vest (1995). Used by permission.
Source: Tanya Das Neves, managing director of Immersence, Inc., and assistant to the artist.
 Immersive Environments and Live Visuals 323

works from the first generation of VR in the 1990s. Osmose has a unique visual
approach, which ref lects Davies’ background as a painter: quasi-abstract natural
and organic worlds (such as the forest in Figure 13.6) are semi-translucently
rendered in an almost mystical reimagining of the natural world. Interaction is
achieved by means of a unique breathing/balance chest interface (see the right
image in Figure 13.6). This interface was inf luenced by Davies’s experience as
a scuba diver: “Through use of their own breath and balance, immersants are
able to journey anywhere within these worlds as well as hover in the ambiguous
transition areas in between.”39
As the video documentation40 shows, the user f loats through the natural
world, in a sense transcending the limits of their physical reality, with the headset
imagery providing a magic-realist immersion in a way that could not have been
imagined or realised without both the 3D software of Softimage and the helmet
interface. In short, Osmose presents one of the deepest, most profound experi-
ences of total immersion in a virtual environment. Users reported an almost
ecstatic reaction to it as an experience:

Based on responses from approximately 25,000 individuals who have been

immersed in Osmose since the summer of 1995, the after-effect of immer-
sion in Osmose can be quite profound. Immersants often feel as if they have
rediscovered an aspect of themselves, of being alive in the world, which
they had forgotten, an experience which many find surprising, and some
very emotional.41

The complexity and visual richness of Osmose was extremely difficult to

replicate in other works (though Davies did complete one follow-up VR
project Ephemere). It is worth mentioning that the production team for
Osmose was quite substantial and required deep programming knowledge
from the developers, in contrast to present-day VR projects, which can be
put together by single individuals or small teams who are conversant with 3D
software such as Unity. First-generation VR suffered from this need for deep
programming expertise, as well as the expense of the computing require-
ments needed to run the projects. By the late 1990s VR seemed like a dead
end that couldn’t be achieved on any kind of mass scale and therefore was
open to charges of elitism. It took another 10 to 15 years before cheaper
devices such as the HTC Vive and Oculus Quest brought VR back as a pos-
sible medium for immersion.

Immersion in Experience: Don Ritter’s Vested

Canadian artist Don Ritter’s work Vested presents a different notion of immer-
sion in a live audio-visual artwork: one in which the participant is immersed in
the physical experience of a traumatic and violent event. While Vested does rely
on large multiscreen projection, its sense of physical immersion is quite different
324 Steve Gibson

from the panoramas of Courchesne, or the full head-body immersion of Davies’

Osmose:

Vested ranks among those artworks that attract recipients’ attention by

means of spectacular events – which ensue in any case from the latter’s
own actions – and induce them simultaneously to turn their attention to
themselves. It is the recipients’ actions, the decisions they take, that com-
prise the real arena in which work is done; and it is here that the sense of
the work is defined.42

While Courchesne’s You Are Here presents a quasi-realist, almost documentary

immersion in real locales, and Davis’s Osmose represents a magic-realist immer-
sion within fantastic landscapes, Vested immerses users in an emotional experi-
ence in which their interactive choices have moral consequences: in which they
play the role of a possible terrorist/freedom fighter, with the ability to blow up
buildings and structures, albeit virtually.

Visitors to the installation encounter a military vest hanging on a stand and

a 14m (45ft) video projection of slow moving clouds. An assistant invites
visitors to wear the vest. When the vested person walks in front of the
projection, a panorama of international buildings appears. By continuing
to walk in front of the projection, the person can navigate through pan-
oramas of well-known art museums, political buildings, ancient buildings,
towers, and temples. . . . If the vested person presses the red button on the
vest, the building panorama is overlaid with large explosions accompanied
with sound.43

As shown in Figure 13.7 and in the video documentation,44 the user is dwarfed
by the projection system, but at the same time they are obviously the central fig-
ure, due to the lights tracking them and the ominous ‘bomb jacket’ interface that
they use to trigger the virtual explosions on screen. The image of the user is also
superimposed onto the video at given moments, further inserting them into the
scene. While the piece can be viewed as an entertaining game-like environment,
in which the audience is given a chance to surreptitiously blow up famous build-
ings, the intimidating presence of the bomb interface, as well as its experience
in the very public venue of the Winter Olympics, renders the in-person moral
choices made more visceral than they would be if this were a traditional game
played at home. While the Live Visual elements are basic, mostly consisting of
still images of famous buildings, with occasional videos of explosions, they are
obviously very responsive to the moral choices of the users, as ref lected by their
willingness to press the red button in front of a potential audience.
The immersion here is less about the physical effect of immersion (although
there certainly is a physical sense of immersion) and more about the poten-
tial for immersive interactive technology to allow us to offer up choices with
 Immersive Environments and Live Visuals 325

FIGURE 13.7 Don Ritter, Vested, 2010. Interactive video-sound-light installation,

12 × 19m, Code Live, Winter Olympics Cultural Olympiad, Vancouver,
Canada. 2010. Used by permission.
Source: Don Ritter [email protected]

implied moral consequences. In a very real way this is psychological immersion,

an approach that has been followed up by a number of interactive media artists
such as Alan Kwan, but also in commercial VR projects which attempt to use
headset immersion to simulate psychological or traumatic experiences, including
examples such as Transference by Ubisoft.45

Virtual Reality 2.0 and Immersion: Alan Kwan and Shezad

Dawood
While headset-based VR took a hiatus of sorts between the early 2000s and
2010s, with the return of VR in the mid-2010s, a number of younger artists
were drawn to the medium. Naturally there were also a number of other uses
of VR 2.0 in design, gaming and medicine, but we focus here on the use of VR
in audio-visual and digital media art. It goes without saying that the availability
of extremely fast computer processers, enhanced graphics cards and, of course,
affordable VR devices such as the HTC Vive and the Oculus Quest were all
instrumental in the reinvigoration of VR as a viable medium. The work of the
new generation of VR artists therefore tends to be less restricted by access to
technology, less consumed with the mode of its production and correspondingly
more interested in its broader cultural effects and use within a wider context
than in digital media arts.
326 Steve Gibson

One of the most striking examples of VR 2.0 in an interactive digital art

context is the work of Hong Kong–born artist Alan Kwan. He completed his
MA at MIT in 2016, and both while attending there and since then he has been

primarily interested in using videogame and virtual reality technologies

to build worlds, stories, and immersive experiences that are outside of the
traditional gaming paradigm. Instead of putting you into fights, competi-
tions, and puzzle-solving, his projects seek to craft emotional journeys
that let you wander through otherworldly places, without the pressure of
completing missions or fighting monsters.46

Kwan’s VR work is particularly intriguing as it focusses on the otherworldly,

out-of-body experiences, which are occasionally fictional – as in his Alien Abduc-
tion Simulator from 2014 (see Figure 13.8) – and at other times quasi-autobio-
graphical – such as Bad Trip from 2012. Regarding the latter, Kwan states:

For over one year every moment of my life has been documented by a
video camera mounted on glasses, producing an expanding database of
digitalized memories. Using custom virtual reality software, I created a
virtual mindscape where people could navigate, and experience my mem-
ories and dreams during this period of my life.47

Bad Trip is remarkable not only because of this laborious process but also because
it manages to integrate the deep immersion of VR 1.0 artists such as Char Davies

FIGURE 13.8 Alan Kwan, Alien Abduction Simulator, 2013–2014. A full-body immersive
experience with virtual reality headset and electrical muscle stimulator.
Used by permission.
Source: Alan Kwan [email protected]
 Immersive Environments and Live Visuals 327

with the psychological immersion of artists such as Don Ritter. The piece, while
not originally intended for experience on a VR headset, presents an acutely
immerse visual landscape made up of fragments of Kwan’s recorded life com-
bined with sharp, but surreal landscapes that pass quickly by the viewer on
their way to new parts of Kwan’s ‘memory palace.’ As the video documentation
shows,48 the viewer can control the f low of the real-time images and the narra-
tive with a game controller, immersing themselves in both the intimate record of
the year in a life but also in the strange geographical worlds passing by them. The
complex database of images is expertly handled, and the sense of experiencing
a dreamlike replay of memories is highly evocative and compelling. The prom-
ise of Kwan’s work in Bad Trip is that it approaches the narrative complexity of
multiplayer games or cinema, something that most work created in VR 1.0 was
not entirely capable of (at least in part due to the lack of computing power and
storage capability in the 1990s).
Kwan’s next project, the Alien Abduction Simulator, presents an even more
psychologically and physically strange experience, albeit in a more limited nar-
rative framework. The project is more of a performance than an interactive
installation per se and consists of Kwan lying in situ with a Quest VR helmet
on and a complex muscle stimulation system attached to his body. The descrip-
tions of the work are brief and cryptic, but relying on the video documenta-
tion,49 the work has a distinct horror quality, with noise music played over a
live experience in which Kwan is presumably experiencing VR representations
of alien abduction while being jerked spasmodically by the muscle stimulation
system.
The work has some similarities with the cyborgian work of Australian artist
Stelarc, but in the case of Kwan the experience has a more game-like, quasi-
cinematic quality than anything in Stelarc’s output.50 As in Bad Trip the psycho-
logical sense of immersion is key here, and while Alien Abduction Simulator does
not have the same obvious narrative complexities of Bad Trip, it plainly is an
experience that can be easily played out in a conventional gallery context, rather
than in the more particular sphere of the media art festival.
Another artist who (even more) comfortably slots into the gallery scene is
British artist Shezad Dawood. Dawood is interesting here, as he represents an
artist working across many disciplines, whereas the artists previously discussed
(and most of the other artists working in immersive digital art and VR) operate
primarily, if not solely, within the somewhat isolated domain of digital media
art. Dawood represents a new generation of post-conceptual artists, who eschew
attachment to a single medium, but instead present a unified output by means of
the related concerns of the individual works, rather than the medium in which
they are presented. Dawood’s output includes “painting, film, film, neon, sculp-
ture, performance, virtual reality and other digital media to ask key questions of
narrative, history and embodiment.”51 The connection between all the pieces of
Dawood’s varied output is unified by the experience of embodiment (in archi-
tecture, space, or nature), which leads us back conceptually to the start of this
328 Steve Gibson

chapter and the immersive ideas of the perspective theorists and artists of the
Renaissance and beyond.
Dawood’s ambitious Leviathan Legacy series (Pt. 1 2018 and Pt. 2 2019) com-
bines science fiction, science fact and VR storytelling and expands on Kwan’s
notions of a VR-based complex narrative. Consisting of a number of films,
follow-on VR installations and a series of exhibitions, the project is an ambitious
(if not the most ambitious) VR-related art project attempted to date. Dawood
describes the project as follows:

If the original Leviathan film cycle is set in an ambiguous 20–50 years

from now, and interrogates where we might be and what the world might
look like by then, the VR trilogy would begin on a beach in Cuba in
150 years’ time. For Leviathan Legacy Pt. 1, I worked with various marine
biologists and engineers to imagine what the reef ecosystem might look
like in the Gulf of Mexico at that time: from species, to symbiosis, to the
background geopolitics. The whole Leviathan Legacy Trilogy becomes an
exercise in where speculative science meets speculative fiction. Leviathan
Legacy 2 jumps 300 years into the future, where sea levels have risen and
the Kent coastline presents an open vista all the way to the Baltic coast-
line, and new hybrid species are to be found in these waters, but have they
developed ‘naturally’ or have they been bio-engineered?52

The Leviathan series presents a fully evolved VR medium, one in which complex
narrative is attached to actual scientific research (and sci-fi speculation) and is
experienced in a deeply immersive artificial reality that is played out over a series
of films, VR pieces and supporting exhibitions. The various versions of the Levia-
than project have been extensively documented, and many include discussions
by Dawood about the project. The exhibition at Bluecoat Liverpool from 2019
collected a number of the individual pieces of the project and housed them under
one roof. As the interview with Dawood 53 makes clear, the project is described
by characters who move through the near future over the cycle of films, with
the speculations based on both scientific and culture input from various sources.
The video from the Bluecoat Exhibition also presents some of the VR aspects
of Leviathan54 and shows beautifully rendered 3D images of both an undersea
environment and a speculation on the physical and material world (including
sci-fi elements) of the further future, 100 to 150 years from now. The VR aspect
is experienced by navigating on an HTV Vive headset and allows for free move-
ment within the visual worlds of the ocean and the future landscape. While the
interaction with the live 3D visual world is not particularly innovative – from the
limited sense of the headset footage on the video documentation – it follows a
fairly standard model of VR viewing by allowing the viewer to walk, f ly or swim
through the environment – the strength of the project is its ability to connect the
disparate elements into a single powerful statement about possible futures, with an
immersive embodied experience of those in VR as the logical endpoint.
 Immersive Environments and Live Visuals 329

The ambitious nature of the Leviathan project, spanning so many mediums as

it does and covering both scientific fact and speculation narrative, presents one
of the most unified statements for the future of the virtual environment as
an artistic medium. In addition, its rich visual world, points to the possibilities
inherent in VR 2.0, in which accessible virtual technologies and relatively sim-
ply 3D software allows artists to create every more complex and detailed visual
worlds, which include real-time visuals that are responsive to the actions of users
using various VR devices (headsets, ‘wands,’ etc.). In narrative terms Leviathan
also sets a massive challenge to the digitally based narrative: by encompassing so
much rich information and speculation, with little limit given on the time scale
of the experience (since it is played out over multiple films and VR installations),
the project provides a deep example for the future of visually based VR work and
stands as arguably the first classic of VR 2.0.

Full-Body Performance and Immersion: Steve Gibson

Opto-Phono-Kinesia
My own work in body-based performance leads us back (in some respects) to
a more characteristic live audio-visual medium, which, while still remaining
focused on immersion, is generally more allied to VJing and live audio-visual
performance (see Chapter 12). For the past 25 years I have worked extensively
with motion-tracking technology, and particularly with one technology, the
Gesture and Media System (GAMS), now in its third iteration. GAMS allows
artists and designers to map 3D space with media triggers and dynamic control
curves. Up to four users can use small motion trackers to interact with audio,
video and lights using their bodies as control technologies.55
This work with GAMS has resulted in a number of performances and instal-
lations, including those intended for audience interaction with live audio-visuals
in real time such as Virtual DJ (2002–2006) and Virtual VJ (2011–2015).56 As out-
lined in my paper “Simulating Synesthesia in Spatially-Based Real-Time Audio-
Visual Performance,” in these projects

changes in sound can be synchronized with changes in light and/or real-

time visual effects (i.e. music volume = light brightness = video opacity).
These changes can be dynamically mapped in real-time to allow the user
to consolidate the roles of DJ, VJ and light designer in one interface. This
interaction model attempts to reproduce the effect of synesthesia, in which
certain people experience light or color in response to music.57

My most recent project in this area is Opto-Phono-Kinesia (OPK) (see Figure

13.9). In this project a single, trained performer navigates a complex immer-
sive space using two trackers, which turns 3D space into a performance control
surface. Using the GAMS 3.0 tracking system, the performer can access live
sound, live video, robotic lights and even fog and can manipulate each element
330 Steve Gibson

FIGURE 13.9 Steve Gibson, Opto-Phono-Kinesia, 2017–2019. Body-based performance

using motion tracking to control live sound, video, lights and fog.
Image from Northern Dance Newcastle performance, Oct. 2019. Photo
by Liam Hardy. Used by permission.
Source: Author’s own image.

by moving in 3D space. All of these mediums are mapped in space using custom
software in order to provide logical relationships between performer movement
and responses in the various mediums. The mediums are linked by predefined
connections between the elements (e.g. music notes to video and light colour),
therefore using a simulation of the effects of synesthesia as a formal method to
organise the piece and to assist the performer and audience in navigating and
understanding the connections between the mediums.
OPK is intended to be a totally immersive experience for the performer, as
well as a partially immersive one for the audience. As the various video docu-
ments of the piece demonstrate,58 the performer is surrounded by media, with
each audio-visual element quite literally at his fingertips. Robot lights follow the
performer and change in synch with the music elements (e.g. new musical notes
to new coloured lights) as well as the video (light and video colour are generally
matched). In addition, the performer can dynamically control media elements in
sync. For example, speed of motion is often used to add delay to the audio and a
scattering effect in the video.
OPK presents an alternative model for control of live audio-visuals, one in
which immersion in 3D space is also key to performer and audience under-
standing of media relationships. It also demonstrates how a reinvigoration of the
 Immersive Environments and Live Visuals 331

physical into digitally based performance can assist with not only performance
and audience comprehension of ‘liveness’ but also can provide a model for a
healthier way of interacting with digital, immersive and virtual technologies.
OPK also posits a formal model for total control of media in immersive, body-
based performance, and as such it is somewhat distinct from the more narrative
and/or conceptual VR and immersive pieces discussed in this chapter. OPK does
lead into a discussion of Live Visuals more directly, on the other hand, given its
real-time use of VJ software (Modul8), DJ/audio software (Ableton Live) and its
explicit performance aspect.
Much of contemporary Live Visuals performance (including VJing and live
cinema) is restricted somewhat by the technology of its performance toolkit
(i.e. control surfaces, tablets), which renders the performance aspect somewhat
static at best and invisible at worst. OPK demonstrates a far more direct, tangible
and visible performance interface: one in which the body as displaced in space
becomes the control technology for a total media environment.

Conclusion: Towards a Holistic Transmedia Immersion

As implied earlier, for immersive performance to be conceived as properly live,
it is essential to design an interface with direct action–result connections that are
visible to the audience. VR experienced in a headset such as the Oculus Quest
and interacted with via user movement and control “wands” has built-in action–
result within the system, particularly when experiencing a world based on a
natural interface model (i.e. anything from early physical-interactive art such as
Shaw’s Legible City, to VR sports games, to quasi-narrative art projects such as
Dawood’s Leviathan Legacy series).
The practice of Live Visuals in immersive art is obviously broad enough to
accommodate a multitude of approaches; however, referring back to the start of
this chapter, I ultimately argue for the unified approach to immersion, in which
the form, the live audio-visual content and the user interaction models are devel-
oped and deployed with a holistic view of transmedia. In this case, I use the term
‘holistic’ to describe the connections between all of the elements of the immer-
sive artwork (the technology used, the audio-visual content, the narrative, for-
mal and/or conceptual elements). Lozano-Hemmer’s Relational Architecture series
or Dawood’s Leviathan Legacy series arguably provide the best examples of proj-
ects which successfully join up all of those elements, at least for the immersive
digital artwork. They achieve this through their complexity, their masterful use
of different forms and their conceptual and narrative richness.
In a different way, my OPK project also seeks to provide a holistic view of
transmedia production, wherein all elements are considered carefully in relation
to each other and are used to enhance both the sense of ‘liveness’ in audio-visual
performance and the audience experience of a technologically mediated environ-
ment. While a simulation of synesthesia is the method used to unify the mediums
in OPK, there is no suggestion that this is the only method for providing unity
332 Steve Gibson

in immersive, total-media artworks and performances. It is posited as one pos-

sible solution to a lack of formal rigour that often exists in audio-visual and Live
Visual performance. In addition, there is also no specific reason to favour strictly
formal solutions to media integration over narrative or conceptual ones.
Ideally the live audio-visual work of the future, including VR, immersive
installation and immersive performance, will explore ways of richly integrating
the means of production with both its formal and conceptual aspects. Immersive
digital artists and audio-visual performers will benefit from looking back to art-
ists such as Norman McLaren and Sergei Eisenstein (see Chapter 2) for inspira-
tion in creating new holistic models for immersive media production. At the
same time, the new and unique features of the VR and immersive mediums bear
deep exploration, as shown in the works by Lozano-Hemmer, Dawood and oth-
ers in this chapter. The immersive features of VR headsets and full-body perfor-
mance can be intensely immersive, allowing for a profound sense of both place
and space for audience, performer and user. Their continued exploration will
benefit from considering all elements as holistically as possible and will (hope-
fully) lead to a mature medium of immersive art. In a post-pandemic world, this
seems both inevitable, desirable and promising.

Notes
1 Oliver Grau, Virtual Art: From Illusion to Immersion (Cambridge: MIT Press, 2003), xi.
2 Oliver Grau, Virtual Art: From Illusion to Immersion (Cambridge: MIT Press, 2003), 4–5.
3 Martin Lister, Jon Dovey, Seth Giddings, Iain Grant, and Kieran Kelly, New Media: A
Critical Introduction, 2nd edition (London: Routledge, 2009), 115.
4 Martin Lister, Jon Dovey, Seth Giddings, Iain Grant, and Kieran Kelly, New Media: A
Critical Introduction, 2nd edition (London: Routledge, 2009), 115.
5 Martin Lister, Jon Dovey, Seth Giddings, Iain Grant, and Kieran Kelly, New Media: A
Critical Introduction, 2nd edition (London: Routledge, 2009), 117.
6 Martin Lister, Jon Dovey, Seth Giddings, Iain Grant, and Kieran Kelly, New Media: A
Critical Introduction, 2nd edition (London: Routledge, 2009), 121.
7 Martin Lister, Jon Dovey, Seth Giddings, Iain Grant, and Kieran Kelly, New Media: A
Critical Introduction, 2nd edition (London: Routledge, 2009), 123.
8 Martin Lister, Jon Dovey, Seth Giddings, Iain Grant, and Kieran Kelly, New Media: A
Critical Introduction, 2nd edition (London: Routledge, 2009), 123.
9 Ivan Sutherland, “The Ultimate Display,” in Proceedings of the International Federation for
Information Processing (IFIP) Congress (London, 1965), 506–508.
10 “Videoplace Myron Krueger,” Aneddotica Magazine website, 2015, www.aneddotica-
magazine.com/videoplace-myron-krueger/.
11 Lanier also coined the term virtual reality.
12 Christiane Paul, Digital Art, 3rd edition (London: Thames and Hudson, 2015), 125.
13 Jeffrey Shaw, Legible City, 1989, The Jeffrey Shaw Compendium website, www.jeffreyshaw
compendium.com/portfolio/legible-city/.
14 Christiane Paul, Digital Art, 3rd edition (London: Thames and Hudson, 2015), 72.
15 Christiane Paul, Digital Art, 3rd edition (London: Thames and Hudson, 2015), 127–128.
16 For a video demonstration of the hand-based tracking system see also https://lozano-
hemmer.com/videos/artwork/displaced_emperors_linz_hd.mov.
17 Nicolas Bourriaud, Relational Aesthetics, translated by Simon Pleasance and Fronza
Woods, with the participation of Mathieu Copeland (Dijon: Les presses du réel, 1998,
2002 English translation), 14–15.
 Immersive Environments and Live Visuals 333

18 Rafael Lozano-Hemmer, “Relational Architecture,” www.lozano-hemmer.com/projects.

php?series=Relational%20Architecture.
19 Rafael Lozano-Hemmer, “Utterance 4: Relational Architecture,” Performance Research,
4(2), Summer 1999, 52.
20 See www.lozano-hemmer.com/voz_alta.php for more.
21 Rafael Lozano-Hemmer, Body Movies, Relational Architecture 6, 2001, www.lozano-
hemmer.com/body_movies.php.
22 See www.lozano-hemmer.com/body_movies.php for various videos of the piece in
different cities.
23 Beryl Graham, “Interaction/Participation: Disembodied Performance in New Media
Art,” in Jonathan Harris, ed., Dead History, Live Art? Spectacle, Subjectivity and Subversion
in Visual Culture since the 1960s, Tate Liverpool Critical Forum, vol. 9 (Liverpool: Liv-
erpool University Press & Tate Liverpool), 254.
24 Rafael Lozano-Hemmer, Solar Equation Relational Architecture 16, 2010, www.lozano-
hemmer.com/solar_equation.php.
25 Rafael Lozano-Hemmer, Solar Equation Relational Architecture 16, 2010, www.lozano-
hemmer.com/solar_equation.php.
26 See www.lozano-hemmer.com/solar_equation.php for a number of videos of the piece
in different locales.
27 Squidsoup, Aurora Imaginaris, 2019,www.squidsoup.org/portfolio/aurora-imaginaris-2018/.
28 Random International, Rain Room, 2012, www.random-international.com/rain-
room-2012.
29 Rafael Lozano-Hemmer, Cloud Display, 2019, www.lozano-hemmer.com/cloud_display.
php.
30 Luc Courchesne, Landscape One, 1997, http://courchel.net/#.
31 Luc Courchesne, Landscape One Video documentation, 1997, https://vimeo.com/5995405.
32 Luc Courchesne, Panoscope 360º, 2000, http://courchel.net/#.
33 Luc Courchesne, You Are Here, 2010, https://vimeo.com/111207649.
34 Luc Courchesne, You Are Here, 2011, http://youarehere2011.info/en/introduction.html.
35 Luc Courchesne, You Are Here, 2010, https://vimeo.com/111207649 (3:55–4:00).
36 Luc Courchesne, You Are Here, 2011, http://youarehere2011.info/en/genesis.html.
37 See the following for more information on the Banff Centre’s “Art and Virtual Envion-
ments programme: Immersed in Technology: Art and Virtual Environments, edited by Mary
Anne Moser and Douglas MacLeod (Cambridge, MA: MIT Press, 1996).
38 Char Davies, Char Davies Artist Biography, www.immersence.com/biography/.
39 Char Davies, Osmose, 1995, www.immersence.com/osmose/.
40 Char Davies, Osmose (1995) – Mini-documentary – 33 min, www.youtube.com/watch?v=
bsT59fp8LpY&feature=youtu.be.
41 Char Davies, Osmose, 1995, www.immersence.com/osmose/.
42 Ryszard Kluszczynski, “Don Ritter’s Vested: Action and Responsibility,” Move: New
European Media Art (Halle, Germany: Werkleitz, 2009), 2.
43 Don Ritter, Vested, 2010, https://aesthetic-machinery.com/vested.html.
44 See the video at the top of https://aesthetic-machinery.com/vested.html.
45 Ubisoft, Transference, 2018, www.ubisoft.com/en-gb/game/transference.
46 Alan Kwan, About, www.kwanalan.com/about.
47 Alan Kwan, Bad Trip, www.kwanalan.com/blank.
48 Alan Kawn, Bad Trip: Navigate My Mind Video documentation, 2012, https://vimeo.
com/47943812.
49 Alan Kwan, Alien Abduction Simulator, 2015, www.youtube.com/watch?v=MWOSMJ
NbJJc&feature=youtu.be.
50 See http://stelarc.org/projects.php for a detailed overview of Stelarc’s works.
51 Shezad Dawood, Biography, http://shezaddawood.com/text/.
52 Shezad Dawood, “Shezad Dawood’s Leviathan,” interviewed by Olivia Burt, Front-
runner Magazine, April 9, 2020, https://frontrunnermagazine.com/shezad-dawoods-
leviathan-legacy/.
334 Steve Gibson

53 Shezad Dawood: Leviathan, Bluecoat Liverpool YouTube page, September 18, 2019,
www.youtube.com/watch?v=9A_DMa5CGyc.
54 For a description of and visual representations from the VR parts of Leviathan see 6:15
and following from www.youtube.com/watch?v=9A_DMa5CGyc.
55 Steve Gibson, “Opto-Phono-Kinesia (OPK): Designing Motion-Based Interaction for
Expert Performers,” in Twelfth International Conference on Tangible, Embedded and Embodied
Interactions (TEI, 2018), 2, https://dl.acm.org/authorize.cfm?key=N43393.
56 See www.telebody.ws/VirtualDJ/and www.telebody.ws/VirtualDJ/virtualvj/virtualvj.
html for more.
57 Steve Gibson, “Simulating Synaesthesia in Real-Time Performance,” in Lanfranco
Aceti, Steve Gibson, and Stefan Müller-Arisona, co-eds., Live Visuals for Performance,
Gaming, Installation, and Electronic Environments (Leonardo Electronic Almanac, 2013),
215, www.leoalmanac.org/vol19-no3-simulating-synesthesia/.
58 Steve Gibson, Opto-Phono-Kinesia (Vimeo Showcase, 2018–19), https://vimeo.com/
showcase/5176262.
14
ARCHITECTURAL PROJECTIONS
Changing the Perception of Architecture
With Light

Simon Schubiger, Stefan Arisona, Lukas Treyer

and Gerhard Schmitt

Introduction
Throughout the past three decades, public space has become increasingly used
for projections of art installations, performances and as a large-scale communica-
tion surface. Evolving from a stage design approach, projections in public space
have increased in popularity as powerful projectors became more affordable, and
simultaneously the increase in computer power and speed made it much easier to
map a projection onto a building’s canvas.
The idea to repurpose a building, or other built structures, as canvases in
order to dazzle an audience with visual effects has equally fascinated artists and
marketing professionals. While there are projections focusing mostly on map-
ping visual content on a building, more often these projections go hand in hand
with a music performance, with the projection being an accompanying visualisa-
tion for a concert or the music supporting the visualisation, as laid out in detail in
Chapter 12. In addition, some works make use of interaction with its spectators
(Chapter 8). In the ‘projection community’ the term ‘architectural projections’ is
now commonly accepted for such large-scale building projections; however, the
content being expressed with these new means of visual communication often
merely uses architecture as a canvas without deep ref lection on the underlying
architectural concepts.
In this chapter, we provide the historical background by discussing early
or renowned architectural projections, such as the seminal work Son et lumiere
by Paul Houdin-Robert at Château de Chambord in France, 1952. The rela-
tionship of an active perception of architecture and computer-generated visu-
als is also illustrated by works such as Krzysztof Wodiczko’s Interrogative Design
approach, Jeffrey Shaw’s The Legible City or Rafael Lozano-Hemmer’s Displaced
Emperors. This chapter then goes beyond using architecture as a blank surface.
DOI: 10.4324/9781003282396-18
336 Simon Schubiger et al.

It analyses the architectural potential of ‘architectural projections,’ shows its

relations to traditional illumination of buildings and discusses the potential to
enhance architecture curricula. It thereby builds on the work carried out at the
Chair of Information Architecture of ETH Zurich, both in terms of graduate
courses taught1,2 and research carried out to establish foundational concepts and
techniques,3 mainly inherited from spatial augmented reality4,5 and traditional
‘linear perspective’ as employed since the Renaissance. To conclude the chapter,
we highlight the possibilities of extending architecture with Live Visuals, with
example works in dance music clubs, in public spaces such as train stations and at
popular city festivals where large custom-built light-emitting diode (LED) light
sculptures have been installed.

The Emergence of Projections onto Façades

in Art and Advertising
The projection of light has been of great interest to humans since the ancient
world. Whether it was Plato’s cave allegory or the stones of Stonehenge, the pro-
jection of light and shadow was used as a metaphor for knowledge. In the ancient
Greek theatre, which might be seen as a ‘simulation platform’ of that time, the
stage designers attempted to create spatial illusions on f lat stage elements. This
need forced them to conduct research on perspective and projecting light. As
shown in Chapter 1, the aspects of performance have consistently been linked to
questions of how to create a desired ambience for a particular setting.
These early world concepts were rediscovered in the 15th century, with the
linear perspective by Filippo Brunelleschi and the camera obscura, followed by
the invention of the magic lantern in late 16th century,6 which was primarily
used by magicians to animate usually inanimate objects or to create the belief in
bringing the dead back to life (Figure 14.1). The similarities between the magic
lantern and the magic of architectural projections are evident, despite the dif-
ferences in technology, scale and perception. At the time, the magic lantern was
prohibited because people connected it to the supernatural; however, present-
day audiences are generally conditioned for an increased intensity of large-scale
visual and audio content, such as at music festivals and concerts. Nonetheless, at
present, architectural projections can also draw negative attention, for example,
when there is an excess of visual communication (i.e. advertisement) in public
space.7
While some of the experiments at the Bauhaus may be considered as the pro-
totypes for stage projections performed at concerts later in the 20th century (e.g.
Karajan’s Berlin Philharmonic Orchestra, Pink Floyd’s The Wall tour in 1980),
the fascination for artificial light in areas where there is seasonally little sunlight
is a possible explanation for the popularity of today’s architectural projections.
Cultures that originate from the forest, mostly in northern Europe, have a long
tradition of artificial light brought into cold dark winter nights. Examples are
lit Christmas trees or Scandinavia’s St. Lucy’s Day. Most other cultures have
 Architectural Projections 337

FIGURE 14.1 19th-century magic lantern.

Source: https://en.wikipedia.org/wiki/Magic_lantern

developed their own symbolic meaning for light and shadow, sun and fire,
including Chinese and Southeast Asian shadow plays as obvious examples.
Contemporary audiences have many different associations with artificial light
in public space, with Christmas lights being only one example. Neon lights, well
known from the famous strip in Las Vegas, are another association. The city itself
becomes a huge light sculpture at night and fascinates us when viewed from the
street level, from the top of a hill or from an airplane. People are accustomed to
the fact that buildings, especially in urban areas, are illuminated at night most of
the time. They do not get stimulated anymore by an illuminated church; how-
ever, they do become fascinated by illuminations beyond the norm, such as those
impressively shown by light-based artists like James Turrell8 who have created
stunning combinations of artificial and natural lighting.
In that respect Paul Houdin-Robert’s concept of Son et lumiere, for the first
time conducted in 1952 at Château de Chambord in France, must be seen
as one of the early initiators for large-scale projections on architecture.9 The
project was concerned with the combination of sound and light to tell the his-
tory of a location with the support of a magical atmosphere. The projections
were shown primarily in France, with additional events around the world in
the 1950s and 1960s. They were an inspiration for many concert stage designs,
as well as for early projection artists in the 1990s, such as the Paris-London
based The Projection Studio, who became well-known for very large-scale pro-
jections, including full projections onto the Houses of Parliament in Westmin-
ster, London.
338 Simon Schubiger et al.

Throughout the last 30 years the capabilities of digital technology, in par-

ticular of computer graphics, have dramatically improved, and the availability of
projection technology has changed the way artists started exploring relationships
of space, time, architecture, society and culture. Polish artist Krzysztof Wodiczko
pioneered large-scale projections onto facades as well as monuments with more
than 80 public projections since 1980, such as the projection on the Hirshhorn
Museum in 1988.10 Wodiczko calls his approach ‘Interrogative Design’ where he
combines art and technology to mitigate social and cultural issues.
An important artwork is Jeffrey Shaw’s The Legible City,11 which premiered
in 1988. (see Figure 13.3 in Chapter 13). The work allowed the audience to
use a stationary bicycle to navigate through a virtual 3D world, in which the
streets are based on a real city (Manhattan, Amsterdam, Karlsruhe). These
streets are represented by 3D letters and words that let the user pursue narra-
tive threads. While Shaw’s earlier works, such as Smokescreen (1969), already
investigated the connection of space, architecture and projection, The Legible
City stands out as a work where an urban environment and the built structure
became an intrinsic part of the piece. While not being an architectural pro-
jection in the sense of other examples discussed in this chapter, it beautifully
shows how an audience experiences a city whose layout is projected onto an
abstract 3D map.
Another seminal work is Displaced Emperors by Rafael Lozano-Hemmer (Fig-
ure 14.2), presented at Ars Electronica 1997 in Linz. Through wireless 3D sen-
sors, the installation enabled the visitors to interactively reveal the interiors of
the Habsburg residence in Mexico City, Castillo de Chapultepec. These were
projected on the façade of the Linz Hapsburg castle. With its historical refer-
ences, it therefore “departs from the supposition that cultural property is cultural
poverty. As an architectural mise-en-abîme, the project supports the idea of per-
petration of culture instead of calls for its vampiric preservation.”12 Later works
by Lozano-Hemmer continue to explore the ways how the audience can interact
with light, projected content and space through sensors, such as Pulse Park (New
York, 2008; Bochum, 2012) or Solar Equation (Melbourne, 2010; Durham, 2013;
Ulm, 2015; Québec, 2018).
In 1998, Raskar et al. coined the term ‘spatially augmented reality,’13,14 which
makes use of computer graphics, geometric mapping and projectors to create a
kind of augmented reality that did not require virtual reality helmets. Pablo Val-
buena conducted one of the first applications in art in 2007 with his Augmented
Sculpture v1.2,15 also shown at Ars Electronica. Since then, spatial augmented
reality techniques became more widespread, with notable artists labels such as the
French-British ANTIVJ,16,17 Germany’s URBANSCREEN18 or Australia’s The
Electric Canvas.19 At the same time, the techniques were quickly exploited for
marketing purposes, as exemplified by PLAYMIND’s projection for Microsoft.20
Others explored the possibilities of mobile projectors, such as BlueBlastMedia’s
Target Velocity Projections21 that did not map content onto building façades, but
rather let animated symbols ‘run’ on buildings.
 Architectural Projections 339

FIGURE 14.2 Rafael Lozano-Hemmer’s Displaced Emperors at Ars Electronica in Linz,

Austria, 1997. (Image © Rafael Lozano-Hemmer. Used with permission).
Source: Mail conversation with artist available on request.

Projection mapping is a key technique for showing media content on irregu-

larly shaped physical objects such as building facades. Two-dimensional content
such as video or rendered 3D graphics is transformed and masked by software
to fit and match correctly onto the physical surfaces when projected through an
ordinary video projection system. The Philadelphia-based Klip Collective22 has
pioneered this technique, and Ricardo Rivera, a member of the Klip Collective,
even holds the projection mapping patent.23 Through multiple art and com-
mercial projects, they pushed the boundaries of projection mapping beyond aug-
menting architecture. By enhancing the experience with interactive elements24
and storytelling,25 the otherwise static facades become spatially rich, vivid stages
for media art which are magically escaping the constraints of 2D projections
screens. By selectively highlighting specific parts in the physical world though
projected light, the Klip Collective is able to create a completely new perception
340 Simon Schubiger et al.

of these usually uniformly lit, static structures for the audience.26 See Chapter 5
for a further discussion of the work of the Klip Collective.
The emergence of artists and art collectives, many of them using a do-it-your-
self approach for realising projections in public urban spaces, also indicates a close
relationship to the many forms of urban art and street art, as often some ‘guerrilla
activities’ were involved. However, while sprayed graffiti and tags mainly fulfil
the purpose to visually communicate with other subcultural groups,27 public
projections immediately attracted a larger audience, most likely due to the larger
scale they were operating at. In 2007, the Graffiti Research Lab established with
L.A.S.E.R. Tag 28,29 an explicit link between tagging and projections by combin-
ing projectors and laser pointers to tag buildings at large scale. In the commercial
arena, companies started to exploit the technical possibilities to push the scale
of projections: in 2008, projector manufacturer Christie used 27 projectors on a
building in Quebec to create an oversized screen, 600 meters wide and 30 meters
high.30
As shown throughout this section, at present there exists a broad range of
activities and possibilities creating projections in public space; however, there
is very little work dealing with explicitly taking the underlying canvas, i.e. the
architecture, into account. The following sections examine the potential to
do so in the areas of architectural communication and education, of creating
media-augmented responsive architecture and of extending architecture with
Live Visuals.

Projections as a Means of Architectural Communication

Similar to street art, art performances or even Christmas lighting, urban projec-
tions are part of public space. They combine the f lexibility and relatively low cost
of lighting with the precision of a drawing or a façade design, respectively. Apart
from its evidently public character and the possibilities to more interactively
include the neighbourhood in local planning, there is another interesting poten-
tial for architecture: generally, if architects want to test the visual perception of
a façade (or a part of it) at 1:1 scale, building a model at this scale is obviously
beyond cost and time constraints. Therefore, projections constitute a valuable
and f lexible alternative to explore new façade designs at 1:1 scale.
The important aspect of 1:1 scale for architecture is that façades are perceived
differently at different scales: typically, architects draw elevations of all the façades
in a street if they are planning a new building in order to get an overview of the
neighbourhood. Usually this totally neglects the fact that on the other side of the
street there are buildings as well, which means that in reality viewers of a façade
never really see a whole street at once, but put together the image of a façade or
a whole street in their minds. This obviously leads to stronger focus on façade
details than its overall design. This effect was observed in one of our case stud-
ies, when the projection surface was larger than the spectators’ visual field. Our
students could experience the different perception of a projection in scale 1:10
 Architectural Projections 341

and scale 1:1. While in the model they could still step back to get an overview,
in the real-scale projection there was not enough space to step back far enough.
This situation is evident in many urban settings. Thus, the simulation of
façades in 1:1 scale forces architects to consider scale and realistic perspectives.
Projections therefore can be seen as a powerful communication medium for
architects. Table 14.1 presents typical architectural concepts that we regard as
well-suited to be visualised using projections.
When shown in public spaces, projections are a useful medium to communicate
with many people at once and to support collaborative urban development pro-
cesses. In this regard, a project can be explained in much more immersive manner,
and thus is potentially more convincing or at least more plausible. Architects can
review and communicate their design decisions and explain why they replaced cer-
tain architectural elements with something they considered better suited for a par-
ticular locale. Competition juries can explain their arguments to a public audience
as well as city officials and planners, so they can be involved in decision-making.
Compared to media façades (i.e. façades with integrated active visualization
elements such as LEDs), projections do not physically affect the real façade behind
it, which allows the façade to be perceived as a normal façade during daytime.
One obvious limitation for testing different façades is that they are visible only at

TABLE 14.1 Typical architectural concepts that are well-suited to be investigated in using
architectural projections

Concept Application

Illumination Testing of different illumination and lighting variants can be

carried out with a single projection setup.
Design References to design decisions (such as proportions or
ornaments) of a given façade can be highlighted directly.
Construction Details such as construction principles, the force model,
insulation or air convection (e.g. heat exchange) can be
visualised directly on the façade.
Operation Projections can highlight factors relevant for facility
management, such as user circulation. In addition,
projections may take an active role, for example, by
guiding people through crowded spaces.
Historical context Relevant historical information of the building, the site or its
surrounding can be displayed.
Spatial context Projections can highlight relations to neighbouring buildings,
streets or squares.
Architecture theory Projections in 1:1 scale can be applied for educational
purposes to explore theoretical issues, such as the use
of perspective in Renaissance façades or the concept of
transparency by Rowe and Slutzky.31
Architecture Façades of different styles from different epochs can be
history projected, analysed and compared.
342 Simon Schubiger et al.

night. Another limitation is the relief of the existing façade, which would need
to be visually neutralised first when projecting a new façade on top of it. To solve
this pragmatically, a building or façade could be disguised in order to visualise a
new project or a renovation on top of it.
Finally, it is worth comparing projections to ongoing advancements in aug-
mented reality (AR) and virtual reality (VR), which allow augmenting a person’s
visual field directly using glasses with heads-up displays. While these approaches
achieve similar effects as those afforded by projectors and are in many cases more
practical for individual users inspecting an object, they are currently not well
suited for larger audiences.

Examining Architectural Façade Theories Using

Real-Time Projections
The manifold communication capabilities of projections described in the previ-
ous section can serve as powerful tools for architectural education and design.
A particular advantage is the possibility of looking at possible design variations
at a 1:1 scale, which is very different from inspecting small-scale models or 3D
models on a computer screen. The examples in this section provide an overview
of the possibilities of using architectural projections in design and education.

Semper’s Principle of Cladding

One theory that fits very well overlaying projections on existing structures is
Gottfried Semper’s ‘principle of cladding,’ and in particular the exploration
of change of material using projections: a structure of a particular material is
being ‘disguised’ with a different material. While the feel of a material cannot
be simulated with projections, visual implications of a construction with a dif-
ferent material can be shown. An example is shown in Figure 14.3, where stu-
dents attempted to turn a concrete wall into a brick construction. The resulting
projections make it evident that a brick building would not only completely
change the architectural expression of the building itself but also change the
impression one would get of a street and its surroundings where the building
is located at.

Proportion
The rules of proportion constitute another domain of architectural knowledge that
can be well studied and explained with projections. Many façades from the Renais-
sance until the 20th century were clearly structured, i.e. they followed specific
rules. Even though uneducated viewers may spot the regularity of a façade, they
most likely are not aware of the rules behind it. Here projections can help visualise
the rules behind a façade design, again with the advantage that is can be done at 1:1
scale, which is particularly important in the case of studying proportions.
 Architectural Projections 343

FIGURE 14.3 “Change of material” simulated with a projection on a concrete façade

by Mario Campi at ETH Zurich, Science City Campus (Image © Lukas
Treyer 2008).
Source: Author’s own image.

FIGURE 14.4 From left to right: St. Katharinen Kirche, Oppenheim 1315. The façade
was then transformed (middle) to fit the canvas façade by Mario Campi
located at ETH Zurich, Science City Campus (right) (Images: Left CC
Patrick-Emil Zörner; middle, right © Lukas Treyer 2008).
Source: Author’s own image.

Visual Overlay of Different Epochs

Another interesting approach is to use a façade as a canvas and then to overlay
virtual façades. This is how different styles and epochs can be related to each
other in a very direct way. Figure 14.4 shows an example of a gothic façade of
the St. Katharinen Kirche in Oppenheim, Germany, built in 1315 and its adap-
tion to the proportions of the concrete façade of a building by Mario Campi at
ETH Zurich’s Science City Campus. The façade of the church in Oppenheim
was chosen because of its similarities in proportion with the canvas façade. In
this instance we intended to preserve as much detail as possible; however, other
344 Simon Schubiger et al.

works might examine which rules about proportion of a given epoch should be
considered when adapting a façade to a new shape.

Perspective
With the development of perspective drawings in the early 15th century in Flor-
ence by Brunelleschi,32 the architects of that time were given a tool by which
they were able to create façade designs that included perspective effects. Even
more, they were able to rationalise space, to understand the impression of space in
a more objective manner. According to Argan and Robb,33 Brunelleschi’s inter-
est in the development of the linear perspective was closely interlinked with his
architectural interests. The Tempietto del Bramante in the San Pietro in Montorio
church in Rome,34 for instance, extends the lines of the top circle on the inner
walls with reliefs in such a way that one would think there are no inner walls.
A similar trick was used by Michelangelo on his Palazzo Senatorio in Rome35 in
which two staircases were arranged symmetrically so that their railings form the
outline of one single imaginary large staircase. Renaissance architects gained the
skill to achieve this from their spatial illusive paintings and frescos, where single
viewpoint spaces and structures seemed to be visually augmented. The play with
the perspective, e.g. through different viewpoints, is still a subject in today’s
street art. Artists like Julian Beever,36 Edgar Mueller37 and Axel Peemöller,38
to name just a few, successfully incorporate perspective distortion in their work
in such a way that holes seem to open in the street or objects appear to pop out
when viewed from a specific vantage point.
In order to examine these façade theories in practice at a large scale, a group
of 10 architecture students of ETH Zurich took the opportunity to create archi-
tectural projections as an art installation for the Stadtfest Baden 2012, a festival in
the town of Baden, Switzerland. Together with the art director and organisers
of the festival they realised projections that explored the artistic possibilities of
architectural projections on a temporary building structure built specifically for
the festival (Figure 14.5).
As indicated, the building in Baden was a wooden structure designed as a
temporary building for the festival. The façade had a width of 60 metres and a

FIGURE 14.5 Change of material and change of style projected on a temporary

building structure. Stadtfest Baden, 2012 (Image © Lukas Treyer 2012).
Source: Author’s own image.
 Architectural Projections 345

height of 8 metres and was covered by five projectors. In order to project on a

surface of this size, multiple projectors need to be synchronised. In addition, the
non-planar nature of the structure requires projection mapping. In this case, five
10,000-lumen projectors were used, and the mapping was performed using the
Mesh Warp Server software.39

Extending Architecture With Live Visuals

As previously pointed out, during the Renaissance, the introduction of per-
spective techniques allowed for much more precise planning of architecture. In
particular, the final outcome could be presented to citizens in a much clearer
way before creating the building. In addition, the ability to create perspective
drawings inf luenced architecture: architects arranged elements of a façade to
integrate perspective effects and to take into account different appearances of the
same object at different distances. More recently, visualisation techniques devel-
oped in the past, such as perspective drawing, photography, film or computer
graphics, had an impact on architecture and inf luenced the design itself. These
new techniques additionally led to new types of architecture, such as cinemas,
theatres, planetariums and newsrooms.
At present, shopping mall façades, concert halls, clubs, bars and other build-
ings are increasingly outfitted with projection canvases by design, i.e. the canvas
is becoming an active and implicit part of the overall space. Obviously, this evo-
lution is very welcome by Live Visuals performers, as they provide new possibili-
ties to combine Live Visuals with DJ and music performance. Notable examples
are the Pitch Club in Porto, designed and illuminated in 201240; the Velvet
Underground at Zouk Club in Singapore (Figure 14.6), renovated in 201141; or
the Nuits Sonores, a concert installation in Lyon, realised in 2009.42
Large-scale displays have always played an important role in public space.
In contrast to traditional 2D projection surfaces, mass assembly of high-power
LEDs allowed the construction of spatial structures. The NOVA voxel (vol-
ume pixel) display (Figure 14.7), inaugurated in 2006, was one of many proj-
ects which were realised as part of the 150th anniversary of ETH Zurich.
NOVA’s purpose was to offer a visual insight into the scientific data from
various ETH research institutes to the wider public. While the scientific data
sets originated from different sources and widely different disciplines, all were
chosen for their aesthetics when translated to the visual domain. During the
design process of the visualisations, it became quickly apparent that most of
these data sets had at least four dimensions (including time) which had to
be mapped somehow into the existing architectural environment of Zurich’s
main train station. Several ways of projecting the data onto the existing spa-
tial structure, as well as volatile projection volumes such as fog and static 3D
structures, were considered.
None of the technologies available at the time provided the desired display
capabilities within the constraints of a public space exposed to several factors
346 Simon Schubiger et al.

FIGURE 14.6 The Velvet Underground dance room at Zouk Club Singapore,
designed by Phillips Connor. The LED ceiling can be controlled
directly through a DVI output for VJ performances (laptop in front,
in this case). Additional projection-mapped surfaces are at the far end
on the wall (illuminated tetrahedral structures and artwork) (Image ©
Stefan Arisona 2012).
Source: Author’s own image.

FIGURE 14.7 The NOVA voxel display at Zurich’s main station. Left: visualisation of
the conceptual design by Martina Eberle based on projections. Right:
final LED voxel matrix (Image © Horao GmbH, 2010. Used with
permission).
Source: Mail conversation with artist available on request.
 Architectural Projections 347

such as sunlight, dust and dirt from trains and more than 300,000 daily passen-
gers, extreme heat and cold, wind and, last but not least, building regulations,
all of which considerably limited the design space. Finally, an LED-based red,
green, blue (RGB) voxel matrix was built which enabled stunning 3D effects
despite its relatively low resolution of 10 × 50 × 50 voxels. Soon after the inau-
guration the installation was extended with interactive elements to further foster
the engagement of the public with the installation. The initially planned opera-
tion time was extended twice thanks to its success with the public, and NOVA
ran for almost seven years in the end.
As these examples show, the rapid development and the dropping costs of
LED video wall technology has made it possible to use lighting elements at a very
large scale. The inclusion of lighting canvases as a design element contributes to
the holistic experience of a space and strengthens symbiosis between the visu-
als, the canvas and the architecture. In addition, the luminosity of LED panels
is much higher than what can be achieved with projectors. These two aspects
allow for a complete rethinking of how visual content can be brought into archi-
tectural space.
The availability of low-cost power LEDs have made it possible to realise
large-scale Live Visuals installations for temporary projects. An illustrative
example is the Versus installation43 at the 2017 Badenfahrt Festival in the city
of Baden, Switzerland, where the original idea was to illuminate the arc of the
city’s main bridge with projective Live Visuals. Due to the large dimensions of
the bridge (main arc 72m wide, 24m high and 18m deep) and the limited pos-
sibilities of projector positions, using projectors would have been technically
and financially too challenging. Therefore, the project team, consisting of an
architect, a designer, two computer scientists and several computer science and
design students, decided to use the void inside the bridge arc instead of directly
projecting onto its surface. As illustrated in Figure 14.8, custom-designed,
4-m-high segment letters were designed and built. With a total of nearly 4,000
power LEDs they provided a very high light intensity that would have been dif-
ficult to achieve otherwise. The segments were then used to display six-letter
antonyms on each side. The main LED display was complemented with a game
installation placed on top of the bridge that allowed players to play for words to
be displayed below.
As previously indicated in Chapter 8, large-scale display and lighting technol-
ogy have evolved hand-in-hand with other rapid developments of the informa-
tion age. A particular element is the way we can interact with this technology
and how easily we can control individual “pixels” (pixels in the sense of a basic
light unit, such as an individual LED, placed in space) with software. This enables
the connection of media and arbitrary data sources to the environment that the
display is embedded in. Thereby, the physical display itself takes a position in
the background, connecting space and content and extending architecture with
information.
348 Simon Schubiger et al.

FIGURE 14.8 The Versus installation at Badenfahrt 2017, Switzerland (Image ©

Matthias Gubler 2017. Used with permission).
Source: Mail conversation with artist available on request.
 Architectural Projections 349

Conclusion
This chapter introduced architectural projections as symbiosis of Live Visuals
and architecture, where each amplifies and extends its counterpart in terms of
effect and perception. As outlined in the introduction, Andrea Pozzo’s frescos
can be seen as an early synthesis of architecture and visuals. This interplay of
architecture, light and vision can be traced back to the ancient Greeks and
Romans, and is even found in certain prehistoric monuments that are carefully
aligned for the celestial configuration and interplay of recurring astronomical
events.
In the present, we see the proliferation of projection mapping emerging along-
side the Live Visuals explosion (see Chapter 5) which facilitated the transition
from mostly f lat surfaces onto 3D structures. Presence and immersion, as dis-
cussed in Chapter 11, require a faithful blending of reality and the digital world.
While interiors can be spontaneously spatially arranged to a certain extent, more
complex architectural design can hardly be ignored for outdoor settings. Fur-
thermore, crossing the boundary between indoor and outdoor spaces adds new
challenges such as legal constraints in the public space, which can usually be
ignored for interior settings.
While architectural projections are growing in popularity for a broad range
of applications, only a small amount of work actually investigates the use of
projections for architectural theory and practice. Our work thereby renders
architecture the subject as well as the object of Live Visuals. Based on selected
architectural areas of interest, we have highlighted the potential of applying pro-
jections for architectural communication and education.
One of the main challenges for testing architectural projection remains in
the potentially very large area of the structures to be lit, which can be techni-
cally challenging as well as expensive. This is one of the reasons why at present
the best solutions often are the traditional projection of Live Visuals from a light
source, but also from light emitting objects, commonly realised using LEDs.
Light-emitting structures transcend the notion of ‘projection’ back to its math-
ematical meaning, which we find in almost every Live Visuals software system.
With the ongoing developments of façade and lighting technology and advances
in software systems, architectural projections will remain a fascinating area for
the future as a means to put Live Visuals into the public space.

Notes
1 Sandra Wipfli and Christian Schneider, “The Sensitive Tapestry: Built Architecture as a
Platform for Information Visualization and Interaction,” in Proceedings of the 13th Inter-
national Conference on Information Visualisation (Barcelona, 2009), 486–489.
2 Christian Schneider and Stefan Müller Arisona, “Responsive Illuminated Architecture,”
in Conference presentation, ISEA 2011 (Istanbul, 2011).
3 Lukas Treyer, Sofia Georgakopoulou, and Gerhard Schmitt, “Using a Shifted Lens to
Achieve Visual Depth in Facade Projections More Efficiently,” in Proceedings of the 16th
Conference on Information Visualisation (Washington, DC, 2012), 410–415.
350 Simon Schubiger et al.

4 Ramesh Raskar, Greg Welch, and Henry Fuchs, “Spatially Augmented Reality,” in First
IEEE Workshop on Augmented Reality (1998), 11–20.
5 Oliver Bimber, Ramesh Raskar, and Masahiko Inami, Spatial Augmented Reality
(London: AK Peters, 2005).
6 Deac Rossell, Living Pictures: The Origins of the Movies (Albany, NY, SUNY Press, 1998).
7 Anastasia Koltsova, Bige Tuncer, Sofia Georgakopoulou, and Gerhard Schmitt, “Para-
metric Tools for Conceptual Design Support at the Pedestrian Urban Scale,” in Proceed-
ings of the 30th eCAADe (2012), 279–287.
8 James Turrell, Perceptual Cells (Vienna: Edition Hatje Cantz, 1992).
9 Pascal Dupont, “Sons et lumières Chambord 2006,” 2006, www.flickr.com/photos/
ysalamar/2730948881/ (Accessed December 16, 2021).
10 Krzysztof Wodiczko on his 1988 Hirshhorn Museum projection, www.youtube.com/
watch?v=XYih-aS6JK8 (Accessed December 16, 2021).
11 Jeffrey Shaw, “The Legible City,” 1988–1991, www.medienkunstnetz.de/works/the-
legible-city/ (Accessed December 16, 2021).
12 Rafael Lozano-Hemmer, “Displaced Emperors,” 1997, www.lozano-hemmer.com/
displaced_emperors.php (Accessed December 16, 2021).
13 Ramesh Raskar, Greg Welch, and Henry Fuchs, “Spatially Augmented Reality,” in First
IEEE Workshop on Augmented Reality (1998), 11–20.
14 Oliver Bimber, Ramesh Raskar, and Masahiko Inami, Spatial Augmented Reality
(London: AK Peters, 2005).
15 Pablo Valbuena, “Augmented Sculpture Series,” 2007, www.pablovalbuena.com/work/
augmented-sculpture-series/ (Accessed December 16, 2021).
16 Joanie Lemercier, “Inode A/V Project,” 2007, www.youtube.com/watch?v=_9y1
Tesw4YY (Accessed December 16, 2021).
17 Joanie Lemercier, “AntiVJ,” www.antivj.com/ (Accessed December 16, 2021).
18 Daniel Rossa, Thorsten Bauer, David Starmann, and Jonas Wiese, “555 KUBIK,” 2010,
www.urbanscreen.com/usc/41 (Accessed December 16, 2021).
19 Peter Milne, “The Electric Canvas @ the National Museum of Singapore,” 2008, www.
youtube.com/watch?v=aeXagIhHwoo (Accessed December 16, 2021).
20 PLAYMIND, “Architectural Video Project,” 2008, https://vimeo.com/2732340
(Accessed December 16, 2021).
21 Blue Blast Media, “Target Velocity Projections,” 2007, www.youtube.com/watch?v=
bG1gLT9VLf4 (Accessed December 16, 2021).
22 Klip collective, www.klip.tv/ (Accessed December 16, 2021).
23 Ricardo Rivera, “Image Projection System and Method,” US Patent 20060038814A1,
2006.
24 Klip Collective, “Interactive building projections,” www.klip.tv/red-bull-art-of-the-can
(Accessed December 16, 2021).
25 Klip Collective, “What’s He Building in There,” www.klip.tv/whats-he-building-in-
there (Accessed December 16, 2021).
26 Klip Collective, “Sundance 2014,” www.youtube.com/watch?v=bPFACantOQs (Accessed
December 16, 2021).
27 Rémy Jaccard, Urban Art Surveillance (Zurich: University of Zurich, 2012), 334.
28 Graffiti Research Lab, “L.A.S.E.R Tag,” 2007, www.graffitiresearchlab.com/blog/proj-
ects/laser-tag/ (Accessed December 16, 2021).
29 Hannes Leopoldseder, Gerfried Stocker, and Christine Schöpf, The Network for Art,
Technology and Society: The First 30 Years of ARS Electronica (Vienna: Hatje Cantz, 2010).
30 Robert Lepage and Ex Machina, “The Image Mill,” 2008, http://lacaserne.net/index2.
php/other_projects/the_image_mill/ (Accessed December 16, 2021).
31 Colin Rowe and Robert Slutzky, “Transparency: Literal and Phenomenal,” Perspecta, 8,
1963, 45–54.
32 Rudolf Wittkover, “Brunelleschi and ‘Proportion in Perspective’,” Journal of the Warburg
and Courtauld Institutes, 1953, 275–291.
 Architectural Projections 351

33 Giulio Carlo Argan and Nesca A. Robb, “The Architecture of Brunelleschi and the
Origins of Perspective Theory in the Fifteenth Century,” Journal of the Warburg and
Courtauld Institutes, 1946, 96–121.
34 Wikimedia, “Tempietto Bramante, Rome,” 2005, https://upload.wikimedia.org/wikipedia/
commons/b/b5/Roma-tempiettobramante01R.jpg (Accessed December 16, 2021).
35 Flickr User Snuffy, “Palazzo Senatorio, Rome,” 2009, www.flickr.com/photos/
snuffy/3705761656/ (Accessed December 16, 2021).
36 Julian Beever Official Website, www.julianbeever.net (Accessed December 16, 2021).
37 Edgar Mueller Official Website, https://metanamorph.com (Accessed December 16,
2021).
38 Axel Peemöller, “Eureka Carpark Typo Melbourne,” 2012, https://urbanshit.de/alex-
peemoeller-carpark-melbourne/ (Accessed December 16, 2021).
39 Mesh Warp Server, http://projection-mapping.org/tools/mesh-warp-server/ (Accessed
December 16, 2021).
40 Lyft Studio, “Pitch Club: Basement,” 2012, https://vimeo.com/53271727 (Accessed
December 16, 2021).
41 Zouk Club Official Website, www.zoukclub.com/ (Accessed December 16, 2021).
42 AntiVJ, “Nuits Sonores,” 2009, www.antivj.com/nuits_sonores/ (Accessed December
16, 2021).
43 Versus Installation by Matthias Gubler, Simon Schubiger, Stefan Arisona, Filip Sch-
ramka, Tobias Baumgartner, Cloé Hüsser, www.badenfahrt.ch/programm/versus-die-
installation (Accessed December 16, 2021).
15
DESIGN AND LIVE VISUALS
Donna Leishman

Introduction
The main purpose of this chapter is to chart how various forms of design practice
interface with Live Visual performance (LVP). This will be discussed in terms of
how design supports the production of LVP and also how design can assist LVP
artists in managing their creative identity, reaching desired audiences and selling
their work in the contemporary market. The chapter will also discuss how LVP
practices, core methods and technologies are also conversely inf luencing design
practice. The examples and insights shared in this chapter are offered as illustra-
tions of the richness of circumstances that finds design and LVP intersecting in
contemporary culture rather than an exhaustive survey.
What is design? To those on the outside, the discipline of design might seem
like an imposing and black box of a field; however, there are many kinds of
design, each aligned with different purposes and goals. The core or main pil-
lars of design are commonly understood as visual communication, product,
environment, interaction and fashion design. Within each pillar there will be
subgroups, often built along perceived expertise – for example, marketing and
graphic design are both practices within visual communication, whereas interior
and architectural design would be regarded as domains within environmental
design. Papanek (1972) describes design as “the planning and patterning of any
act towards a desired, foreseeable end.”1 In the contemporary sphere designers
purposefully order materials or processes aligned to an intent. This intent can be
socially, personally or commercially driven and is typically set out in a structured
manner in response to a design brief. Design is essentially problem solving in
action which may result in products, prototypes or new processes being gener-
ated. Designers often work for clients whose needs frame the remit for the proj-
ect or can work more collaboratively where the brief setting is more equitable.
DOI: 10.4324/9781003282396-19
 Design and Live Visuals 353

Designers are also increasingly electing to self-initiate their own briefs similar to
how artists might instigate their personal practice.
The creative process or design method is typically iterative and inf luenced
by the affordances of the media and/or the brief. In this sense, at a basic sys-
temic level, design is already similar to LVP, as both are inf luenced and arguably
driven by innovation through contemporary tools and hardware. Both designer
and Live Visual performers can also be process driven – allowing the creative/
production systems to yield understanding (through prototypes and test pieces)
towards a final outcome. However, there are nuanced tensions and counter-
points between design and LVP. For example, LV performers normally identify
as being artists rather than designers and, as such, will more readily set their own
constraints (and goals) for the realisation of their intent – traditionally artists
have also been given more sanction to work with higher-order concepts such
as emotions, expression (of a personal inner life), social critique, aesthetic tran-
scendence, etc., whereas the designated role or function of the designer, argu-
ably from modernism onwards, has been to assist and improve lived experiences
through considered and socially engaged design practices – often through the
purported virtues of innovation, efficiency and productivity. This underlying
impetus could be described as design for good. That said, design has also helped
create and sell the virtues of luxury and throwaway products.2 Victor Papanek3
termed the latter as a form of “misdesign” (i.e. practices that do not account for
the needs of all people and disregard environmental consequences). Papanek’s
original red f lag was raised in 1972 but has taken on a very contemporary reso-
nance through today’s escalation of the climate and environmental crises.
The theoretical and critical work of design studies4 also points out how the
outcomes, products and interventions of design also have secondary impacts in
the world by way of inf luencing the development of culture – this is a contribution
that sits alongside the acknowledged culture forming activities of the arts. The
power of this secondary relationship has been made even more explicit through
the work of design’s critical design community,5 who foreground designed
objects and experiences not for commercial ends, but as sites for debate and dis-
cussion. Superf lux’s project Stark Choices (Figure 15.1) is an example of this criti-
cal design approach – where they produce an experiential fictional simulation for
the client Varkey Foundation. Stark Choices speculatively depicts human life in
the year 2030, illustrating how current projections around automation might be
realised – this is delivered through audio-visual installations.

Materiality, Immateriality and Interdisciplinarity in Design

In industrialised societies almost everything in the built environment has been
‘designed’ for us; traditional material-bound products have expanded into digi-
tal products, user experience and service design that is pervasive and increas-
ingly sophisticated to such an extent that our needs are pre-emptively, or at
354 Donna Leishman

FIGURE 15.1 Stark Choices (2018) by Superf lux is an example of immersive theatrical
simulation as critical design. Client Varkey Foundation.
Source: Image credit: Superf lux. Used by permission.

times invisibly, being met. As design’s pillars and their associated sub-domains
are accompanied by the key changes in human production (advanced capital-
ism, automation) and distribution capacities (online, mobile), more overlap and
hybridity have taken place. This has resulted in diversification away from object-
or outcome-orientated ‘work’ towards a more sociological and service-centred
focus. Fallman (2003)6 also captures this by positing a correlation between design
research (as knowledge generation) with the design process (that is driven by an
outcome) and how both processes are unified by a proactive problem-solving
nature as opposed, say, to theoretical critique only. These recent shifts have also
fostered more participatory, co-design approaches whereby end users are seen as
key and formative in the design creation process.
During this transition from traditional materiality to the blending with intan-
gible digital social spheres, services and products design started to look towards
the arts and other fields – a turn towards interdisciplinarity.7 Partly, this is a logi-
cal response to the complexity and intersectionality of problems and briefs now
being presented. It is also the outcome of an innovation ideology as espoused and
exemplified by the tech industry. This move could also be regarded as an active
response to the internal limitations of design’s overreliance on the single expert
provider, based on its past craft-guild model. The fusion of art with design
practice could also be seen as response to design’s over-reliance on the rational
structured model,8 a model that assumes that there is an established problem
 Design and Live Visuals 355

that requires solving. This design-as-science model doesn’t formally acknowl-

edge designers’ imaginations, emotions or the role of experimentation and cre-
ative non-rational strategies such as ‘chance’9 which have been more legitimised
within the arts.

Live Visual Aesthetic Prerogatives

Interdisciplinarity is fundamentally about strengthening outcomes through
working with others, and this is where the role of LVP is especially interesting.
One way this occurs within Live Visual performance is through the use of sam-
pling. If we think of LVP as a by-product or branch of electronic dance music
(EDM) as well as expanded cinema, it is worthwhile considering how EDM
helped to popularise and normalise experimental prerogatives such as mixed
production and sampling methodologies (remixing, mash-up, glitch, plunder,
musique concrète, vidding10). Sampling offered the possibility to bring a whole
world of sounds and images within a performer’s toolbox. As discussed in Chap-
ter 4, sampling is also a pragmatic tool for avoiding the expense of recording
real instruments or sourcing real film, with artists turning the practice into a
“compositional tool.”11 Furthermore, musical genres and communities strug-
gling with visibility at their inception, such as dub and early hip-hop, utilised
sampling to enable a musical expression of hybrid identities.12
Sampling as an approach is agnostic on the thorny issues of taste and cultural
hierarchy. The practice of lifting sources, references, collaging and remixing
takes the practitioner consciously into the in-between space between high and
low culture – with the latter being a highly consumerist and designed envi-
ronment. The politics of sampling and remix are hinged on the possibilities of
appropriation and free creation taken from other formats, modalities or tech-
nologies. This history of unauthorised sampling is used to problematise com-
mercial structures and realities and furthermore questions what the purpose of
culture is or should be. This is a topic for discussion and debate within current
design theory.13 With each mixing of audio-visual samples, the LVP performer
draws together both content and cultural references through time and space,
something the artist Moor Mother14 has described as being like time travelling,
a method well placed to offer ref lection or commentary on the social legacies
of the past. Remix culture in particular also draws into the work the politics of
taste, as it intersects with class – through the performers’ arbitration of what is
worth sharing and worthy of inclusion in their production or remix. An exten-
sion of LVP’s contribution to taste and class can also be traced in the heritage of
early rave and techno (see Chapter 4), which was by design (and intent) a low-
cost, democratised and inclusive mixing of many socio-economic cultures, often
deeply rooted in a geographical expression of undervalued or mis-represented
communities. The dance f loor was collectively understood (sold as an idea) as
a social leveller and was desirable specifically because of this collective experi-
ence, and in such an environment access to new cultural concepts via the audio
356 Donna Leishman

or visual samples also provides a more open access to this content than through
traditional means such as cinema, galleries or television.
Audio and or visual samples, as well as being economically pragmatic as an
approach, also potentially provide audiences with accessible contextual/narra-
tive meaning within their interpretation of a LVP performance, whereas for
the performers sampling provides a rich cultural database of potential ‘content’
as an ever-widening expanse. Taking the broader perspective, the design cura-
tor Jean-Yves Leloup described electronic music as the “pulse of modernity”15
and points out that “[t]hrough its culture of mixing, remixing, sampling and
live audio-visual performances, the genre gradually spread into graphic design,
video, contemporary art, cinema, dance and major concerts. Which now bring
together music and digital arts, stage design and technological innovation.”16
Leloup amplifies both the intrinsic value of a creative crucible and the conf lation
of remixing and crossover as part of modernist innovation.
This conscious reappropriation/recontextualising strategy can become a key
aspect of an individual LVP’s practice. The Spanish performer RRUCCULLA17
(Izaskun González) is a good example of this intent, as she uses a holistic approach
in her recontextualising. González’s Live Visuals are self-created, digitally ani-
mated Dadaesque collages of textures and sourced images that sit as a comple-
ment to her similarly amalgamated musique concrète–infused sound samples.
Together the audio-visuality fosters a maximally packed set references for her
audience to experience.
Another example of reappropriation in LVP audio visual practice is the long-
standing relationship between filmmaker Adam Curtis and the band Massive
Attack. Curtis’s individual essay film oeuvre has been described as response to
the 21st century’s digital deluge, in which he f lags the destabilisation of percep-
tion and loosening of context through use of archive footage.18 This makes him
the ideal collaborator for Massive Attack, who similarly, through their use of
hip-hop and other stylistic musical references, comment on a host of past and
present eras and sentiments. Massive Attack have extended their approach further
through the self-ref lexive restaging of their Mezzanine (1998) album through the
Mezzanine XXI anniversary tour (see Figure 15.2).
Described in their press release as a “personalised nostalgia nightmare head
trip,”19 the audio-visuals of the live performance re-sequence the band’s original
tracks and the original meanings, whilst reframing these works through Curtis’s
updated videography, which uses sampling and animation to reference a host of
key culture moments during the 20-year time span (e.g. 9/11 tragedy, oxycontin
scandal, El Paso mass shooting). This time-travelling ability is used in this proj-
ect to foster a sense of noir-like nostalgia, whilst asking the audience to consider
what may come next on the stage of world geopolitics.
Alongside audio and visual sampling, there is also a trajectory in LVP of
abstract visualisation and minimalism – represented by algorithmic image mak-
ing or computational, generative and/or math art. This mode of production as
an aesthetic prerogative can be fuelled by the performer’s own sensibility and/or
 Design and Live Visuals 357

FIGURE 15.2 Massive Attack Mezzanine XXI tour (2019), with visuals by Adam Curtis.
Source: Image credit: Babycakes Romero / www.babycakesromero.com @babycakesromero. Used
by permission.

her or his close technical relationship to code in the production and composition
of the practice.
The history of design also intersects with the minimalist art movement of the
1960s – whose clean lines and ‘less is more’ as a visual ideal was amplified in the
post–World War II recovery plans. This movement drew together industrialisa-
tion and fabrication innovations to achieve a purported healthier, more hygienic
social reality through new technologies and materials. At its peak minimalism’s
form/function ethos combined with innovative production methods and became
synonymous with the very idea of ‘being modern.’ Thus, the abstracted mini-
malism of computer art may be regarded as a form of continuation of this ideal
of modernity. Today’s contemporary creative coding scene continues to explore
emerging technological tools, including generative graphical output, which at
times is synced to music and used in art as well as commercial contexts – e.g.
iTunes ‘music visualisers.’
The creation of visuals from computational data or computationally manipu-
lating the visual language for LVP can be done through many production con-
figurations – one approach is through collaboration with creative technologists.
An example of this can be seen in the CGI artist Nate Boyce and Oneohtrix Point
Never’s (Daniel Lopatin) project Reliquary House (Figure 15.3), a commission
358 Donna Leishman

FIGURE 15.3 Reliquary House (2011) by Nate Boyce and Oneohtrix Point Never,
commissioned A-V performance for MoMA/Pop Rally.
Source: Image credit: Nate Boyce. Used by permission.

from MoMA to respond to their sculpture collection through a translation of

the physical objects into audio-visuals that were in turn performed live to an
audience within MoMA. Perhaps the other end of the production/relationship
spectrum sees the audio-visuals being delivered by a singular virtuoso artist such
as Daito Manabe, Ryoji Ikeda or alva noto (aka Carsten Nicolai), all of whom
self-identify as working in the transitional area between music, art and science.
It is interesting to note that ideologically, Nicolai actively resists representational
work, an approach that he has, as a former East German, termed “audio-visual
propaganda.”20
Occupying a third position is also possible – situated in both collaborative and
soloist VJ/DJ positions is the Scottish artist Konx-Om-Pax (Tom Scholefield)
who has proven himself equally in his self-initiated practice (Figure 15.4), as well
as within the role of VJ and visual designer for many of his peers such as Clark,
Hudson Mohawke and Oneohtrix Point Never.21

Design, Live Visual Performance and the Marketplace

The sociologist Pierre Bourdieu 22 emphasised that cultural capital is intrinsically
linked to economic and social capital and refers to the collection of symbolic
elements such as skills, tastes, posture, clothing, mannerisms, material belong-
ings, credentials, etc., that one acquires through being part of a particular social
class. Bourdieu broke his concept down into three distinct layers. Embodied cul-
tural capital is drawn from the formative experiences of community, family and
 Design and Live Visuals 359

FIGURE 15.4 Konx-Om-Pax A/V set at Lunchmeat Festival (2019), Prague.

Source: Image credit: Gabriela Prochazka. Used by permission.

various key institutions. These inf luence our sense of authentic self, and within
LVP (as discussed in Chapter 11), the particular experience, social presence and
engagement of the temporary community is a key attribute – a conduit to further
embodied cultural capital. Bourdieu also offers up objectified capital – which is
typically evidenced by property and things we own. Within LVP, this could be
the purchase of a digital collectable such as a non-fungible token by an LVP art-
ist, retention of a ticket after the performance, associated merchandise or shared
personal documentation of being at the performance. His last layer is institu-
tional cultural capital – traditionally thought of as bestowed formal recognition
or qualifications in educational contexts. However, within LVP, this could be
experienced through the sense of peer prestige, that is, the recognition offered in
specialist forums and in social media. Designers have key expertise in manipulat-
ing and negotiating the multiple value systems in these exchanges, most notably
in the process of signification of objectified capital, a view supported by design
theorists Sianne Ngai and Guy Julier, who argue that contemporary aesthetics is
a motor for consumptive practices.23
As part of this reality, designers will work with LVPs and or their publishing
labels inside the known and emerging trends and structures of leisure/pleasure
consumption. Performers are required to compete in their marketplaces by com-
ing up with creative ways to distribute and monetise their audio-visual work
and any related collectable ephemera. Many audio-visual artists, their label or
rep will deploy advertising or branding specialists to pitch ideas based on the
360 Donna Leishman

FIGURE 15.5 Aphex Twin’s famous logo. Original sketch and final logo by the
graphic designer Paul Nicholson.
Source: Image credit: Paul Nicolson. Used by permission.

artist’s actual/desired persona for their work or what message needs selling. The
designer or design studio can produce performance-related artwork: logo, brand
identity (see Figure 15.5), press release, posters, album or a single artwork along-
side associated digital/social media content such as music videos, lyric videos,
behind-the-scenes documentation, edits of live events and perhaps even offer
material merchandise products and physical collateral for the event/tour itself.
Art directors have a designated role within visual communication to help cre-
ate, control or extend this developing vision for the LVP performer. As discussed in
Chapter 11, the performer or their visual identity can be designed into the LVP. As
an embodied figure or symbol, she or he becomes a key character within the stage
setup and/or becomes part of the screened visuals within the performance – the
art-directed overview is intrinsic to any cohesive artistic production. Also impor-
tant in the marketing of LVP is the promotion of the performer – a strategy that
ensures they effectively reach their audience. This is now achieved almost entirely
digitally through multiple social media platforms and is commonly a full-time
endeavour to keep this relationship maintained – a task often delegated to web
content managers, the Live Visual performer’s or the label’s budget permitting.
The extent to which these audio-visual communications are synchronous
with the aesthetics of actual LVP itself can vary – most would be aligned and
harmonious so as not to confuse or mismanage the audience’s expectations.
Some designers and creative agencies go on to develop close and long-lasting
collaborative relationships with the artists and may even direct or co-create the
visual imagery for the Live Visuals. An example of such a relationship is Björk
and Warren Du Preez and Nick Thornton Jones (WN Studio). The latter are
attributed through their direction, photography and videography as enhancing
 Design and Live Visuals 361

Björk’s ‘otherness’ as part of her visual identity through subverting conventional

notions of beauty and sexuality. WN Studio’s approach can be experienced in
Björk’s Not Get Virtual Reality documentation video (2017).24
The performer themselves can also closely manage, direct and/or control their
own public image, and this is often the case when the artists are working on a
limited budget and/or they are interested in disrupting and directly control-
ling the politics of identity norms. Zebra Katz (Ojay Morgan) is a good exam-
ple of a performer who purposefully foregrounds his black and queer identity
both within his creative persona and his creative output. Katz, alongside Arca
(Alejandra Ghersi) and Mykki Blanco, are seen as part of a cultural trend that
Waugh (2017)25 described as “digital queering” – an approach on the part of
post-internet contemporary identity which sees a host of artists embrace gender-
f luid post-humanism.
Another approach within LVP in negotiating the commercial market and
social identity is the counter-stance of active anonymity. Various electronic
dance/experimental musicians have selected this mode of identity as non-identity,
often through mask wearing during the live performance. This obscuring of
real identity on one level seems to be the antithesis of current mainstream celeb-
rity culture and thus can be understood as a countercultural tactic, an approach
claimed by Daft Punk 26; however, mask wearing and the narratives or myths
surrounding this choice can also be co-opted as a form of strategic marketing as
utilised by performers such as DeadMau5, MF Doom or The Knife.27
As an extension of Bourdieu’s cultural capital, art and LVP can be under-
stood as a commercial commodity that embodies a set of intangible concepts
or a system of signs that is often marketed, sold and in turn collected as part of
modern societies’ conspicuous consumption of cultural goods. Alongside the
esteem indicator of the likes of MoMA (NYC) and the Design Museum (Lon-
don) programming LVP into their events and exhibition spaces, we can also see
tentative steps to commoditise LVP and associated digital material through new
platforms such as SuperRare. SuperRare is a new digital marketplace/avenue to
collect and trade unique, single-edition digital artworks. Each artwork is created
by an artist in their network and tokenised as a crypto-collectible digital item
that collectors can own and trade. It has been reported that DJ 3Lau28 sold his
first collection of 33 audio-visual non-fungible tokens (NFTs) for $11.7 million,
whilst the culturally inf luential artist Aphex Twin has also sold a one-off audio-
visual clip for $128,00029 – suggesting that the NFT movement may become an
intrinsic part of the enterprise landscape for performing artists, potentially a new
economic market that LVP and designers need to position themselves within
to sustain income. As cultural commodities, the design practices of marketing
and branding play a very important role in LVP, navigating consumerism and
articulating the cultural identity/public image of LV performers. As Guy Julier
describes it, the inf luence of ‘style’ in the modern market – is now paramount.
“In the end, a style is not just the matter of the object in question but can be
understood as a what of ‘perceiving an object,’ that is, it can be seen as producing
362 Donna Leishman

a specific perceptual setting, a literacy”30 and as such has major inf luence and
ramifications in terms of how members of the public develop these literacies and
the designer’s/artist’s role in setting these perceptual conditions, along with the
ethics implied in being excluded or enabled access to culture and markets.
The changeover in our personal freedoms as the working day ends and as dark-
ness falls has long been exploited by entertainment providers, a space in which
disinhibition is promoted through forms of recreation and the offer of intoxica-
tion. Night-time performance venues, music venues and professional nightclubs
are a part of the night-time creative ecosystem. For historian Roger Ekirch, the
night holds the possibility to “loosen the tethers of the visible world.”31 It gives
permission to participate in what could be described as carnivalesque, social
behaviours that offer an important subversion/parody of the normal/dominant
work life codes. Ekirch summarises this difference: “Night, by contrast, was
neither a set piece of ritual license nor a temporary escape from reality. Instead,
it represented an alternate reality for a substantial set of the preindustrial popula-
tion, a realm of its own that, at a minimum, implicitly challenged the institutions
of the workaday world.”32 The recent global COVID-19 pandemic has thrown
into the spotlight the cultural economy of urban spaces after dark and how this
night-time economic system is important to the creative industries.33
LVPs are typically experienced in the evening and can be presented within
different environments, each of which will each offer different expectations for
the audience/customer. As discussed earlier, LVP’s heritage is that of experi-
mental and potentially disruptive provocation, once offered as a low- or no-cost
experimental practice linked into youth culture and underground countercul-
ture (see Chapter 4). LVPs, when offered as part of mainstream club culture or as
part of a festival, can also be considered as part of a hedonic ‘product’ – a concept
associated with notions of luxury. Hagtvedt and Patrick discuss how artworks
have the ability to create luxury perceptions by referring to the notion that “art
is intrinsically tied to a heritage of high culture, with connotations of exclusiv-
ity, luxury, and sophistication.”34 Mainstream club and festival culture typically
comes with a relatively high cost of admission – factoring in the drinks or food
consumed during the event – making the total experience a considered financial
outlay/consumer investment. LVPs when explored within dance club context
have also benefitted from increased cultural profile, as observed through recent
ambitious exhibitions such as Night Fever (Vitra Design Museum, V&A Dundee)
and Electronic (Design Museum, London). Both exhibitions use different case
studies to evidence the unfolding impact of club culture on broader contem-
porary culture. Jochen Eisenbrand, chief curator at the Vitra Design Museum,
points out that the economic margins of the club scene are under pressure and
need its advocates.

Today’s younger generation is meeting and exchanging digitally, and

doesn’t necessarily need the physical place to meet. Another reason is that
festivals, especially electronic dance music festivals, seem to have become a
 Design and Live Visuals 363

new competition for clubs. Lastly, of course, is the fact that vivid nightlife
always needs open urban spaces to evolve.35

It is interesting to note that the one perceived challenger to club nightlife –

electronic dance festivals – have also now evolved into interdisciplinary (visual
art and music) festivals such as Sónar (Spain), Mutek (Canada), Atonal (Germany)
or III Points (United States), and it is also not unusual to see LVPs feature explic-
itly as an actively encouraged submission genre within the like of Ars Electron-
ica, Transmediale, ZKM Karlsruhe or SIGGRAPH (Arts). Enric Palau, discussing
his festival Sónar, points out that such shifts in recognition weren’t conscious but
an act of will and a growing confidence within these communities. He describes:

What Sónar did differently was to treat this music as an important cultural
asset – programming artists in the context of a museum as opposed to in
a nightclub or a rave. That’s why Sónar By Day and Sónar By Night have
existed since the first edition. We wanted to show that this music could
exist in both contexts.36

Sónar is also an interesting case study in terms of how it has helped urban regen-
eration in its host city, Barcelona. Palau recently cited that “the last independent
report in 2015 put the financial impact of the festival at 126m euros; a 226%
increase over the previous decade, with a net value to the city of 559.7 euros
per attendee,”37 with their tickets ranging from 125 to 270 euros, showing the
additional income generated around each city visitor. Kerstin Mogull, managing
director of the Tate (London), points out how these institutions have addressed
the reality that

“[t]oday we live 24-hour lives. Tate Britain and Tate Modern are part of
the night-time economy: evening opening hours, specific events like Late
at Tate. . . . More generally, creative towns and cities depend on their
nightlife. They are the key to attracting creative people to cities in the first
place.”38

Rethinking the value of our nightlife and the night-time economy was
foisted into the public consciousness after the 2020 global COVID-19 pandemic
effectively closed it down. Customers, artists and participants alike were required
to retreat to a socially distanced and more isolated domestic interior as almost
all hospitality and urban entertainment temporarily closed down. The concept
of nightlife (and all its connotations) has a history of transgressive symbolism.39
After the COVID-19 pandemic, as the global night economy tentatively tries to
reopen, this idea of nightlife is dynamically being resignified, and at the time
of writing this chapter, it is unclear whether it will trigger a new moral panic
around youth culture and their disavowing post-pandemic safety protocols.
History has shown that if this were to happen, then such communities, feeling
364 Donna Leishman

disenfranchised, could in turn reinstate illegal gatherings as part of a burgeoning

anti-establishment behaviour. Furthermore, the public debate about mandated
vaccine passports to access such leisure experiences, if agreed, would inevitably
change the culture of such events.

The Designed Environment

Another circumstance where design meets LVP is the materiality of the per-
formance as considered through the interior (or exterior space) in which the
experience is grounded or situated. In LVP the physical contexts are often com-
binations of both mediated screen-based presence and the multidimensional
physical interior or exterior environment that represents the ‘club’ ‘venue’ or
‘space.’ Clubs as spaces have been described as windows into all that design can
offer, bringing together multiple design disciplines: “Nightclubs are an example
of a total designed experience, employing architecture, art, fashion, graphics,
lighting, performance and sound to create an immersive sensory experience
where design, music and technology meet on the dancef loor.”40
This description f lags how the club is the context for many leisure possibilities
and user/audience expectations – they are a place of social richness, and entering
such a space facilitates a change in human-to-human proximity, eye contact and
intimacy of conversation topic (see Chapter 11).
Clubs and venues have also become sites of community making and personal
identity forming. Nightclub promotors and venue owners capitalise on this by
developing a scalable brand identity for their environments (e.g. logo, interior
design, merchandise, social media content). This can be witnessed in both niche
clubs such as the SubClub (Glasgow, UK – see Figure 15.6), Haçienda (Manches-
ter, UK) or Fabric (London, UK), and the intentional extravagance of the big
venue brands such as Amnesia (Ibiza) or Ministry of Sound (London). We also
see branded hybrids who, by osmosis, share cross-over brand values from other
design disciplines, e.g. Armani Privé (Dubai/Milan), who market themselves as
the ultimate luxury destination/experience.41

Relationship Intersections
As discussed earlier, the content creation of LVP as experiences or augmented
‘products’ can take multiple forms. LVP can be produced by a studio, a collective
or collaboration between creatives or be the product of an individual performer
himself or herself. Furthermore, the complete ecology of creative production
and release will often include the artist’s label or the distributor. The performer
or the label, or both, can take on the function as the client and work to brief dif-
ferent designers. This can change with each creative work or tour or evolve into
a longer-lasting relationship. The available budget will inf luence the ambition
and technical complexity of these briefs and what designers the LVP can afford to
work with. In smaller budget settings, designers can look for efficiencies through
 Design and Live Visuals 365

FIGURE 15.6 Still from <SUBSCAN_2021> a 3D Lidar film from ISO (isodesign.
co.uk) features the famous Subclub logo. <SUBSCAN_2021> was
commissioned by V&A Dundee for their exhibition Night Fever – Designing
Club Culture and shot during COVID-19 lockdown. Used by permission.

repurposing software or approaches and amending content rather than generat-

ing bespoke new visual material, etc.
The tension between generalist and specialist is well known within contempo-
rary design. This can be seen with contemporary design studios who seek to posi-
tion themselves as latch-key providers or ‘one stop shops’ who offer a set of related
in-house services such as ISO (UK)42 or Supermafia (CH),43 who offer digital expe-
rience design (digital content, interaction, installation, moving image direction)
alongside the likes of the international agency Pentagram (UK/US/GER),44 who
offer an even more expansive array of design services (graphics, identity, prod-
ucts, exhibitions, digital, communications). Pentagram’s approach of using affiliated
partners and guest freelance designers provides both scope and agility in the high-
value design sector. An example of this one-stop shop offer can be seen in Penta-
gram’s commission by the apparel brand UNIQLO (2019) to design a large-scale
immersive and experiential showcase essentially to tell UNIQLO’s brand story.
This project included an exhibition identity, graphics, advertising, video content,
exhibition design, installations and a complex Live Visual in the form of a genera-
tive visualisation based on textile science for the shows in their HEATTECH room.
In addition to working with studios or collectives, is it very common to see
Live Visual performers engage in collaborations with different singular practice
creatives. Collaborations can be held with creative technologists, filmmakers
and visual artists, whilst analogue processes can be used within LVP production.
366 Donna Leishman

Most will end up in digital production/post-production, given the multitude

of affordances allowing for a mix of diverse media types to be brought together
(2D/3D animation, data visualisations, generative visual effects, live action, real-
time CG, game cutscenes, machinima, etc.). There are many examples of endur-
ing art direction collaborations that have matured and evolved through multiple
projects featuring Live Visuals – such as Adam Smith and Marcus Lyall (Smith &
Lyall) with The Chemical Brothers45 or Andreas Nilsson’s work with Karin
Dreijer of The Knife/Fever Ray.46 There can also be project- or tour-specific
collaborations such as Es Devlin’s work with Adele and Beyonce,47 Chiara Ste-
phenson for Björk’s Cornucopia tour (2019)48 or the cross-media collective Left/
Right Project’s installation work with UNKLE (see Figure 15.7).
Design is an important driver within capitalism, and unsurprisingly designers
can also be seen to co-opt the formal properties of Live Visuals for other com-
mercial ends. There are numerous precedents for this whereby fringe, under-
ground or alternative countercultures such as punk or psychedelia have been
tactically redeployed as a visual aesthetic to advertise non-associated branded
products.49 Furthermore, the art or artist can also be explicitly used to promote a
commercial brand,50 whereby through a form of osmosis, the product or service
benefits from this positive association and endorsement.
Given the growing recognition that LVP has been receiving culturally in the
mainstream, design can also be seen to be utilising the more formal qualities of

FIGURE 15.7 Beyond the Road, by Colin Nightingale and Stephen Dobbie (A Right/
Left Project) and James Lavelle (UNKLE), immersive exhibition held
at the Satchhi Gallery.
Source: Image credit: Julian Abrams. Used by permission.
 Design and Live Visuals 367

LVP. There has been a notable rise in public-facing urban projected events such
as – Deadmau5’s Nokia Lumia sponsored LV performance in London (2011).51
Similarly, there has a co-opting of LVP community engagement structures
through the commercialisation of the seasonal summer festivals, whose origins
in countercultural movements are increasingly supplanted through explicit or
covert corporate sponsorships. Sponsors, in turn, find it desirable to be associated
with these communities and the inherent community values. The Eaux Claires
festival was founded by Aaron Dessner (The National) and Justin Vernon (Bon
Iver), who in turn curated the festival. In a 2017 interview Dessner described
Eaux Claires as an ‘anti-music-festival festival.’52 This represents a trend that
is moving away from mass festivals to the curation of boutique events that are
branded as being not only smaller but more independent and more intimate
than Coachella or Lollapalooza, who have an estimated number of attendees of
250,000 and 400,000, respectively. There is also a growing interest in artist-led 53
or artist-curated events: the UK festival All Tomorrow’s Parties is an early prec-
edent (2001–2016) of this current phenomenon.
Recent developments in commercial immersive media have meant increased
delivery of compelling virtual, augmented and mixed reality experiences, lead-
ing to renewed questions based on where media ends and reality begins and
what might future reality do for audiences? LVP, at its core, is an interdisciplin-
ary and intrinsically immersive experience, offering different and potentially
more emotional and liberating ways for humans to socialise and interact with
each other. Immersive or responsive design is another example of boundary
blending with LVP as used within design as a new engagement strategy. This
is often realised through the adoption of immersive live performance and/or
installations and projections within a range of design practices offering mul-
tisensorial and branded experiences that are also consciously ‘Instagrammable’
environments. Retail brands – especially those in the luxury market – are mov-
ing into immersive gallery and/or hybrid retail shops as user experiences – an
example of this is the Comme des Garçons’–owned Dover Street Market (Lon-
don, Tokyo, NYC, Singapore, Beijing), which is described as concept store
that positions and sells fashion brands through creative physical installations.
Another example of this cross-discipline blurring is the artist FKA Twig’s invi-
tation to be creative lead of an ambitious immersive theatre sponsored by Veuve
Clicquot. FKA Twig’s project titled Rooms combined 30 performers with 12
set designers to deliver the immersive three-day event to the public.54 Another
high-profile interdisciplinary fusion of design meets LVP culture is the DJ Plas-
tikman’s work with Raf Simons (fashion designer) and Rem Koolhaas (archi-
tect), who together produced an audio-visual ‘digital runway,’ wherein Prada
models performed the Prada collection in a digitally rendered mausoleum to the
audio sounds of Plastikman. This was streamed live55 and then edited as a docu-
mented episodic experience afterwards. These projects highlight the contempo-
rary appetites within commercial brand to push and develop their engagement
strategies through utilising associated LVP techniques and/or the creativity
368 Donna Leishman

of the artists themselves. Installations/immersive exhibitions are becoming

increasingly synonymous with innovative audience engagement for design and
are a burgeoning design sector.
Space or projection mapping is regarded as a new and increasingly important
marketing tool in the production design of installations and immersive exhibi-
tions. Sophisticated projection mapping broke through within live music and
the visual arts around the mid-2000s. In today’s environment the approach is
now used to transform large-scale surfaces such as high-rises, grand buildings,
complex surfaces such as sculptures, through to taking over ordinary objects and
spaces. This approach transforms these surfaces into temporary pieces of visual
art often as tactical communications (media launches, celebrations) and is often
deployed at considerable production complexity and expense. An example of this
high-profile media event approach is Double Take Projection’s work that fea-
tures a carefully mapped optical illusion of giant Irn-Bru drink bottles holding
up Scotland’s Forth Bridge.56 Projection mapping builds on the phenomenon of
artificially lit public advertising as showcased in the semi-permanent/permanent
urban spaces of NYC’s Times Square, Singapore’s Orchard Road or London’s
Piccadilly Circus. Live Visual projections have also been utilised as a form of
guerrilla marketing – whereby the projected messages are political acts of protest
illicitly deployed in the public space. An example of this is Led By Donkey’s
(@ByDonkeys), a British campaign group who frequently use unsanctioned pro-
jection mapping as a form of street theatre, with a goal to raise awareness and
leverage extra media coverage around a host of social issues. This disruptive
mode or usage of Live Visual practice is an interesting return to the Fluxus-like
ideology that blurs space/the art object with politics (see associated discussion on
Fluxus in Chapter 3).

Conclusion
Design plays an important role in how LVP is produced and is done in various
configurations, between LVP as the client or in a more collaborative relationship
structure. Design services can be offered by an individual designer, by a specialist
design studio or by larger studio who can cater for all designed elements of the
LVP production, including the physical design of a staged performance. Both the
designer’s promotional skills and the broader inf luence of design on culture help
to inform both the soft and hard sell of how LVP is in branded and commodified
within the marketplace, and these functions play a notable role in the continued
evolution of LVP.
With the trajectory of design as a discipline pointing towards increased poros-
ity, hybridity, sociological awareness, and service-centred practices, questions
arise. Will designers, in dialogue with Live Visual performers and LVP audi-
ences, be able to co-create new (and authentic to LVP’s histories) applications
of the practice? The power dynamics of this dialogue and relationship structure
will be key here, as Live Visual performers are well positioned to argue and
 Design and Live Visuals 369

understand the intrinsic characteristics and benefits of LVP as a situated experi-

ence and as a culture or community.
Inevitably, designers and design disciplines will actively explore the com-
mercial applications of LVP, but this could be done through the superficial lens
of aesthetics/styles or technologies. As this chapter discusses, there are many
salient qualities worthy of future consideration – for this author, the ability for
LVP through the remix prerogative to counter or critically comment on the
social legacies of the past, changes in perception and mainstream culture has
positive implications for socio-cultural participation and cultural diversity at
large. Secondly as Eisenbrand 57 points out, increasing numbers of us are meeting
and exchanging digitally, LVP participation can potentially offset this trend and
through participation in live events offer positive conditions to facilitate human-
to-human proximity, eye contact and intimacy of conversation – all socialisation
skills that may being eroding in the technologically mediated workday. LVPs
may also be useful as an artistic community for the undervalued or misrepre-
sented, providing a useful break from socio-normative behaviours, as well as a
space to generate new identities (see Chapter 11). A rich countercultural seam is
essential for the life cycle of mainstream culture, protecting against homogenisa-
tion and entropy. Kotler and Scheff describe “augmented products” as products
that have “features and benefits are beyond what the target audience normally
expects.”58 In this sense, LVP might offer (through the promise of something
different, novel or experimental) social experiences that individuals are not con-
scious of even needing yet would benefit from accessing and experiencing.
Whether designers will be able to authentically or best utilise LVP and all
the facets within the practice, or through a more superficial adoption become a
diluting or distorting external inf luence, is unknown and is in some sense mir-
rored by design’s inherent duality (i.e. of design for social good or design for
commercial ends only). We can already see traction with LVP in the design-for-
profit models through activities such as the purely consumerist positioning of
elite deluxe club brands (Armani Prive) and high-value luxury product collabo-
rating with LVP and associated artists (Prada, Uniqlo, Veuve Clicquot). These
brands benefit by association – capitalising on LVPs’ perceived countercultural
and/or innovation credentials to ultimately sell more (designed) products. LVP
artists in this instance can benefit from exposure to new and different audiences,
as well as potential identity change by their association with the brand they are
choosing to collaborate with. We can also see a potentially more positive trend
in the form of smaller, more participatory-focused audio-visual festivals where
intimacy and new audience dispositions (inclusive mindset) are typically being
fostered.
Jaron Lanier, the original advocate of utopian possibilities of virtual reality,
also points out that society should never separate a discussion of technological
advance from its human effect. He proposed that we need to actively “dou-
ble down on being human.”59 Extending the notion of the LVP as a portable
unbound form that is transcending the traditional club – it is possible that LVP
370 Donna Leishman

will be increasingly used to create social events in public civic spaces – a pos-
sibility accelerated by new post–COVID-19 pandemic social norms, i.e. of being
outside, together, feeling safe. The human experience and design of these social
spaces are likely to be drivers in ongoing metropolitan economic reconfigura-
tions in line with ongoing changes in perception regarding work, leisure and
touristic behaviours, and Live Visual performers, with support from design, will
be key to delivering all these transformations.

Notes
1 Victor Papanek, Design for the Real World: Human Ecology and Social Change (New York:
Pantheon Books, 1972).
2 Advertising as a design discipline became a major influencing force as part of the expan-
sions in 19th-century Britain and America, wherein commercial for-profit markets
expanded in line with globalised capitalism.
3 Victor Papanek, Design for the Real World: Human Ecology and Social Change (New York:
Pantheon Books, 1972), 3.
4 Design studies is an academic community focused around understanding design process
across all design disciplines. It is an interdisciplinary forum exploring the fundamental
aspects of design activity, from cognition and methodology to values and philosophy.
5 See: Anthony Dunne, Hertzian Tales: Electronic Products, Aesthetic Experience and Criti-
cal Design (London: RCA CRD Research Publications, 1999) and Anthony Dunne
and Fiona Raby, Design Noir: The Secret Life of Electronic Objects (London: Bloomsbury,
2001).
6 Daniel Fallman, “Design-Oriented Human: Computer Interaction,” in CHI ’03: Pro-
ceedings of the SIGCHI Conference on Human Factors in Computing Systems (New York,
2003), 225–232.
7 Julie Klein, Interdisciplinarity: History, Theory, and Practice (Detroit: Wayne State Univer-
sity Press, 1990). Within design Victor Papanek’s (1972) Design for the Real World he
argues for cross-disciplinary in teams, as did other high-profile modernist designers such
as Frank Lloyd Wright, Le Corbusier, and Walter Gropius.
8 The rational model or ‘design science’ is discussed here: Herbert A Simon, The Sciences
of the Artificial (Cambridge: MIT Press, 1969). A more plural of position is charted by:
Ulla Johansson-Sköldberg, Jill Woodilla, and Mehves Çetinkaya, “Design Thinking:
Past, Present and Possible Futures,” Creativity and Innovation Management, 22(2), 2013.
9 ‘Chance’ was a significant creative strategy and process that fosters a lack of control over
the nature of the outcomes. As a creative method it has been used by a number of writ-
ers, symbolist poets and artists linked to the Dada, Surrealist and Fluxus movements.
See: Margaret Iversen, Chance (Cambridge, MA: MIT Press, 2010).
10 Coppa (2008), describes how fannish vidders use music in order to comment on or
analyse a set of pre-existing visuals, to stage a reading or occasionally to use the footage
to tell new stories.
Francesca Coppa, “Women, Star Trek, and the Early Development of Fannish Vid-
ding,” Transformative Works and Cultures, (1), 2008.
11 Brian Eno, “The Studio as Compositional Tool,” in Cristoph Cox and Daniel Warner, co-
eds., Audio Culture: Readings in Modern Music (New York: Continuum, 2004), 127–137.
12 Matt Masson, “We Invented the Remix,” in The Pirate’s Dilemma: How Youth Culture Is
Reinventing Capitalism (New York: Free Press, 2008), 68–102.
Mark Katz, Capturing Sound: How Technology Has Changed Music (Los Angeles: UCP,
2004).
13 Design theorists discussing the intersection of culture and design include:
 Design and Live Visuals 371

Guy Julier, Anders Munch, Mads Nygaard Folkmann, Hans Christian Jensen, and
Niels Peter Skou, Design Culture: Objects and Approaches (London: Bloomsbury,
2019); Guy Julier, The Culture of Design (London: SAGE, 2013); Peter Dormer, The
Culture of Craft: Status and Future (Manchester, Manchester University Press, 1997).

14 Jordan Darville, “Moor Mother on Collaboration, Community, and the Power of

Sampling,” Thefader.com, 2020, www.thefader.com/2021/09/21/moor-mother-on-
collaboration-community-and-the-power-of-sampling.
15 Jean-Yves Leloup, Gemma Curtin, and Maria McLintock, Electronic: From Kraftwerk to
the Chemical Brothers (London: The Design Museum, 2021), 8.
16 Jean-Yves Leloup, Gemma Curtin, and Maria McLintock, Electronic: From Kraftwerk to
the Chemical Brothers (London: The Design Museum, 2021), 8.
17 RRUCCULLA, “Live SHuSH A/V Show,” Clips filmed in Club Kokomo, Groningen
(NL), YouTube page, 2019, www.youtube.com/watch?v=pmiLqrmvzsQ.
18 Rob Coley, “Destabilized Perception: Infrastructural Aesthetics in the Films of Adam
Curtis,” Cultural Politics, 14(3), 2018, 304–326.
19 Henry Bruce-Jones, “Massive Attack Announce Mezzanine XXI 2019 Tour Featuring
Elizabeth Fraser,” Factmag.com, 2018, www.factmag.com/2018/10/30/massive-attack-
announce-mezzanine-tour/.
20 Peter Kirn, Gaze into the Geometric Sound and Visual World of Alva Noto, with UNIEQAV,
2020, https://cdm.link/2020/02/alva-noto-unieqav/.
21 The collaborative projects where Scholefield works as a visual designer is discussed in
the interview: Riccardo Villella, “Truancy Volume 175: Konx-Om-Pax,” 2017, http://
truantsblog.com/2017/truancy-volume-175-konx-om-pax/ also see:
22 Pierre Bourdieu, Distinction: A Social Critique of the Judgement of Taste (London: Rout-
ledge & Kegan Paul, 1984).
23 Sianne Ngai, Our Aesthetic Categories: Zany, Cute, Interesting (Cambridge, MA: Harvard
University Press, 2012), 1–19.
Guy Julier, The Culture of Design (London: SAGE, 2000), 81.
24 Björk’s Not Get, VR production / release is documented in 2D form, 2017. Youtube
page: www.youtube.com/watch?v=pJDcwXQc5CU.
25 Michael Waugh, “My Laptop Is an Extension of My Memory and Self: Post-Internet
Identity, Virtual Intimacy and Digital Queering in Online Popular Music,” in Popular
Music, 36(2), 2017, 233–251.
26 Daft Punk developed a complex and counterintuitive communication strategy through
their use of helmets as masks. Suzanne Ely’s (2006) interview discussing this is available
online: www.daftpunk-anthology.com/.
27 DeadMau5, deploys a comic-style graphic mouse head mask, discussed in EDM NEWS
“Deadmau5 Helmet: What It Looks Like Inside of the Mask,” 2018, www.edmsauce.
com/2018/03/07/deadmau5-helmet/.
MF Doom is a moniker/alter ego of Daniel Dumile (deceased 2020), who adopted a
Marvel villain–style mask. This was used during his performances and videos. See: Rich-
ard Milner, “The Truth about MF Doom’s Iconic Mask,” Grunge, 2021, www.grunge.
com/305320/the-truth-about-mf-dooms-iconic-mask/. Similarly Swedish electronic
band The Knife (1999–2014) refused to attend awards ceremonies and appeared in
public wearing Venetian plague masks. A YouTube page documenting mask wearing
as part of their performance at Marble House can be found at www.youtube.com/
watch?v=n-6oNlonEyc.
28 DJ 3Lau (Justin Blau) custom-built a website to launch his Ultraviolet album and facilitate
selling his NFTs collection. See: Abram Brown, “Largest NFT Sale Ever Came from a
Business School Dropout Turned Star DJ,” Forbes Online, 2021, www.forbes.com/sites/
abrambrown/2021/03/03/3lau-nft-nonfungible-tokens-justin-blau/.
29 Declan McGlynn, “Aphex Twin Drops Surprise NFT Auction: Sells for $128k,”Djmag.com,
2021, https://djmag.com/features/aphex-twin-drops-surprise-nft-auction-sells-128k.
372 Donna Leishman

30 Guy Julier, Anders Munch, Mads Nygaard Folkmann, Hans Christian Jensen, and Niels
Peter Skou, Design Culture: Objects and Approaches (London: Bloomsbury, 2019), 106.
31 Roger Ekirch, At Day’s Close: Night in Times Past (New York: W.W. Norton & Com-
pany, 2005), 152.
32 Roger Ekirch, At Day’s Close: Night in Times Past (New York: W.W. Norton & Com-
pany, 2005), 152.
33 Oxford Economics Report, “The Projected Economic Impact of Covid-19 on the UK
Creative Industries,” 2020, www.oxfordeconomics.com/recent-releases/The-Projected-
Economic-Impact-of-COVID-19-on-the-UK-Creative-Industries. Also: Oxford Eco-
nomics Report, “Developing Economic Insight into the Creative Available: Industries,”
We Are Creative (previously the Creative Industries Federation), 2020, www.weare
creative.uk/champion/publications/.
34 Henrik Hagtvedt and Vanessa Patrick, “Art Infusion: The Influence of Visual Art on the
Perception and Evaluation of Consumer Products,” Journal of Marketing Research, 45(3),
2008, 381.
35 Spencer Bailey, “Vitra Design Museum Puts Nightclubs in the Spotlight,” Interview
with Jochen Eisenbrand for Surface Magazine, 2018, www.surfacemag.com/articles/
vitra-design-museum-spotlights-nightclubs/ also Jochen Eisenbrand, Catharine Rossi,
Nina Serelus, and Mateo Kries, Night Fever: Designing Club Culture: 1960-Today (Vitra
Design Museum, 2018).
36 Kelly Rae, “Sounds of the Future: How Sónar Grew to Become the World’s Main
Pageant of Digital Music and Artwork,” interview with Eric Pilau for All About EDM,
2019, https://allaboutedm.com/sounds-of-the-future-how-sonar-became-the-worlds-
leading-festival-of-electronic-music-and-art/.
37 Kerstin Mogull in: Evy Cauldwell-French, Eliza Easton, and Caroline Julian, Because the
Night: Why What Happens after Dark Matters to the Creative Industries (London: Creative
Industries Federation, 2017). This report sets out why the night-time economy is impor-
tant for the UK’s creative industries. www.wearecreative.uk/champion/publications/.
38 Evy Cauldwell-French, Eliza Easton, and Caroline Julian, Because the Night: Why What
Happens after Dark Matters to the Creative Industries (London: Creative Industries Federa-
tion, 2017), www.wearecreative.uk/champion/publications/.
39 Kate Levitt, Turning the Tables: Nightlife, DJing, and the Rise of Digital DJ Technologies (PhD
Thesis, University of California, 2015), https://escholarship.org/uc/item/3fs4b8q3.
40 V&A Press statement in support of Night Fever: Designing Club Culture is a UK-exclusive
exhibition at V&A Dundee, from May 1, 2021 to January 9, 2022, www.vam.ac.uk/
dundee/info/night-fever-to-reopen-va-dundee.
41 See the online presence of the brand Armani Privé, www.armanihoteldubai.com/
prive/.
42 ISO is a digital media and software studio who design, direct and build large-scale inter-
active and immersive media projects. See: https://isodesign.co.uk/.
43 Supermafia are a Swiss visual artists collective who create works with innovative digital
materials and processes in order to twist the audience’s perception. See: www.super
mafia.com/ and http://encor.studio/.
44 Pentagram is independently owned and run by 24 partners. The company has offices
in London, New York City, San Francisco, Berlin and Austin, Texas. See: www.penta
gram.com.
45 Marcus Fairs for Dezeen interviewed Adam Smith and Marcus Lyall (Smith & Lyall)
discussing their work for The Chemical Brothers, including the acclaimed 2019
Glastonbury set, 2018, www.dezeen.com/2018/11/23/chemical-brothers-marcus-lyall-
adam-smith-show-designers/.
46 Kin Woo, “Andreas Nilsson: Like A Knife,” Dazed, 2009, www.dazeddigital.com/
music/article/1697/1/andreas-nilsson-like-a-knife.
47 Es Devlin’s personal website documents her projects: Adele’s Live In New York, 2015,
https://esdevlin.com/work/adele-radio-city and Adele’s World Area Tour, 2016, https://
esdevlin.com/work/adele-world-tour, and Beyonce’s Formation World Tour (2016)
https://esdevlin.com/work/beyonce.
 Design and Live Visuals 373

48 Chiara Stephenson’s design direction for Björk’s Cornucopia Tour (2019), discussed
in Augusta Pownall, “Designers of Björk’s Cornucopia Show Ignored the Con-
cert Rule Book,” Dezeen, 2019, www.dezeen.com/2019/12/13/bjork-cornucopia-
chiara-stephenson-set-design/.
49 For a discussion of brand value by osmosis brands see: Don Ritter, “Content Osmosis
and the Political Economy of Social Media,” Without Sin: Freedom and Taboo in Digital
Media, 19(4), 2013, https://journals.gold.ac.uk/index.php/lea/article/view/45.
50 An example of this is the countercultural icon Iggy Pop who actively participated in
Swiftcover insurance adverts (2009+) Campaign documented on YouTube, 2011, www.
youtube.com/watch?v=haAziVvHIu8.
51 Deadmau5 Nokia Lumia Launch London, 2011. YouTube documentation, www.youtube.
com/watch?v=Jd4k8mHt4rM.
52 Eric Spitznagel, “Bon Iver’s Justin Vernon and the National’s Aaron Dessner on Their
‘Anti-Festival’ Eaux Claires,” Billboard, 2017, www.billboard.com/music/features/
bon-iver-justin-vernon-the-national-aaron-dessner-eaux-claires-7751657/.
53 Examples of contemporary performer/artist led festivals: Posty Fest (Post Malone) https://
postyfest.com/; Camp Flog Gnaw (Tyler The Creator), www.campfloggnaw.com/; OVO
Fest (Drake), www.ovofest2021.com.
54 Trade review of the branding project between FKA Twigs and Veuve Clicquot. Lauren
Fads, “FKA Twigs and Veuve Clicquot in Pictures,” The Drinks Business, 2016, www.
thedrinksbusiness.com/2016/10/fka-twigs-and-veuve-clicquot-in-pictures/18/.
55 YouTube documentation of Plastikman’s soundtrack for Prada’s Fall / Winter 2021
Menswear Show. www.youtube.com/watch?v=mYKP3RSBPlQ.
Steff Yoka, “Miuccia Prada & Raf Simons’s Musical Collaborator, Plastikman, Weighs
In On Soundtracking Their First Shows” for Vogue, 2021, www.vogue.co.uk/fashion/
article/prada-plastikman.
56 Double Take Projections own documentation of their project work for Irn-Bru,
including the Forth Road Bridge project, 2015. https://doubletakeprojections.com/
irnbru-guerrilla-projection-marketing.
57 Spencer Bailey, “Vitra Design Museum Puts Nightclubs in the Spotlight,” Surface, 2018,
www.surfacemag.com/articles/vitra-design-museum-spotlights-nightclubs/.
58 Philip Kotler and Joanne Scheff, Standing Room Only: Strategies for Marketing the Perform-
ing Arts (Boston, MA, Harvard Business School Press, 1997), 193.
59 Jaron Lanier Interviewed in: Tim Adams, “Jaron Lanier: ‘The Solution Is to Double Down
on Being Human,” The Guardian, 2017, www.theguardian.com/technology/2017/
nov/12/jaron-lanier-book-dawn-new-everything-interview-virtual-reality – also Jaron
Lanier, You Are Not a Gadget: A Manifesto (York: Vintage, 2011).
PART IV

Interviews With Key

Practitioners
16
INTERVIEW 1: TONY HILL,
EXPANDED CINEMA PIONEER
Steve Gibson

Introduction
This interview with expanded cinema pioneer Tony Hill covers the making of
his early live and performance cinema works such as Floor Film, as well as his
more recent digital films and commercial projects that make use of complex
‘rigs’ of his own construction.

STEVE GIBSON (SG): Your work in the 1970s set much of the ground for expanded
cinema. Can you outline what led you away from film as a playback medium
to film as a performance medium?
TONY HILL (TH): Okay. Well, in fact, I started with the Floor Films, so I wasn’t
really making traditional film before that. I obviously saw a lot, but I
wasn’t making them. I was studying sculpture, and I was put off a bit from
gallery sculpture, and the plinth, and making objects that people would
come and wonder what they were about. I was attracted by the fact that,
with film, people had a specific time that they saw the film, and it was a
social event. I liked that relationship to the audience better at the time. That
got me starting to think about how to use film. And I like the sculptural
aspects of projection.
SG: But the live aspect was one that was not common at that point in time?
TH: Not, not at all, this was 1971 or something like that. I had the idea to make
a film projected on the f loor. And I originally thought actually that I would
project out of a window onto the street, which I never did. But I did then
make these experiments with making films to be projected on the f loor,
and the audience would stand on the screen, which later developed to Floor
Film,1 which had a whole structure for it with a mirror above, so you could
have two audiences. One audience would go inside and stand on the screen
DOI: 10.4324/9781003282396-21
378 Steve Gibson

to watch the film, and the other would be behind the projector, looking in
the mirror. So the people who are watching on the screen become the actors
in a way for the people watching outside, and then they swap around. That
was very much a live situation. It put the audience in a position of being
actors, which I liked very much . . . that way of working with an audience.
That was really my first, earliest kind of thinking about making films: using
film in that way. And to do that, you’re not working with the same kind
of editing ideas. You know, all the grammar has changed, you’re in contact
with the screen. So if you have, in cinema terms, a close up of a mouth,
you’re standing on it, it’s a giant mouth, you know that scale becomes actu-
ally more real. You know it might be a giant mouth on the screen, but we
read it as a close up of a mouth. And time was a very difficult one to deal
with in a way because it depended on the audience. So how long should a
shot run for on a f loor film? That’s a difficult question because it’s real time
now, not film time. Because people are experiencing it in their time. And
of course, it starts to depend a bit on the interaction of other people who are
watching the film as well. So that sort of live element, of what’s going on
and the interaction becomes important.
SG: But then you also use yourself as a performer, which is a slightly different
way of looking at live cinema or expanded cinema.
TH: I did a little for 2nd Floor Film,2 (see Figure 16.1) which was shot from
underneath a glass f loor. So that again was a very different way of me using
film that went a long way away from the normal way of using it. Okay, this
is a medium. How will we use it not telling any stories? For many years I

FIGURE 16.1 Tony Hill, 2nd Floor Film, Super 8 mm, 1972. ©Tony Hill. Used by
permission.
Source: Mail conversation with artist available on request.
 Interview 1: Tony Hill 379

made films with no stories, no music, no dialogue, but working in a kind

of sculptural way to do with perception and how we see the world. For the
2nd Floor Film originally I made a kind of grid to hold a big sheet of glass
that people would then be on top of. Then I back-projected it up onto the
same structure that I’d shot it on. So there’s a sort of one to one relationship
of the screen to the image.
SG: That experience must have been quite a strange one for the audience of the
2nd Floor Film especially. I would think because, as I recall from the docu-
mentation, it almost feels like someone might fall down on you.
TH: Well they’re there, you’re looking down, and there are people on the under-
side. And I came across the idea where the people in the films appear to
notice the people who are watching, which was another sort of element that
I enjoyed, where you can just have this sort of impossible, crazy communi-
cation between the filmic people and the real people who are watching the
film. 2nd Floor Film with the people underneath the glass was about trying
to define the space under the glass, just by using people walking across,
people moving over and under each other, and so on, to give us a feeling of
the space down there.
SG: When was that first shown?
TH: Well, in my degree show, it was a student piece. That would have been
1972.
SG: So that’s just a little after Gene Youngblood’s book [Expanded Cinema]?3
TH: That’s right. I think just to get back to your question, the fact that I
started making films in that way inf luenced how when I went more
towards making cinema films, it had a lot of inf luence on how I used
film later on.
SG: Yeah you can see that in some of your later works, for example, the piece
that you’re pulling on the camera, Holding the Viewer.4 Even though it’s a film
intended to be screened, it seems related to your expanded cinema work in
some way?
TH: Yeah.
SG: In your experience, what are the fundamental differences between tradi-
tional film and expanded or live cinema?
TH: I think in live work, that interacting with the audience, there’s got to be
much more of an awareness, if you like, of the physical presence of the audi-
ence. And then in what you’re calling traditional cinema on a screen over
there, you’re not so much dealing with a physical presence of the audience,
you’re dealing with the mental presence of the audience. You know, it’s all
going on in the back of someone’s head in a way, like the dream screen.
I think that’s one of the differences for me: it is that you are aware of the
physical presence of the audience [in expanded cinema]. Like I was say-
ing with the Floor Film, the scale is created by that physical presence. And
some of my later installations, like The Doors5 piece that I made, is all about
380 Steve Gibson

physical presence entering the space. So that’s one of the fundamental dif-
ferences for me.
SG: And, of course, especially if it’s performed, you have to be there. Whereas
in the traditional cinema you can send out multiple copies to be screened
simultaneously. It [expanded cinema] is a bit trickier in that regard.
TH: Yes. And I was doing a little expanded cinema show with super eight films,
including a ceiling film, film projected onto my chest, and so on. So I had
to be there for all of that. And if you’re doing these projections in different
places, there’s no way you’re going to send that off and say, “oh, you need
to do this, take your shirt off.” So yeah, you’re right. There’s the physical
presence of the filmmaker involved in that [expanded cinema], which I also
enjoyed a lot. Because I did go around: there was a scheme with the Arts
Council called Filmmakers on Tour, which started in maybe 1974–1975.
It was interesting, because they realised that they were funding some of
this work, and I got a bit of funding for the Floor Film. And they wanted to
promote showings of it. They set up this scheme initially with eight film-
makers, including myself and Malcolm Le Grice, William Raban and Derek
Jarman, I think as well. I don’t know if he ever did any. They promoted the
scheme, and then people could book the filmmakers quite cheaply. We got
paid £25 or something and travelling money. And I did a lot of that with
both the live expanded cinema show and the Floor Film, in everything from
schools, colleges, youth clubs, art galleries, lots of different sorts of contexts.
And it was just fantastic as an experience to get so much feedback, to have so
much contact with your audience as a filmmaker, was quite unusual really.
SG: It’s kind of prescient that you did that, because today, I think it’s fair to say
that traditional film is in a bit of a crisis state. There are a lot of directors that
have moved away from traditional film, even Peter Greenaway has moved
to doing live cinema. And so you could say that what you and other people
involved in expanded cinema were doing was setting the ground for people
who are now performing visual artists, not filmmakers who just sent their
work out. You can also see the result today that a lot of traditional filmmak-
ers, maybe not a lot, but a few anyway, have moved into the live arena. And
then there’s a lot of the younger crowd of people, like The Light Surgeons,
or other Live Visual artists, that do all their work live.
TH: And then another one that comes to mind is a friend of mine, Andrew Köt-
ting, who’s a filmmaker, who makes feature-length films, but then quite
often as well as the film there’ll be a performance and an installation of dif-
ferent versions of the work in different kinds of contexts.
SG: You’ve probably answered a bit of this, but we can discuss the audience as
participants, which is interesting for a lot of people involved in more com-
puter-based Live Visuals where audience interaction is an important feature.
Your piece 2nd Floor Film from 1972, and Floor Film from 1975, expanded
your work to include some form of audience involvement – or maybe there
was always the idea to do that – but can you describe what led you to do
 Interview 1: Tony Hill 381

this? And what do you think of the conceptual ramifications of including

audiences as participants?
TH: It was kind of my starting point, making that work.
SG: But trained as a sculptor. I mean, you’re considering audience because of the
3D nature of sculpture, but it’s not the same as including them as a part of
the work.
TH: No, that’s right.
SG: So fundamentally, what do you think are the conceptual ramifications of
doing that? I think there’s been a lot of discussion of this as an issue, par-
ticularly coming from Fluxus and forward. What if you cede some of your
agency as an artist to the audience? What does that mean?
TH: I wouldn’t do that. I don’t go that way. I mean, I’m not wanting the audi-
ence to sort of make the work, or give them too much of the responsibility
for it. I’m making a lot of decisions in the work. And the actual manifesta-
tion of the work doesn’t change, it’s not altered by [being] live, it’s only how
the people respond to it. The audience response to it can be different.
SG: Although in The Doors, there is the possibility of the audience . . . .
TH: Well, they’re kind of more included in it visually. Floor Film it was just fasci-
nating, because I was the only person showing it. So I’d be travelling around
and I saw an awful lot of people experiencing it. And that was very interest-
ing just in itself. Learning how girls would respond differently to boys, and
adults obviously will respond differently to children, and how there’s the
playfulness element encouraged by the work, which I didn’t really realise
until I started showing it. Then I noticed that children would show adults
how to deal with how to watch the film, in the physicality of their response.
An adult response is, “Right, so let’s think about this. I’m thinking about
this before I’m actually going to do anything.” Unless you’re being attacked
by a lion or something, you’re always making a mental response before the
physical one. Whereas children very often, they just get right into it physi-
cally. So that was always interesting in a situation where you have children
and adults together, watching a film where the children sort of inf luence
the adults.
SG: I think you said you did it in schools as well. That must have been a bit odd
to see what you conceived of as a work for a gallery situation then moved
into a very public [venue].
TH: I didn’t think about that possibility when I was making it [Floor Film]. It was
only when I started showing it, it came out. And sometimes schools would
use it as well, they would see it as a kind of tool. And they would get kids
doing prints, drawings, writing poems, as a result of the experience of see-
ing the film. Sometimes they would be drama students who would go on
the film one at a time and do stuff. I have got a sequence with the sea waves
breaking on the shore. I remember once there was a girl in there and she was
rolling as if she was a bit of driftwood. [That was such] a nice thing: that
things happen that you can never predict really (see Figure 16.2).
382 Steve Gibson

FIGURE 16.2 Tony Hill, Floor Film, 1975, from HD video version 2016. ©Tony Hill.
Used by permission.
Source: Mail conversation with artist available on request.

SG: I think that is part of the interesting idea of audience participation, particu-
larly if you kind of let them do what they’re going to do. I know they don’t
have any effect on the actual visual of the film itself, but they do on the
environment, with other people watching it, because there’s also a strange
little performance that’s ongoing, maybe for weeks on end as well.
You’ve continued to make experimental films that are intended to be
screened instead of performed. In many of these features, such as Holding the
Viewer from 1993, there seems to be a notable inspiration from your instal-
lation and expanded cinema work. Can you describe how your live cinema
and installation work has impacted your screen-based films?
TH: I have recognised that as a fact: that I started making these films like The
Floor Films, and that set up a way of working that has inf luenced the [more
screen-based] filmmaking. I think the other thing that I recognised very
early was that film was a fantastic medium for seeing things that you can’t
normally see, and seeing in ways that you can’t normally see. I suppose that I
picked up and then ran with the idea parallel to the audience stuff, although
those films, such as the Floor Films, have that element as well, that you’re not
normally standing on a fire or whatever. And so that was a very strong kind
of thread that went through: to relate it to perception, which I’ve always
been interested in, and orientation, which way up you are, also has an ele-
ment in those other films. Space and orientation and gravity, and how grav-
ity affects how we see. The fact we’re vertically moving around on a more
or less f lat plane really inf luences our vision. Breaking away from that using
film was very interesting. Like the film Downside Up.6
 Interview 1: Tony Hill 383

SG: Holding the Viewer, did strike me as an odd experience too, because you’re
kind of put into the position of the person, and as I recall, they kind of slide
back and forth and you’re given first-person perspective.
TH: It’s like the person in the film is holding all the people in the audience. “I’m
sending you up here. No, I’m dropping you down.” (see Figure 16.3)
SG: Which again, you would never see in a Hollywood film as a camera technique.
TH: No. One other thing that we didn’t dwell on was Gene Youngblood’s
book, which obviously I saw, and I think it was published in 1970. So that
was perfect timing for me. I was reading that, at the same time was starting
to think about projection and film, and then all those possibilities. And that
was one of the inspirations: Stan Vanderbeek, pleasure domes, projections all
over the place, and so on. Although, there was a kind of psychedelic element
to it that didn’t really . . .
SG: . . . didn’t really appeal to you? Because there is the whole other side . . .
TH: . . . the liquid light shows! I used to see those as well. Soft Machine was a
band that used to have really good light shows . . . .
SG: You’re also a well-known as a developer of complex rigs that can be used
to shoot film or video from unusual perspectives. God’s View Rig and the
Horizontal Satellite Crane for me seemed particularly intriguing. Can you

FIGURE 16.3 Tony Hill, Holding the Viewer, 16 mm, 1993. ©Tony Hill. Used by
permission.
Source: Mail conversation with artist available on request.
384 Steve Gibson

describe what led you to develop rigs and how they served a particular goal
in their respective projects?
TH: Actually, apart from those two, they’ve all come from the thinking about
using film to see in a way that you can’t normally see. In Short History of the
Wheel,7 for instance [I wanted to be] able to stop the wheel that’s rolling
along, and have the ground roll around the wheel. Then I figured out a way
to make that work in camera. So the rig was constructed in order to achieve
that, and I was filming on different wheels – I had to build different-sized
wheels to rotate the camera – and different distances, because I wanted to
keep the wheel roughly the same size in the frame each time. So very much
the rig comes out of the idea of the film. In another film To See,8 which is all
filmed in the ref lection of a hemispherical mirror. [In To See] the ref lection
of the camera is in the centre of the frame, although it is very small. The
mirror is fixed to the camera, but in order for me to be out of the frame I had
to devise ways to take it quite a long way away from me while still moving
around with it. If you just hold it [the camera] in front of your eye, then your
head will be in the top of the frame and your feet will be in the bottom. In a
lot of the film, I’ve got the thing looking upwards, so if the ref lection is see-
ing upwards, then you’ve got the sky in the middle, and then the horizon –
the landscape – all around the edge. And if you look down, then you’ve
got the sky all around the edge, and the earth looks like a sort of like a
football. Then I thought it’d be interesting to see how you get from one to
the other? So I built another rig, that rotates the whole camera rig through a
vertical circle so it’s f lipping between the up view and the down view.
SG: It seems interestingly related to kinetic sculpture. Actually, the fact that you
have trained as sculptor beforehand: did that have an impact on [the rigs]?
TH: I wouldn’t say I was a sculptor beforehand. You say trained? At the art
school? Well, training kind of doesn’t come into it. Maybe it’s just the lan-
guage and the wording that you’re using is different to here. But yeah, ele-
ments, obviously. I like making things basically. That’s what I like to do.
For Floor Film, I had to build this whole structure that was portable, that I
could put on my car. That is one of the defining things with all my stuff: we
have got to fit it on my car, and it needs to be able to be erected in one hour
maximum. So yeah, building, making things. And actually, more recently,
two or three times now I think, I’ve had exhibitions in galleries of the rigs,
and shown the films as well, and sometimes the rigs with cameras on them,
so that people can play with the rig.
I can also say something about those other two rigs [God’s View Rig and
the Horizontal Satellite Crane]. I became known a bit for these rigs because
I used them directly for TV commercials, and people hiring them for music
videos. So both of those rigs were sort of commissioned if you like. One was
the God’s View Rig.
SG: Good name.
 Interview 1: Tony Hill 385

TH: It is not even my name, the guy who wanted it called it that. And he said,
“I want to be able to put the camera f loating over this actor’s head, but the
camera is attached to the actor.” I figured out how to do that and I built this
rig, and he made a short film. I have used it more recently on a film called
Bike9 where I’m actually riding my bike. The camera is f loating over my
head [as I cycle along].
If it interests me enough, then obviously I get involved in doing that.
Another rig that I built very early on was Falling Over Slowly.10 Originally,
I was just having someone in the landscape and the camera was tilting with
them. It’s a bit like History of the Wheel: somebody is standing and the landscape
is going around them. But actually more recently, I’ve rejigged that and some-
one used it in a theatrical performance. And the Horizontal Satellite Crane,11
that goes in a big arc (see Figure 16.4). That was commissioned for a TV com-
mercial. It was a serious challenge, really an engineering challenge to make,
to make a rig that moves the camera through 180-degree horizontal circle,
with a radius of seven metres. And it’s got to be able to do that in four seconds.
SG: And does it?
TH: Yeah, it works! And you do not see the crane in the shot.
SG: It’s a bit tricky! Have you used that in any future projects?
TH: I haven’t no, because the company that commissioned it – we were filming
in Sweden – they thought that I might start using it on other jobs, and it
would take away from the power of their [project]. They kept it, they kept
the structure. So it doesn’t exist anymore. It could be rebuilt . . . .

FIGURE 16.4 Tony Hill, Satellite Crane ‘Downside Up’ Rig, built in 1984. ©Tony Hill.
Used by permission.
Source: Mail conversation with artist available on request.
386 Steve Gibson

SG: In the early 2000s you moved away from 35 mm film and into video and
video installation. What prompted this change? I do have to note that many
of your peers from the original expanded cinema movement are still using
film stock. How has computer technology inf luenced the production of
your newer works?
TH: For years I was making 16mm films actually, and then I did make some
35 mm, but the 16 mm [films] were all edited on a Steenbeck12 by reeling
the film and the sound. And you’ve got a shot that is in the middle of that
roll . . . god, I’ve got to reel that through, tape it up . . . .
SG: Kids today, they don’t know anything do they? [Laughs].
TH: It was crazy. And you could never see a dissolve until it was finally printed.
You just had this horrible greasy line cross from one shot to the other. Such
old technology it feels now. . . . Some of the later films were edited on
computers, or maybe I would sort of do a pre-edit. I remember on Laws of
Nature, a film I made, I had a 35 mm Steenbeck, and I could copy the images
onto video because I couldn’t afford to have telecines made of everything.
Then I’d do a U-matic edit at home, which then I could take to some-
where where we could do a full, proper computer edit – in fact, in a facilities
house in London, because I’d been doing some commercials there, and I had
made contacts with people who were doing it [computer editing], and they
were up for just doing it because they liked the work.
SG: But obviously later you started to do that yourself?
TH: Yes, I just found the f luidity of it, the speed of it, the possibilities of it were
just great.
SG: Today it’s now weirdly nostalgic to use film. You see people using 16mm
projectors, even younger artists. I don’t know if you know this group God-
speed You! Black Emperor? They are a performance group from Montreal,
and they don’t use any computer stuff, even in their music. They use guitars,
and cellos and stringed instruments, and they use 16mm projection in their
performances. It could be a little bit of a political statement in a way. There is
a movement to go back to 16mm, and or just analogue technology, whether
it’s sound or video or video or film terminology. Would you . . .?
TH: I wouldn’t go back. Just behind you there I’ve got a 35mm camera, and a
great bunch of lenses. I might try and get the lenses altered so I could use
them on another camera, but I can’t imagine, for a start, getting enough
money to be able to buy the stock and shoot film. It’s kind of gone really,
I think now it has mostly gone. Other filmmakers of my generation that I
know – John Smith, William Raban – they all recognise that now. It was
good while it lasted, but it wasn’t ever really about the film stock itself, it’s
the ideas and what you want to do with it. There’s an element of nostalgia
for lots of people, but actually, I don’t really feel like I want to . . . . I mean
it’d be quite nice to shoot some more film again, but to figure out the light
meter! I forgot how to use it. And the discipline of it, actually I think that’s
one thing that has probably gone a bit. Although I think maybe it’s got better
 Interview 1: Tony Hill 387

again now, but when video was really taking over from film, especially with
younger people . . . I was running a course where we had both, we had
video, we had 16mm film – people that had a Bolex and 100 feet of film: I
said go and make a film, [but] you’ve got to get it right. Get your exposures
right, and get really focused when you do it. And then you had this other
side, where people have video cameras. And they would, in my view, hoover
up: it doesn’t matter if it’s not quite right, we can do it again. Just keep doing
it till we get something a bit vaguely right. So there’s that discipline element
that was inherent in the film thing. It was expensive and hard, and equip-
ment wasn’t that easy to get hold of sometimes.
SG: I think the ease of use of computers actually can be the problem in a way,
because there are too many options.
TH: Did I answer the question properly then?
SG: Maybe not actually, or you haven’t answered the last part of it, which is how
the has the use of computer technology inf luenced the production of your
newer works?
TH: I did make a piece called North Cross,13 which is about a roundabout in
Plymouth. There was a group of sound artists, and they set up a project
where they would pair up a sound artist with a filmmaker, and then each
would choose a particular location or building in Plymouth. They made
about 10 different pieces. Anyway, I chose this roundabout which I walked
through when I went to work. It is quite a nice space, you go through tun-
nels into the space in the middle, and the separation is very good between
pedestrians and traffic. There’s this crazy traffic thing going on and this
nice sort of calm space in the middle. I made this piece which just uses a lot
of very, very simple digital editing techniques to bring that out really, that
whole separation. You have got cars hugely speeded up, or at one point blur-
ring into a thing, and people going in slow motion through the middle of
it. I was playing with some of the possibilities of it [digital video], but with
quite a specific end in mind.
SG: You’re using the fact that a nonlinear editor or a digital video editor could
actually speed things up and slow things down?
TH: Slowing down, and having part of the shot speeded up and part of it slowed
down. And funny little things I discovered. I got a shot with people walking
towards and away from the camera, and I divided it into three sections and
I timeslip them slightly. So people are walking, but their head is moving up
and their body is moving down. You get this odd, almost comical, strange-
ness going on. That’s called North Cross.
SG: Is that the name of the roundabout?
TH: Yeah.
SG: I went under that roundabout yesterday. It was quite unusual actually, I have
to say.
TH: Yeah. So things like that, and now more recent installations like The
Doors. The possibilities are great, but you just need to know what the
388 Steve Gibson

thing is that you’re trying to achieve, and stick to that focus, because you
could easily obviously be distracted away from it. So that discipline is still
necessary.
SG: Great. This is the last of the questions directed specifically to your work.
You’ve also worked as a commercial filmmaker for a wide variety of cli-
ents as well. Can you describe any technological or conceptual crossover
between your experimental work and your commercial work?
TH: Well, if you look at them. . . .
SG: I have to admit, I haven’t looked at the commercials.
TH: Well the commercials are basically inspired by my own work. Mostly they
are because that’s how it works, people making TV commercials, advertis-
ing: the agencies would be sent my reel with clips from films on it, and then
if something came up that seemed to fit in, then I would be contacted as a
possible person to do it. In some cases, when I got the job, I was recognised
as the expert in my own techniques, which meant actually they just let me
do what I wanted to do. You get a very minimal script for a commercial. It’d
never been more than one side of an A4 and a few notes. You don’t really get
very much just as a starting point, but then sometimes I was able to say, well
here’s some other ideas.
SG: And clearly you used your rigs.
TH: And I used my rigs. And it enabled me to build other bigger and better ver-
sions of my rigs, which was good. Also, for me, it was just a fantastic learn-
ing curve, to do work in the industry, which I didn’t know anything about.
Digital production was just starting when I was starting to do that. I knew
nothing about that. So it informed my own practice, not least because I was
teaching. It almost became embarrassing to be teaching in film, video, and
media. I knew a lot about film, nothing about the other side. So obviously
that was really great to learn all of that stuff.
SG: And I guess that one fundamental difference between working on a com-
mercial project like that is that you have a very limited timeframe, whereas
when you’re working on the art projects, unless you’ve got a gallery exhibi-
tion breathing down your neck, you have a little bit more freedom timewise.
So did you work with a crew usually on the commercial projects?
TH: Yes. Always really. You can work with top DPs. You know, they would pick
them out: “I saw a really good film. Who shot that, okay we’ll get them.” It
was also actually a time of pretty big budgets which was lucky as well, trav-
elling and everything. But to get back to the question, as you said, I did use
my own rigs as well in films and pieces, and so there was a lot of crossover.
I mean, some of them if you look at them. . . . If you look at Short History of
the Wheel,14 and then the BMW commercial,15 it starts off looking exactly
the same. Also, because I’d made those one-minute films, Holding the Viewer,
Short History of the Wheel, and they were like the longest commercials at one
minute. So they already had that kind of discipline and compactness that was
attractive: you can get across this idea in this time.
 Interview 1: Tony Hill 389

SG: Anyway, we’re off the track here. I think you answered that question . . .
TH: Well . . . the satellite crane that I built to make Downside Up, I then also used
to make a commercial for NatWest bank. And then it triggered something in
lots of people’s imagination luckily in different countries. I was being asked
to use that technique quite a bit. If I was going to do it again, I wanted to
extend it, find something new and push it further. There was always the
danger that if you got hooked on to one thing you could actually become
bored really. Making the money, but it would be just boring, and the same
thing over and over. Things wouldn’t be very interesting.
I did manage to push it quite a lot further when I did one [a commercial]
for Andalucia. They loved it because you can go into Sierra Nevada and ski,
and then in the afternoon, get down to the beach. You’ve got these differ-
ent sorts of environments. And so we were just going from the beach, to
the snow, to something else, and so on. When you go from the beach to the
snow, there’s someone in a deck chair, and as you go past them, you’re start-
ing to see the snow scene. So, the wipes, as we called them, from one shot
to the other, there would be things crossing over, which was not easy to do.
Which is good. I like it as a challenge, you know. And that’s always been an
element actually of the things I’ve done. Quite often I’d start off on a project
and I’d think “this is impossible, I don’t know if it is going to work,” but
then you just keep at it. Make it work!
SG: Discipline.
TH: Yeah.
SG: Good. This next one is a little tricky because it’s meant really for live visu-
alists, but let me ask it for you anyway. Do you think the medium of Live
Visuals has built in traits that are common to all live visualists, and if so
what are those traits? We’ll broaden it out to include expanded cinema
installation that’s in a live context. Do you think there are any traits that
are similar?
TH: That’s a difficult one.
I start thinking about audience really, and the possibility of audience
interaction somehow, immersive experiences maybe? I guess perhaps that it’s
happening in the moment. So at another time, at another moment it’s going
to be different, or the next day it’s going to be different. I guess that’s an ele-
ment there. Maybe even with installations I’ve done, where you’ve got the
same piece of footage playing, you’ve got different interactions with the way
that people are responding to that every time, and that changes how other
people see it at the same time.
SG: So even when you have a fixed set of materials, because it’s being done in an
installation context it’s reception . . .
TH: . . . people respond differently. And I think the other thing maybe, is that
ideally, you need a group audience. If you just have a one-person audience,
then it’s not the same. It’s the social aspect if you like: we’re all experiencing
this together, and we’re seeing each other experiencing it.
390 Steve Gibson

SG: Maybe the term is spontaneity, or maybe spontaneous improvisation also has
something to do with it. Whether it’s in the broad sense of improvisation,
so the audience, even though they’re not improvising, they’re changing the
way that the work appears. And if you’re a performer you would never do
the work the same way twice.
TH: I haven’t talked about Point Source.16
SG: That’s a performative piece.
TH: It’s a live performance, a visual performance and obviously there is sound
as well. That’s a strange one really, because I’ve never seen it . . . (see Fig-
ure 16.5)
SG: Other than documented?
TH: Well, it’s kind of impossible to document.
SG: I agree, that piece is tricky to document.
TH: Well, I think all good installation should be impossible to document, other-
wise it almost kind of kills itself. I have shown the composite version of The
Doors two or three times. That shows what people will see, but then you’ve
got to imagine that there’s another door there where people actually come
in. I mean it’s just so different. It gives you something, but it doesn’t really
give you what the installation is.
Anyway, so Point Source: I did it in Cork in an old church, I think it was
even a cathedral once, Triskel Centre in Cork. And I do remember doing
it once, and someone saying this is a bit like some kind of religious experi-
ence where I’m the witch doctor, and everyone’s sharing this experience.
It does seem to enhance the shared experience element somehow. It’s kind

FIGURE 16.5 Tony Hill, Point Source, Performance piece, First Performed 1973. ©Tony
Hill. Used by permission.
Source: Mail conversation with artist available on request.
 Interview 1: Tony Hill 391

of an immersive experience. It seems to – more than films I’ve noticed – it

seems to bring people into a kind of common feeling. I don’t know, I can’t
describe it.
SG: My experience of it was very . . . it was so immersive and so simple at the
same time. People still say to me, “that Point Source piece was an amazing
thing,” even though it is such a simple procedure that you’re using. I agree
it’s hard to describe what the experience of it is like live, and I can’t imagine
that documenting it would really cover it. So the liveness aspect there is
crucial.
TH: Yeah.
SG: It really is only to be experienced in the moment.
TH: Yeah. And of course it’s different every time.

Notes
1 Tony Hill, Floor Film Upgrade, orig. 1975, upgrade 2016, https://vimeo.com/198752309.
2 Tony Hill, 2nd Floor Film, 1972, https://vimeo.com/200461458.
3 Gene Youngblood, Expanded Cinema (New York: E.P. Dutton & Co., 1970).
4 Tony Hill, Holding the Viewer, 1993, https://vimeo.com/201314571.
5 Tony Hill, The Doors, 2010, https://vimeo.com/202017419.
6 Tony Hill, Downside Up, 1984, https://vimeo.com/351442846.
7 Tony Hill, ‘History of the Wheel’ Rig, 2010, https://vimeo.com/202537699.
8 Tony Hill, Steve Marshall (sound), To See, 1982, https://vimeo.com/200546351.
9 Tony Hill, Bike, 2013, https://vimeo.com/251786288.
10 Rag & Bone, Co-directed by Benjamin Millepied, Aaron Duffy and Bob Partington,
with cinematography by Darius Khondji, music by Thom Yorke and starring Kate Mara
and Ansel Elgort, with the Falling Over Slowly rig by Tony Hill. Why can’t we get along,
2018, https://vimeo.com/256420417.
11 Tony Hill, Small Horizontal Satellite Crane, 2018, https://vimeo.com/225554302.
12 Steenbeck official website, www.steenbeck.com/.
13 Tony Hill, Gavin Huck (sound), North Cross, 2008, https://vimeo.com/201526031.
14 Tony Hill, A Short History of the Wheel, 1992, https://vimeo.com/185513425.
15 Tony Hill, TV Commercial for BMW, 1996, https://vimeo.com/201844008.
16 Tony Hill, Point Source, Performance Piece, First Performed 1973, https://vimeo.com/
481979872.
17
INTERVIEW 2: CHRISTOPHER
THOMAS ALLEN, FOUNDER
AND DIRECTOR, THE LIGHT
SURGEONS
Steve Gibson

Introduction
The Light Surgeons have operated for over 20 years as a studio creating a variety
of multimedia projects and are one of the international pioneers of live cinema.
This interview with founder and director Christopher Thomas Allen covers
their recent live cinema work SuperEverything* as well as their involvement in
the development of VJing and the practice of creating Live Visuals for various
music acts such as ZERO7, U.N.K.L.E., the Sneaker Pimps and U2 in the 1990s
and early 2000s.

STEVE GIBSON (SG): Could you outline what led you to live cinema and how
The Light Surgeons have developed your distinct identity over the past two
decades?
CHRIS ALLEN (CA): Sure. The Light Surgeons as a collective or a company, began
when I was studying Media Design at Portsmouth University. I had started
experimenting a little bit with projection before going to study when I was
very young, about 18. My brother was a hip-hop, rare groove DJ in London,
in the early 90s acid jazz scene. I was very inf luenced by what was going on
with graphic design and what you might term as retro music back then. I
was also inf luenced by what was happening with hip-hop music and sample
culture in general. Rather than replicating what my brother was doing with
his DJing and music production, I thought it would be interesting to try and
find a way of visualising these things by exploring that kind of sampling and
cut up culture in a visual way.
My dad was a photographer, so I was brought up around a lot of 35 mm
film and slides. He used to produce a lot of his work on slide transparency.
I’d also been exposed to cine film from very young age. My mum had
DOI: 10.4324/9781003282396-22
 Interview 2: Christopher Thomas Allen 393

lost her father when she was quite young. My grandfather was a very early
adopter of technology apparently, he built his own television she told me,
he was an electrical engineer and he’d been a real film buff when she was
a kid. My mum had continued this passion for shooting home movies, and
so as we grew up we had a lot of cine films of us that we would project on
to the walls at home. My interest in moving image really began with that,
just being amazed at how projecting onto things changed their context, and
being able to layer and collage other things together with projection was
really exciting. It seemed to have a real synergy with the music I was listen-
ing to at the time, so when I went to university to study graphic design I was
looking for ways I could involve these ideas in my work.
The Light Surgeons really began as a group or collective of artists that I
met at university who were all interested in exploring experimental work
with projection, mainly myself and an artist called Paul O’Conner. For Paul,
me and a few others it was an outlet for us to make a bit of money on the side
while studying. We ended up doing visual projections for a few local clubs
and music gigs, and that’s where it really started as a name and profession.
It didn’t really become an actual limited company and studio until I
graduated from university. I met another artist called Andy Flywheel who
became my business partner. We began doing a lot of shows together, he was
already doing these projections with Super-8 film projectors so we teamed
up and started to work under The Light Surgeons name. I’d had developed
this collection of graphic slides using back and white reversal lithographic
film and lighting gels which gave us these very screen-printed graphic ele-
ments to composite our Super-8 film projections around (see Figure 17.1).
We quickly outgrew Super-8 and started to shoot and process 16mm film so
we could get a negative of the work we had done.
All of the early stuff we did was very analogue. We didn’t have the money
to get our hands on any video stuff. We were both really into this analogue
aesthetic, we loved the analogue process and happy accidents it brought into
play. I think that’s what people were drawn to in our work. We were creat-
ing these fuzzy analogue collages using multiple slide projectors (see Figure
17.2). We used a lot of sequences of graphic slides which had a super high
contrast. When you projected them, you wouldn’t get any anything other
than the shapes in the image so they sort of f loated in space.
When you’re using lots of projectors, you can composite stuff together
into these immersive, layered compositions. I guess we were doing a form
of analogue compositing without any digital tools or access to video edit-
ing software. We achieved these effects on film by masking things off in
camera. To begin with, we would take samples from our favourite films
from VHS tape back onto 16mm film. This is before we had access to digi-
tal cameras and editing kind of facilities like Final Cut. You could only get
access to non-linear editing in Soho post houses on AVIDs and that cost
way too much money back then.
394 Steve Gibson

FIGURE 17.1 The Light Surgeons, Ninja Tune Idents, 1996. Video by Chris Allen and
Andy Flywheel, Music by Kid Koala and DJ Vadim, www.lightsurgeons.
com/com/ninja-tune-idents/ Used by Permission.
Source: Mail conversation with artist available on request.

We invented our own way to sample by basically re-filming little bits

from the films that we loved on VHS tape that we had recorded or hired.
It was a direct parallel to what people were doing with music. We’d go
and take a tiny bit from a science fiction film that had a beautiful graphics
sequence, like the titles, or a GUI interface, or a moment that we thought
that we could make an interesting loop out of. We’d sample that section of
the film by going frame by frame from the VHS tape back onto 16 mm film
by setting up a camera and tripod in front of an old TV set. We could tweak
the contrast and effectively do a colour grade on it by changing the settings
on television. It was a slow and painful process, but we could reanimate the
sequences in a very precise way using this technique. We could create per-
fect loops, layer them, do multiple exposures all from video back onto 16mm
film. It was a strange reversal of what most people were doing then but the
limitations gave us a lot of creative possibilities.
A lot of our early light shows combined that approach to making
film loops with this very strong graphic design aesthetic. With a lot of
 Interview 2: Christopher Thomas Allen 395

FIGURE 17.2 The Light Surgeons, Andy Flywheel and Christopher Thomas Allen in
operation at The Blue Note club, Hoxton Square, East London, 1996,
Photo by Jamie B. Used by Permission.
Source: Mail conversation with artist available on request.

analogue projectors we were able to cover huge amounts of space, and were
able to create these immersive environments that would totally surround
people in projections. They became entangled in these images and the pro-
jections became architectural, it was really exciting. As things developed
and we made some money, we gradually started to introduce elements of
video into these shows. We started with VHS tapes and using the Panasonic
MX50 vision mixer, and then eventually we managed to scrape enough
money together to invest in our own a digital camera with a Firewire con-
nection. When Final Cut Pro came out for the Mac, that really changed it
all really. Suddenly we were able to film a lot more of our own stuff and the
work took on a more documentary direction. The slides started to become
much more focused on my own photography of actual things, the world
around us, travel, transportation and architecture. I also got very interested
in taking multiple exposures on slide film. I would take a photo and hold it
in my memory until I took another. It was a way of compressing time and
space for me. It was also a way adding more layers to the projected collage,
rather than needing several projectors I could take multiple photographs.
We just got really into this idea of multiple layers and collage to form these
static visual structures that we could then bring to life with the film loops.
SG: Did you abandon sample appropriation at that point?
396 Steve Gibson

CA: It got to a point for me where I felt like there was only so far we could go
with sampling. I just got a lot more excited by taking a more documentary
approach. I mean, why spend so much time rehashing someone else’s work
when you can create your own images? I think it’s fine if you’re doing it
to make a comment on something. For me, if you’re using someone else’s
material, then you’ve got to be saying something with it, there needs to be
a conscious statement in the action in my opinion. It’s way too easy to just
remix or mash stuff together. It can become meaningless. I think I also just
became really interested in documenting the world around me, that excited
me a lot more and it was what led me to recording audio interviews with
people and start making work with a narrative, which ultimately led to us
making more short films and work I would describe as live cinema.
SG: This next question will lead on from that, leading into your live cinema
work. I know you have an interest in in traditional film, but in your expe-
rience, what are the fundamental differences between traditional film and
live cinema?
CA: The obvious fundamental difference is that it’s presented live and can
change or be different each time. There are similarities, but the fundamen-
tal differences are that live cinema is something that’s evolving. There’s a
finality to film, you make a final cut, you go through a process and arrive
at a final point, and there’s something quite nice about that in a way. Part
of what frustrated me about the creative practice of VJing in the early days
of doing shows was that it became this endless remix. You could always
re-appropriate the footage, change it a little bit, rehash it, mix it with some-
thing else and there was no end point. A frustration with that and a lack of
critical feedback is what actually drove me towards making more linear,
narrative-driven film work. This frustration resulted in a series of short
experimental documentary films that were commissioned by onedotzero
festival.1 (see Figure 17.3)
After making these shorts, when we went back to making our own live
shows. I think we were looking to introduce more of a cinematic structure,
we wanted them to have a beginning, middle and an end. We wanted a story
structure but we also wanted to maintain an element of live improvisation
too. Having been inf luenced by jazz music from a very young age, I felt it
was a bit like that with our live cinema work. There’s a structural element,
which might often be the narrative, and then there’s a percussive time signa-
ture for the music which gives the piece a structure. And then around these
structural elements there are more improvised elements, elements that can
move around, elements that change and are different each time.
I think one of the fundamental differences in live cinema performances are
that the work can evolve in real-time, and upon ref lection after each show.
You can change it and add to it over time, it responds to the performer and
reacts to its audiences. When you talk to people after the show, you get a sense
of, did that work? Or did that not work? And you can change things around,
 Interview 2: Christopher Thomas Allen 397

FIGURE 17.3 The Light Surgeons, All Points Between, World Wide Video Festival
Amsterdam, 2001. Commissioned and developed by onedotzero festival.
Photo by Matt Jinx. Used by Permission.
Source: Mail conversation with artist available on request.

so it’s a much more a responsive form of film. It’s an iterative process in

a way, a bit like software in a sense. Cinema can mean so many different
things to different people. Our work is quite structured around interviews
and spoken word narratives which gives it a rooting in actuality and docu-
mentary. So that fixes it in time and gives it a story and structure which we
then perform and improvise along to, trying to illustrate the story and the
musical aspects through our different audio-visual material.
There are obviously lots of other things that live cinema can do that tra-
ditional cinema can’t do. I would say that live cinema is somewhere between
cinema and theatre as it happens in space in front of an audience. We love
this spatial aspect in our work and our shows have always explored this.
Most of our live cinema performances have been developed around this
dual screen setup. This involves a theatrical gauze which covers the front
of the stage and places us inside our material. We use multiple projections,
multiple layers of images to play with multiple themes and layers of mean-
ing, this creates a musicality to the pieces and allows us to improvise how
these elements combine on stage and in space. The material can combine
in unexpected ways, it can move around in that space. This use of multiple
layers of screens allows us to actively play with our shadows and integrate
the moments of the other live musicians. In this space we create something
398 Steve Gibson

that intersects between live music, cinema and performance art. It combines
all of these things together. This collision of different sonic and visual ele-
ments creates these sublime moments, these happy accidents. I love the fact
that you can have some structure, but then you can also kind of also just
detour from that and just let things happen. And maybe because you have
the structure there it has more meaning and significance, but it’s very magi-
cal and what keeps me fascinated in it.
SG: The cinematic aspect of your work really strikes me as quite different than
a lot of other people’s Live Visual work. Let’s give you an example: Carsten
Nicolai, his work is quite abstract and geometrical. And it’s very much driven
by how the sound relates to it, which is true in your work too, but it seems
to me that when you watch one of your longer projects, it does have, like
you said, a sense of a beginning, middle, and end in a kind of a filmic sense.
CA: Maybe. I think part of what drove me to the work that we we’ve been mak-
ing, and it’s not just my work, there’s a lot of people that have collaborated
and worked together on these shows; is a bit of a frustration with abstraction.
It too easy and I think I became a bit jaded by seeing it everywhere. Prob-
ably because I spent a lot of time doing work in lots of different event spaces
and contexts, working with electronic music in particular for many years
and often creating these static visual environments for people, it became a
bit stale and meaningless for me. When I started making live cinema work,
when it became an audio-visual thing, when it wasn’t just us producing a
visual illustration to someone else’s music, we were actually creating the
music, we were creating audio-visual work that was also exploring a narra-
tive as well. I think it was a reaction to abstraction in a way. We wanted to
bring a more human, storytelling element into this cultural space because
it felt to me like it was too easy to just create abstract works. However,
abstraction is a very commercially successful way to approach art, it always
has been. It allows the audience to project their own meaning and that’s
very appealing to people. It’s not that I don’t enjoy this type of work, I think
Carsten Nicolai’s work is amazing and enjoy it very much. I think it’s actu-
ally a very good ref lection of that Raster-Noton 2 kind of sound. It’s a very
minimalist, and purist type of sound and it captures that electronic aesthetic
perfectly.
SG: I probably shouldn’t have used him as an example.
CA: I think he’s a good example. I think there’s lots of other artists’ work you
could look at too, you see it in painting as well. You can look back over the
history of contemporary art and see that lots of artists have made this kind
of abstract expressionism and have become very commercially successful
because people can put it in their homes or in their offices without offend-
ing anyone. Audiences can project their own meaning and see themselves
ref lected back at them. Everyone loves that. It has an openness that maybe
work that’s more narrative or figurative doesn’t, work that’s more rooted in
reality, or that could be seen as realist doesn’t.
 Interview 2: Christopher Thomas Allen 399

I became a lot more motivated by trying to explore how one might rep-
resent reality through this lens of live cinema, and particularly through this
passion for research and interviewing people. I love going out into the world
and gathering the material. I became interested in the history of this prac-
tice, in the ideas around psychogeography, the history of mass observation
and the beginnings of documentary film, in filmmakers like Humphrey
Jennings.3 I love this idea of participating in the world and ref lecting it
back in the work. I felt some responsibility to do this, in having access to
the technology, in having the luxury to own this ability, I really felt there
was an obligation to attempt to tell stories that were relevant to people, that
were asking questions and were ref lecting what was happening socially and
politically at the time we were living through.
That was my choice and the other people that I worked with shared that
idea I think, so it’s ref lected in the work. There are also layers of abstrac-
tion in it too, particularly in graphic material which responds and ref lects
the music more perhaps. Our performance called SuperEverything*, that has
quite an interesting use of layering and there are parts of that show where we
go off into abstraction, into glitchy weird stuff, but it’s got a meaning and a
reason to be there (see Figure 17.4). It’s always rooted in extending the story

FIGURE 17.4 SuperEverything* live cinema performance by The Light Surgeons at the
Asia Society Theatre, Houston, Texas, 2012. SuperEverything* supported
by Arts Council England. ©2013 by The Light Surgeons. Used by
Permission.
Source: Mail conversation with artist available on request.
400 Steve Gibson

that’s being told by the people we have interviewed. I think there is a sort
of Neo-realism within the work that I like, that I find interesting, I think
if you if you are going to be an artist, it’s important to you make work that
you appreciate, and hope that other people will share that.
SG: Your work with The Light Surgeon spans quite a number of diverse pro-
ductions, including live cinema, audio-visual performance, more commer-
cially oriented work and public installation projects. So you’re covering a
lot of ground. Do you consciously use different methods and approaches in
these different spheres? Or do you move technologies, forms and concepts
between them?
CA: We’ve diversified from our roots from working pretty much in a live per-
formance environment, and have moved into doing gallery-based installa-
tion work and exhibitions. I’ve always been a bit suspicious of the art world
myself. I am someone who trained as a graphic designer so maybe this relates
also to your previous question. I’ve always approached my work with a sense
of design and problem solving. I always feel there’s got to be a message, there
has got to be something you’re trying to express or say to the audience. I
think people that come to this type of work, or this area, from a more fine art
background, maybe they don’t have that sensibility so much. I have this deep
feeling that something has to have some tangible meaning or that there’s got
to be a use or utility to it. That it’s not just navel gazing, that it’s not just all
about me or my personal angst. I mean, it’s not I’m against personal angst, that
has its place and can also be meaningful, but I think, for me, I’ve always tried
to approached things in a more pluralistic way and to communicate a message
to an audience and maybe that’s because I come from a design background.
That sensibly has definitely led us as a studio and as a group of collabo-
rating artists, designers and filmmakers. We’ve managed to balance doing
design work and creating our own collective artworks. Sometimes the two
things are very related and inform each other. I’ll give you an example, the
very first live cinema show we did was drawn from material from a design
project that we’d been paid to create. Very often our artistic vocabulary was
generated from the different experiments and creative ideas we had gathered
through the creation of our more commercial design projects. Often it was
after we delivered a project we were able to repurpose it and put it back
together as something else, as an artwork. So there’s always been this kind of
relationship for us between the artwork and a design process.
I think we always bringing an element of design thinking to our projects
subconsciously. Even when we’re making our own artwork, we try to create
a brief for ourselves and trying to think, what’s the best way to express this?
What kind of materials do we want to use? We’ll never jump to a conclusion
like this needs to be video mapping, or this needs to be done like this, or
with this technique. It’s the same when we work with a client, we are always
asking, what are you trying to say? Who are you trying to talk to? What’s the
context? We go through a very methodical design process in order to choose
 Interview 2: Christopher Thomas Allen 401

the right approach to the information, to choose the right approach to the
technology, to choose the right materials. And all these things change with
different situations – if we’re working in a museum context, for instance,
sometimes you’re dealing with physical design decisions that aren’t yours,
that are out of your hands. Then sometimes you can be working in a situa-
tion where you can define your space, you can define the kind of physicality
of the space and define the boundaries your working with more. There’s
been quite a few projects where we’ve ended up working with an exhibition
designer who has already conceptualised and defined a lot of these things,
and we’re in a position where we have to retrofit our ideas in order to make
sense of their design decisions, and it can be frustrating and a challenge, but
it can also throw up some really interesting results because of this.
As a designer and an artist, I always try to see any problems as the solu-
tion. And I know that’s a cliché, but I think it’s proved really true in my
experience, often its these barriers, these limitations or boundaries, these
things you come up against where you’re like, oh how’s this gonna work?
That you really have to fight against to find a way around. It’s in these situa-
tions were creativity kicks in, ideas are created, where art is created, I think
you can often find something special in those limitations. I think limitation
definitely breeds creativity.
SG: There’s a quote from Stravinsky, the composer. He said, when I sit down in
front of the page, I think of what I shouldn’t do.
CA: I’ve actually been stumped by offers to do projects where they’ve not really given
me any limitations. I have found that quite difficult. I always feel there needs to
be some sort of time or budgetary limitation in order for me to conceptualise
it, I can be ridiculously ambitious with projects, probably to my detriment, so if
someone says, “Oh, you can just do whatever you want.” it freaks me out!
SG: Let’s switch over to the music side of things. You’ve obviously worked with
a really wide variety of musicians for your own projects, and for the projects
with The Light Surgeons. Could you describe how you work with musi-
cians in a live context and explain your process of collaboration, maybe
focusing on one or two collaborations?
CA: Are you speaking more specifically with our live cinema work?
SG: It could be your live cinema work, or even the more commercial work
you’ve done with some bands.
CA: We’ve done lots of projects that are audio-visual, from creating our own
short films, which also have a large amount of sound design and music in
them, to our installation projects. Sound is a really important aspect of our
work. It’s 50% of the project most of the time. People don’t think of that
because maybe they associate us with Live Visuals and our work for other
artists, but a lot of our work involves the recording of sound and the com-
posing of music alongside the creation of the visuals. We’ve actually pub-
lished a double vinyl album of the music and documentary narratives in
our True Fictions live cinema performance. It’s a very important thing to
402 Steve Gibson

acknowledge. I’ve always been personally really inf luenced by sound and do
like to play with making music, but I can’t take much credit for that aspect
of our work, the musical compositions and sound design have been mainly
created by artists and composers such as Tim Cowie, Jude Greenaway and
Malcolm Litson along with various collaborators in the US and Malaysia.
A good example of how we integrate the sound and musical produc-
tion into a project would be something like our live cinema project Super-
Everything*. This was initiated through the music department at the British
Council, so that the whole project came out of the desire to initiate a musi-
cal collaboration across cultures. After it was initially created, the project
subsequently got UK funding from the Arts Council, also through music
rather than visual arts. I think that’s quite interesting, because people prob-
ably don’t see us as composers. But that project, which I would say is one
of the best projects we’ve ever made, came from a funding platform which
supported music and the rest of the filmmaking kind of blossomed out of
that initial invitation from the British Council to collaborate in Malaysia.
We were introduced to a range of different artists and musicians during
a research trip in 2010. We chose four sound artists from Kuala Lumpur to
work with. There were two major groups: one was a Chinese drumming
group called Hands Percussion,4 who were quite established and have several
troupes and do these quiet theatrical performances with traditional drums.
There was a group called Rhythm in Bronze5 who are largely a female
gamelan group. And there were also two other solo artists who contrib-
uted, an artist called Flica, who does post-rock electronica music, and then
an artist called Chor Guan Ng, a Chinese-Malaysian composer who really
contributed the most musically and ended up becoming very involved in
the project, touring with it and performing live. He’s a classically trained
pianist who can play the Theremin like no one else I’ve ever heard. He also
became very involved in the writing and composing of a string trio which
we added to the show when we toured it in the UK. Tim Cowie, the other
main composer and musical director of the project, worked together on
developing that and it was really beautiful and is soon to get its own vinyl
release on the record label Utter.
That project also had a very large and ambitious documentary film aspect
to it, the project tells this story about identity, ritual and place through
a series of audio-visual tracks that travel across the landscape of Malaysia,
looking at everything from shopping, religion, and the environment and
how these things effect people. We interviewed lots of different people from
all the main ethnic and cultural backgrounds, including the indigenous
Orang Asli people. As a result, it’s got this very epic, pluralistic story of our
collective humanity.
The music which came out of the project is a combination of Tim’s
interest in electronic synthesis, he’s really into modular synths and elec-
tronic processes, but also contemporary classical compositions. It was
 Interview 2: Christopher Thomas Allen 403

also shaped by an experimental approach to field recording during our pro-

duction trips. We did a lot of recording with this thing called a loop inductor,
which is often used in exhibitions for picking up transmitted sound for the
deaf and hard of hearing, but you can also use them without any transmission
to pick up interference from the environment. When we were filming we
took it with us and we recorded different electrical interferences, this weird
unheard soundscape that you can find listening with these things. We tend to
use a lot of field recording in our projects. When we are filming we’re always
very conscious of capturing sound, we are always gathering different wild
tracks and ambiences that we think will form interesting textures in a show.
SG: Did you work [in the field] with the four artists as well?
CA: Yes. Some of the sound artists came and worked with us when we were
filming. Chor Guan Ng did some of the production when we went and
filmed in shopping centres. He took a zoom out and he recorded stuff most
days I think. For the major musical groups that we worked with, Hands Per-
cussion and Rhythm in Bronze, because of the very limited amount of time
we had with them, and because of their huge instruments, we only had a
couple of days in a black box theatre space to record their contributions. We
essentially did sessions where they performed different improvised arrange-
ments that they wanted to contribute. These were also filmed because we
knew we wanted to include their contributions in the visuals at some point,
as we knew they might not be available to perform live everywhere that
project might go. So there was a series of these experimental recording ses-
sions where they played different arrangements, and we played arrangements
back to them. We also showed them video footage we had gathered and they
responded to it live. It was a really interesting and collaborative process.
These recordings were then re-arranged and sampled to form the structure
of the music in the show by Tim and Guan. We had the opportunity to
invite them to tour with the project as live musicians later on which was
amazing, we took them to perform at the shows we did in Australia and a
big outdoor performance we did in LA in the United States.
With our live cinema work, what we attempt to do is to make both the
music and the film aspects of the projects together. We don’t make a film
and then score it, we’re making the music and the film at the same time.
That’s really ref lected in the work I think, there’s a symbiosis between sound
and image, we strive to make things very audio-visual, and there’s a lot of
cross fertilisation that goes into making the work and then that all feeds into
how it happens live as a performance.
Chor Guan Ng is probably the artist that’s performed with us live on
SuperEverything* the most. He plays live piano and he does this really cool
live Foley sound as well. His presence on stage also got us excited about
creating interesting shadows and using directional lighting to blur the per-
formance with the live video. We really wanted to play with this idea of
a modern take on the ancient art of Wayang Kulit, the traditional shadow
404 Steve Gibson

puppet theatre from Malaysia. This is something I’ve been really inf luenced
by, this idea of the origin of film and cinema, that cinema really originates
from this very live theatrical medium that’s thousands of years old, that goes
back to the telling of the ancient myths of India, and these incredible stories
of the Bhagavad Gita. I’d like to imagine there’s some connection there with
what we’re trying to achieve with film and cinema today, using modern
tools to tell contemporary stories about now with images, sounds and voices.
Guan’s a very central character on stage during the performance, he’s
doing all sorts of mad stuff during the show like chopping up vegetables, we
do a whole rhythmic thing with a knife and a chopping board and contact
mics at one point. He also does other weird foley things throughout the
show that are caught as shadows by the lighting in beautiful ways, like play-
ing bowls of water as percussion by slapping the surface of the water with his
hands and creating these ripples and refractions of light on the screen. The
creation of sound becomes a very real visual element in the piece to the point
that sometimes, we’re thinking how can we create interesting shadows? So
the audio and visual elements are really inf luencing one another all the way
through from production to performance.
SG: But it’s not like using digital signal processing, which a lot of people would
use just to get that kind of rhythmic connection. It’s much more tactile, if
you like.
CA: There is digital signal processing, that’s part of it, but we have both. There
are layers of each going on at the same time. In a physical live sense, Guan
is doing a lot of live, improvisation on stage, and myself and Tim are run-
ning all of this other stuff live, but electronically. We’re using the Ableton
Live software, and we’re using the VDMX software live. We’ve got three
computers that are chained together over a network, we’re using MIDI over
a LAN connection so that when we’re triggering stuff it stays in time to
a central MIDI clock. I’m also triggering video clips using an Akai drum
machine that also has sound on it, and that audio is running through a Pio-
neer DJ mixer that has MIDI on it. If I add a filter from the Pioneer mixer,
I can link that sound effect to a visual effect on the computer, like adding
a blur or something in VDMX. In this way I’m able to link visual filters to
audio filters at the same time. Then I’m free to express myself and impro-
vise by building up and subtracting layers, adding textures and ambiences,
punctuating and manipulating the sound and image together. That’s where
the live audio-visual thing happens for me during the show, and all these
elements come together to form a whole.
SG: What do you consider to be the core conceptual, formal, artistic, techno-
logical, political, or any other values that inform your work?
CA: With the design work, I think it’s responding to the brief, and it’s really con-
sidering what the best response is to the context and the audience. I think
for me, that’s really central for any kind of project I create: what is being
communicated and what is the message, who are you talking to and what
the context is for this is central.
 Interview 2: Christopher Thomas Allen 405

FIGURE 17.5 SuperEverything* performance at Warwick Arts Centre,” 2013/ Photo by

Nik Eagland. SuperEverything* supported by Arts Council England. ©2013
by The Light Surgeons. Used by permission.
Source: Mail conversation with artist available on request.

With our artwork, our films and our live cinema performances, I think
we set out with a goal to transport people somewhere. Where they may
be able to view the world slightly differently and form a new perspec-
tive (see Figure 17.5). I was very inf luenced by quite ambitious and lofty
films like Koyaanisqatsi,6 and films that provide a ‘Mondo’ view on the
world. I love to contemplate the macro and micro, to see the whole and
the details at the same time. I think this is an approach that has really
inf luenced me and I’ve been trying to re-create that with my work: to
elevate people to contemplate the world around them in a slightly dif-
ferent way to before that experienced the work hopefully. I hope the
result is not just entertaining, but it asks make people think a little and
contemplate too.
SG: There is a bit of an agenda, if not quite in a didactic way?
CA: I like to ponder things. I like to consider the world. I’m interested in the
world around me, and I’m interested in people, and I’m interested in politics,
and I’m interested in why we exist, I’m interested in theology, I’m interested
in deep questions about who we are as human beings, and I like to provoke
people to think that about these ideas too.
SG: So it comes back to the question about meaning?
CA: I think art has a very important a role to play in society, it’s about ref lect-
ing on ourselves, about understanding who we are, about imagining where
406 Steve Gibson

we might want to be and where we might go in the future. I think art has
got a central role to play and we are very fortunate to be able to create the
work we make. Everything in this room has been imagined, every single
object, everything that you see that is man-made, has been imagined, so
that means it has come from someone’s brain, it has been realised and turned
into an artefact or object. That’s quite a profound thing to consider I think,
to take on board that responsibility of imagining, and that’s why I think I
want to try to make something positive, to provoke people. I think it would
be a shame not to do that. I just like to see work that attempts ref lect the
wonder of the world, so I try to make work that provokes a sense of wonder.
I think the more I do that in my daily life, the better my daily life is. The
more time I spend not sweating the small stuff, the better I can cope with
all the uncertainly and stress of life. The more I think about the world in
a more abstract sense, like the fact we are all just very small organic things
on a crust of a ball of fire that’s f loating through space, the more I’m able
to think of myself within that broader idea of the universe. The more I can
relate to this wider view for the world and my place in it, the less stressed I
am and the more open I am. I don’t think that these ideas and thoughts are
necessarily didactic, there is just a passion for trying to engage people in a
sense of wonder, to transport them somewhere with the work we make. If
people are going to give you their attention for 45 minutes or an hour in a
performance, it’s nice to think, okay, they’re going to come out of it with
something, rather than make it an endurance for them.

Notes
1 Onedotzero Official Website, www.onedotzero.com/.
2 Raster official website, https://raster-media.net/.
3 David Parkinson, “Where to Begin with Humphrey Jennings,” BFI Official website,
www.bfi.org.uk/features/where-begin-humphrey-jennings.
4 Hands Percussion Official website, www.hands.com.my/.
5 Rhythm in Bronze Official website, www.rhythminbronze.com/.
6 Godfrey Reggio, The Qatsi Trilogy, Official Website, 1972–2021, www.koyaanisqatsi.org/.
18
INTERVIEW 3: GREG HERMANOVIC,
CEO, DERIVATIVE
Steve Gibson

Introduction
Greg Hermanovic is the founder and leader of Derivative, makers of the Live
Visuals software TouchDesigner. Greg is also well known as the co-creator of the
3D software Houdini. In this interview Greg discusses Derivative’s work with
high-profile artists and producers. He also discusses his own work as a VJ and
collaborator with various experimental music and film artists.

STEVE GIBSON (SG): You’re well known for being one of the co-founders of Side
Effects,1 which is a company that makes Houdini. Can you describe how
you made the transition from being a developer of animation software to a
developer of real-time visual systems, and a creator and performer of Live
Visuals?
GREG HERMANOVIC (GH): I think it’s a long continuum. I was always interested
in diverse performed and improvised music. I grew up in an era seeing Jimi
Hendrix and early Led Zeppelin live and the full-on electrification of music,
and moved into the hybrid electronic jazz era of Miles Davis and Weather
Report, and then into more experimental, avant-garde stuff, and ethnic
music like Balinese gamelan and shadow plays. I had seen Sun Ra and his
Solar Orchestra a couple of times, including his show with the Joshua Light
Show at the Newport Jazz Festival in 1969, which really set me off. So
I’ve always been really interested in the aspects of music that translate well
into visuals and graphics, especially the use of digital computers for making
images complementing music, and the free-f low of improvisational music,
which morphed for me into VJing.
I was also really interested in early experimental films, inf luenced a lot by
the Expanded Cinema book by Gene Youngblood. I’d seen a lot of the John
DOI: 10.4324/9781003282396-23
408 Steve Gibson

Whitney, Norman McLaren, Bill Viola, Stan Brakhage and Michael Snow
experimental animation films. At one point I sketched out an imaginary
computer system design that would replicate McLaren’s Synchromy2 as I was
intrigued by its very mechanical translation of sound to picture to sound.
Then I had the chance to work in aerospace on the US Space Shuttle
and learned a lot about engineering and computer graphics. There was a
real-time computer graphics simulator for training astronauts on the shuttle
project, and I was a member of the team of programmers on that. Then I
wanted to take that knowledge of computer graphics, and along with music
and improvisation, apply it to visuals at some point. So I joined a company
in Toronto called Human Computing Resources, where I learned UNIX
programming and later ported the Cmusic computer language,3 which came
from University of California in San Diego, to run on a PDP-11 mini-
computer. There I was exposed to Cmusic’s paradigm of the node-based
procedural pipeline. Although Cmusic is text based, it had that idea of nodes
feeding other nodes with audio samples and control signals. Before then I
was exposed to analogue music synthesisers, like the ARP 2600, the Synthi
AKS, and the Moog. Those are all modular synthesizers, where you plug the
output of one module to the input of another module and, especially with
the Synthi, you tend to generate wild feedback loops.
That audio experience and that interest in visuals led to leading a group
building a computer graphics system at Omnibus Computer Graphics4 called
PRISMS (see Figure 18.1). But when Omnibus started acquiring its largest
competitors, it imploded. One of my co-workers at Omnibus, Kim David-
son and I started Side Effects, and we bought PRISMS source code from the
bankruptcy of Omnibus. That was the beginning point for our product line
of Side Effects. There was a healthy diversity of animation systems around
1990 – Wavefront, Alias Power Animator, SoftImage, TDI, Vertigo from
Vancouver and the beginnings of 3D Studio. But none had a procedural
node-based workf low.
When we founded Side Effects with PRISMS as our starting point, we
engineered into it what we call procedural modelling, or Surface Opera-
tors, which paralleled the patchable modules of analogue modular synthesis-
ers. PRISMS can take a geometry in one ‘operator,’ like a curve of points,
pass it to another operator which puts a circle at each point, then pass that
to another operator that skins them together. The resulting output was a
3D shape. So all these were done using stages of procedural nodes, which
were very analogous to patching modules together in a music synthesiser.
The great thing was, as it is with synthesizers, if you twiddle a knob on an
oscillator it affects things all the way down the chain immediately. So the
procedural modelling idea we designed into PRISMS led to our first Acad-
emy Award. And we engineered procedural 2D image compositing into our
next product, Houdini. Then this whole procedural thing kept on evolving
once I spun off from Side Effects to start Derivative to make TouchDesigner.
 Interview 3: Greg Hermanovic 409

FIGURE 18.1 Kim Davidson, Side Effects Software, PRISMS software interface,
1987, ©Side Effects Software. Used by permission.
Source: Mail conversation with artist available on request.

So yes, my interests and career have followed a music/graphics/art/real-

time continuum. My work in animation follows along with some of the
ideas of music. For example, I was subscribing to Computer Music Journal,
looking at how they were designing software instruments and interfaces for
digital control of audio, with analysis and synthesis. It had an inf luence on
my ideas of visual software. And then there was the inf luence from MIDI
devices and Open Sound Control (OSC) and how systems interconnect. In
a way, the visual animation world was kind of following the audio develop-
ments that occurred a few years before, with a twist of course, because visu-
als are much harder to construct than audio. Audio is simply a voltage or a
channel of sound samples, but imagery is more complex – there’s polygons,
lights, 3D modelling and motion, texturing, occlusion, shadows, and all that
kind of stuff. So it’s quite a complex thing we are trying to describe in the
procedural world.
SG: Just as a bit of an aside about the output though. What you’re generating
from working in Houdini was 3D animation, whereas the output when
you’re working with TouchDesigner is quite different. Well, maybe not just
410 Steve Gibson

the output, but the entire ecosystem of TouchDesigner is different (see Fig-
ure 18.2 for an example of a TouchDesigner interface).
GH: Yes, that’s right. The output of PRISMS, Houdini and the animation tools
are almost always non-real-time generated sequences of images. And the
output of the real-time tools, of course, is video, audio and data, all pro-
duced in a 30th or 60th of a second. And it needs to be fully controllable
live. As a result, you can improvise with it a lot more because it’s all at your
fingertips. That opens up a whole other domain of input devices and touch-
screens that really don’t apply as well to non-real-time rendered animation.
In VJing I was interested in improvisation, and just kind of winging it,
but also with some structure. It’s like a jazz musician who is playing a struc-
tured theme with the other people in a group, it is evolving on its own, and
then they go off on tangents, and come back to the themes. They do that
very naturally and f luidly. But when it comes to making digital images, that
was new territory. So that’s why I got interested in doing what’s now called
VJing: I was making visuals at first for electronic music parties. Electronic
music is easy to work with because it’s somewhat repetitive yet has a lot of
variation to it. And you can follow a groove visually, while the music’s doing
its own kind of variation. In VJing I feel like I’m another instrument that
you don’t hear, that you just see. That’s what I think of when doing visuals,
I consider myself the extra guy in the group doing an extra layer visually.
Yeah, doing something along with the music, but a bit different than it.
Alas however, most of the time I spend developing tools for performing
visuals versus performing with them. Well, I do perform with them enough

FIGURE 18.2 Keith Lostracco, TouchDesigner Luminosity media server interface,

2015. ©Keith Lostracco. Used by permission.
Source: Mail conversation with artist available on request.
 Interview 3: Greg Hermanovic 411

to continually redesign them but unfortunately I spend 5% performing and

95% designing and building the software!
SG: What techniques, strategies and forums have you brought from the rendered
3D animation world to the live visual world?
GH: Things are different in performing live. For instance, in the 3D animation
realm you’re usually keyframing motion or doing things very simply by
linearly interpolating between A and B. But in order to get more articula-
tion, there emerged forms like motion capture – body motion capture, facial
motion capture or capture of simply a mouse movement. I wanted to bring
a bit of that into Live Visuals too. So I put in some techniques for captur-
ing gestures, whether they’re from a mouse, touchscreen, or a joystick, and
taking captured gestures and playing around with the way they’re blended
and looped, trying to follow rhythmic music in different ways. Blending
between different pre-made moves, poses and presets gives opportunity for
live-performed variants.
You may be just moving around a bunch of physical MIDI sliders at one
time, controlling five different parameters using your fingers on the con-
trollers: a colour, light position, a speed, a camera, etc., but you may not
even know what you’re doing on an individual controller basis. Yet you end
up with something interesting looking, and unconsciously finding a rhythm
from the motion of the controllers. So those things are interesting in per-
forming visuals. That may be parallel to musical performance – most of the
time, musicians don’t really think about what notes they’re playing: they are
aware of the result, they may not think of how they’re achieving it at the
time. So that was really important to me: to try to build visual software tools
that allowed performers to perform that way. To improvise, without think-
ing, and still be able to guide within some structure.
SG: It’s like an intuitive performance in a way.
GH: Yes, mixing intuitively with some kind of structure. Then what emerges
is that you may want to put more structure in, so you want to have presets
and jump-to points, and then your tool gets more complicated because you
have all these presets to manage: then how do you get from one preset to
the other and still improvise on top? So that became my focus of my inter-
est since we started Derivative. But everybody builds different tools with
TouchDesigner, like there’s probably 10 or 20 different video mixers out
there. There’s a bunch of live tools that are just designed for one show, or
one event or for one song. It’s quite a variety. The intent for us is not to come
up with the one way of doing it but just opening the door to providing the
building blocks so users can piece things together and make their own mini-
applications that suit them.
SG: The now legendary Interactive Dance Club5 at SIGGRAPH 1998 would
have to be considered one of the ‘NASA moments’ in the development
of computer-based Live Visual culture (see Figure 18.3). I say ‘computer-
based’ Live Visual culture because obviously there was a Live Visual
412 Steve Gibson

FIGURE 18.3 Ryan Ulyate, Dave Bianciardi and Greg Hermanovic, Interactive Dance
Club at SIGGRAPH 1998, Photo by Ryan Ulyate. ©Ryan Ulyate,
Used by permission.
Source: Mail conversation with artist available on request.

culture before that. Could you describe your involvement in the event and
outline your thoughts on the inf luence it had on future developments in
Live Visuals?
GH: Interactive Dance Club was conceived by Ryan Ulyate and Dave Bianciardi.
Ryan’s a musician and remixer. Dave’s a sound and control systems tech-
nologist. They pitched the idea to SIGGRAPH of a four-night dance club
with lots of music-synced interactive devices and visuals. They got a sub-
stantial budget to put Interactive Dance Club on during SIGGRAPH 1998 in
Orlando. SIGGRAPH had no idea what they were getting. I was invited in
to help design and produce the visual component of it, since they wanted to
have interactivity and generate controlled sound and visuals. I immediately
got a loan of six Octane machines from Silicon Graphics (about $40,000 US
each!), and those were feeding six large projection screens, in six ‘zones.’
We then got together with a bunch of Houdini artists to produce 24 differ-
ent real-time visual pieces, four for each screen. There were four musical
pieces: a ‘Tiki’ one, a drum and bass one, a funky one and an ambient one.
With the Houdini artists we designed visuals for the six zones. And each
zone had some special input devices: one was a set of eight stomp pads that
you could jump on, another had a pair of car tires you could turn to get
0–1 angle values, another had two hand proximity detectors, one had a big
spherical orb with 18 pods that you stuffed your hands into to trigger things
 Interview 3: Greg Hermanovic 413

up on its projector screen. Every area zone had a unique input device, a
screen, four different visuals, and music or sound samples specially com-
posed for that zone.
SG: So those were going on simultaneously?
GH: Yeah, so every song was about 20 minutes and had a multi-track music base
to it, which kept the rhythm and song-structure going. Plus every zone had
a whole bunch of sound effects and loops that were triggered or altered by
different actions in the zone. The input devices, being played by the ‘club-
bers,’ directly affected stuff on screen and the zone’s sound. All the graphics
were real-time generated, sometimes 2D but mostly 3D, and they of course
had connection to the zone’s input device. All the sounds were mixed by
Ryan at the mixing board with feeds coming from the audio generator in
each zone. We knew that there would have been a real cacophony of sound
if there wasn’t a foundation to the rhythm and the bass tracks. They designed
the songs quite interestingly: they all had an intro section and then a build-
up, and then a chorus, and other sections of the song, and Ryan could stay
in a section as long as he wanted and hop between sections (Ableton wasn’t
born yet). It was all structured so it could stretch from 15 minutes to 20 min-
utes a song. The participants would get into the zone and take turns playing
with the input devices.
We followed a 10 Commandments of Interactive Experiences that our
design group came up with. Some of them are: “Input devices, visuals and
sound shall encourage movement. Actions and responses shall be repeat-
able. Immediate and positive response shall be given to all actions. No prior
expertise shall be needed to play. No instructions shall be needed. No think-
ing shall be allowed.” Stuff like that. So with these 10 commandments the
team of Houdini artists set forth in designing and building, troubleshooting
and tuning all these twenty-four visuals to the four songs in the six zones.
I think it was nicely structured. And I think that’s what proved to be its
success: there was enough structure and linearity to it that it was comfort-
able, but there was enough ability to go off on tangents and improvise by
the participants and by Ryan in the mixing. There was lots of visual variety
throughout each song and it was hugely amusing.
SG: That’s a good description of it.
GH: So what was the question again? [Laughs].
SG: I think you’ve answered the question about your involvement in the event,
but do you think it had an inf luence on future developments, either on your
own or amongst other artists?
GH: The 24 pieces that the Houdini artists made as a result of that are compiled
together with other visual synths into a CD called TouchArt 017 along with
40 newer visual synths which came out in 2004 with TouchDesigner 017,
and was subsequently placed on the Derivative web site (search TouchArt
017). All the pieces were converted into a form where you can start it up
on a PC and play it – with equivalent controls of the original art pieces. We
414 Steve Gibson

learned how important it is to tune, test, observe, tune, test, observe for each
specific situation that an interactive piece exists in.
In fact, during the Interactive Dance Club, we were tuning things every
night based on what we observed that day, re-engineering things, simpli-
fying things, getting timing down, making the art look better and fixing
bugs. The whole team was actually developing night to night to night to
make the experience better.
SG: So it inf luenced the development of your software?
GH: Yes, that’s right. For one, what came out of Interactive Dance Club in terms of
software product was the development of a new subsystem which we called
channel operators – CHOPs, which makes controlling things in real-time a
lot easier, similar to control signals in modular audio synths. And then once
we spun off Derivative to make TouchDesigner, a real-time compositing
subsystem also emerged because real-time GPU compositing became fea-
sible with computer graphics cards in the late 1990s and early 2000s. The
speeds became so fast that you could actually do a lot of image processing
on the graphics card.
SG: Your own work as a live visualist has a kind of painterly quality and a
visual richness that I would say is probably lacking in some other in some
other Live Visual work. Can you describe the inf luence of painting on
your work?
GH: First, the art that I enjoy the most visually and sound-wise has a lot of tem-
poral scale to it. I first VJed at numerous raves and electronic music events
in Toronto, like Chemistry, Fukhouse, wabi, Tribe, Simplicity, Industry,
Turbo, Area 51 and more. I formed a collective called El Kabong where
we did tag-team-style jamming typically on two computers plus tape and
a mixer. In VJing with electronic music, what I mean by temporal scale is
that some layers of music may be very slow moving, and kind of meander-
ing, some would be more rhythmic, some would be very syncopated, and so
on. I like hearing all those aspects of the music at one time, dissecting it in
my head, and then piecing it together. I like to make imagery that mirrors
or counterpoints that. To see imagery that’s always jarring, f lashing every
frame, that kind of stuff – that’s not enough. I like seeing combinations of
layers that may be slowed down or frozen, blurred out or slowly smearing
along with rhythmic or syncopated stuff on top of it, with regular/irregular
patterns, mixed, timed and tweaked in varying proportions. I construct and
adapt visual synths that can be performed with simple controls, like the 16
sliders and 16 buttons of a Peavey 1600 MIDI controller. This way of gen-
erating and mixing matches the structure of many sub-genres of electronic
music, for example drum and bass which is very fast and variable, but it has
a lot of underlying constancy to it. Techno is very much like that too, with
long builds and so on. That’s the kind of music I like, and that’s the kind of
visuals I like to produce (see Figure 18.4 for an image of Greg Hermanovic
live VJing).
 Interview 3: Greg Hermanovic 415

FIGURE 18.4 Greg Hermanovic performing with TouchDesigner, with Tom Kuo @
Project, Toronto for Kuo+Marshall, 2001, Photo by Tom Kuo. ©Tom
Kuo, Used by permission.
Source: Mail conversation with artist available on request.

SG: Just like electronic music will often slowly develop, and may, as you said,
have a lot of f lutter on the top, but as something constant maybe it has a
sustained note or something like that?
GH: Yes, even if it’s a long, sustained, drifting tone. This led to my building
a video mixer with the features that give you that kind of control. I use
blurring/softening a lot to give some layers a rest: mixing very blurred or
smeared so they are indistinguishable, with some layers that are very sharp,
being able to blur-unblur them with plenty of speed-control.
SG: It’s kind of similar to foreground and background in painting in a way. A
lot of painters will do exactly what you’re describing, particularly if they’re
working in a more abstract medium.
GH: So foregrounding and backgrounding meaning that they emphasise or
diminish some of the elements? Yeah, that’s true. Simple things like taking
colour out or pushing colour using chroma modification techniques that
you find in video compositors.
SG: Your Live Visuals software TouchDesigner has been used in a wide variety of
contexts and also by some pretty high-profile artists and producers – Plastikman,
416 Steve Gibson

Rush, Disney. Could you refer to one or two of these collaborations and
outline how TouchDesigner was used?
GH: In the case of Plastikman, I’d met Richie Hawtin [aka Plastikman] a couple
of times at different performances of his and kept in touch. Richie and his
visual designer Ali Demirel came to the Derivative studio one day with a
copy of Norman McLaren’s Synchromy. I was astonished as it was one of my
favourite films.
SG: Right. Interesting [laughs].
GH: He said, “Can you do something like this? I am thinking of doing a show in
the next couple of months.” [Laughs].
Richie was one of the first musicians I talked to that really understood
how visuals could be locked into music in a way that a performer can con-
trol both at the same time. Being a one man show, he doesn’t want to have
a VJ sitting there, he wants to just do it all himself. You know, control the
musical and visual elements. At that time Ableton Live had just come out,
and so we hooked TouchDesigner in with Ableton. We built a couple of art
pieces that were part of Mutek in 2004, and some of them look pretty amaz-
ing, some of them were really connected: like during the show, and during
rehearsals, there were times where he was tweaking the sounds, and the
connected visual behaviour was an artefact of the sound. But then he oper-
ated the opposite way – he would look at the picture and say, “I want the
spheres to pop out a little bit more this way.” And to do that he was playing
with the MIDI controls, which simultaneously had a clear corresponding
effect on the image and sound. That behaviour where “I’m controlling the
visual and as a side effect I’m controlling the sound,” – I hadn’t experienced
that with anybody at that point. Although I hadn’t expected it, Richie really
got it and it was pleasing to see one aesthetic applied in both visual and
sound at the same time.
We then hooked up with him about five years later for the Plastikman
2010 tour (see Figure 18.5) which involved more songs, more design, and
more structure. Jarrett Smith, my partner at Derivative, worked with him
closely on all the tracks along with Markus Heckmann and others, engi-
neering the visuals for each song: like what instrument is connected to what
visual, and what the relationship of the visual is to the sound elements, and
how you transition from one song to the next, what things are dropped
out, how you pull things in and then how we leave parts of the songs open
so Rich can improvise on things on a very simple level. Even a POP sound
connected to a dot appearing and animating is very effective if they are
synced and controllable, becoming easily ‘graspable’ by the audience.
To maximize the data we had to play with in TouchDesigner, we streamed
instrument-separated pre-mixed sound channels into TouchDesigner. We
pulled in tempo, loop triggers and MIDI drum events. Some visual elements
were connected to sliders or drum pads. Each song was a combination of
numerous mappings of generative visuals driven by automated or manual
control.
 Interview 3: Greg Hermanovic 417

FIGURE 18.5 Richie Hawtin aka Plastikman live at Time Warp in Mannheim,
Germany 2010. Photo by Stefan Solf, ©Richie Hawtin and Stefan Solf.
Used by permission.
Source: Mail conversation with artist available on request.

SG: It sounds like it was like an interesting combination of the very formal pro-
cess of building things for the songs, which have particular visual elements,
but then allowing control of those elements to be spontaneous.
GH: Yes, that’s right. Some of these songs could go completely without him
touching anything, but he could and does intervene at any time, at any
level. Some songs needed a lot of control by him on the visual side. Most
challenging are the long transitions from one song to another – with visu-
als it’s not simply fading things out and in – there are a lot more param-
eters at your disposal – lighting, zoom, speed, texture, effects, and one layer
affecting another layer. The visual elements act as intertwined layers in the
experience.
The collaboration was organic: we all had design ideas and we absorbed
them into the tracks in certain ways, and then they mutated a lot from the
first show to the last show. We were always trying to make them better, react
to Richie’s perpetual musical refinements and work with the stage lighting
as well.
SG: Your HD video mixing tool Mixxa has been used in a feature film by Peter
Mettler, The End of Time.6 Working with Live Visual software for a ren-
dered film intended for theatres rather than performance seems to me to be
a pretty different experience than preparing material for a live music event.
Could you describe how this collaboration came about and how you use
Mixxa to generate the films used in the sequence?
418 Steve Gibson

GH: Peter had done live video mixing quite a bit before I met him, and we
were mixing side by side at a couple of electronic music events. He was
using video cassettes, mostly Hi8 and VHS, and a Panasonic MX50 mixer
and sometimes Isadora. I was using an MX50 video mixer as well, but I
was feeding it digitally-generated output from our software. I made some
TouchDesigner synths specifically for some events, and a VHS and Betacam
tape machine got in the mix as well, sending NTSC video to projectors.
At that time I became interested in making a video mixer whose inter-
face was a touchscreen, leading to the first inception of Mixxa in 2007. It
had three video channels and did a couple of groovy things, and I was quite
happy with it. I got numerous people to perform with it – watching them
gave me good feedback, but then Peter started to use Mixxa. This changed
my thinking about how to arrange the user interface of a video mixer – to
be similar to the way a person would live-mix audio, with each of the audio
sources passing through a couple of filters, a volume control at the bottom,
and a couple of effects, Aux sends and receives, and some patching that
mixes the channel to the master output.
It mimicked his analogue video and audio mixers. Then we expanded
Mixxa to be a 9 × 9 matrix of cells, nine video channels wide and nine
effects high. Each cell takes video in, applies effects/mixes using the UI
controls or MIDI, and outputs video, Some cells of the matrix combine pairs
of cells into one, and some process the previous cell such as blur. Columns
are composited together in pairs until you have one master mix. I followed
the paradigm of the audio mixer design style into Mixxa to see how far I
could go while keeping it functional, graphics-focused and operable with a
touchscreen.
I followed some design principles for Mixxa. For instance: all the ele-
ments, effects and features should be accessible with one, maximum two
mouse clicks or finger presses in order to keep the interface as f lat as possible;
no things pop up in front of other things unless you’re doing maintenance;
the menus are integrated in a way that they are simple to get at with your
fingers; the graphic elements of the interface are as simple and mono-
chrome as possible because all your video elements are where the colour
should be. This video mixer became a huge personal project – I kept adding
more and more features, driven by our joint experience with it. Then I add
more complexity – presets, each comprised of all cell parameter values as
well as the arbitrary patching between them. Then we faced the challenge of
how you get from one preset to another without being jarring. So I imple-
mented a way of going from one preset to another step by step, controlled
by one slider only. Then we thought of a semi-structured show as a bunch
of presets that you arrive at and improvise from.
Peter had an interesting approach to mixing because he is very much an
improviser of visuals, but he also wants to get back certain prior looks, or
combinations of movie files that he had created in the past, so I implemented
 Interview 3: Greg Hermanovic 419

a history list so he could easily trace his way back. He quickly surpassed
200 bins of movies with about 15 to 40 movies in each of them. The size
of it got absolutely huge and it became impossible to run on the computers
that we had at the time. I was always trying to confine his system to run
on a portable mini-Shuttle PC and I kept him down to that, but then he
kept on blowing up memory and blowing up graphics cards. A year or two
later he bought a graphics card that was twice as fast as before. So for a short
moment Peter was happy again and I was happy, until he added more. Then
we tried to get a good frame rate – 30 frames per second plus with a lot of
compositing. It’s a very graphics-heavy tool – to actually have the whole
graphically-complex editing and the compositing underneath with multi-
outputs to your audience all in one tool is daunting. But Mixxa works.
Fortunately I work with our programmers here at Derivative, and Mixxa
is a great test case to optimize the speed of TouchDesigner’s video playback
and compositing.
I’m always designing small prototypes and experimental tools, and then
determining, can TouchDesigner do this? For me, I’m lazy – I don’t want
to program anything, though I don’t mind some Python scripting. I want
to be able to build everything by just connecting nodes together and adjust-
ing parameters, with a minimum of typing. Within those constraints, our
team at Derivative determines what’s lacking in TouchDesigner. Then our
developers get involved in providing new operators or features, or optimiz-
ing the speed of TouchDesigner to reach certain performance goals. It’s a lot
of back and forth between Peter as a user of a tool and myself and our dev
team. He conceives of capabilities he wants, how he wants to organise and
perform things, and I am in the middle with my own design ideas – then our
programmers help to realise something practical.
SG: It’s an interesting concept. I know he is a live visualist, but he’s also a film-
maker. So it’s interesting to use what is a Live Visual tool, and then to turn
it into something that goes into a rendered film.
GH: He had shot lots of video footage for The End of Time. And he knew that he
was going to mix some of it live, and then record and insert it into the film.
He chose about 100 clips that he wanted to mix live. So we converted them
into a slightly lower resolution for Mixxa to get a good playback frame rate.
He practised a bunch of mixes and ideas at some live events informally, and
then it came down to some combinations that he wanted to live mix for
the movie. So basically he was still performing live, but this time it was
in the studio, working the controls, going from phase to phase, trying to get
some timing right while performing along some testbed electronic music he
chose for the film. We recorded numerous takes, picked the best ones and
rendered them out at full resolution. Those became ProRes clips that ended
up in the film. Like I said, this is at the border of what was feasible on a
portable gamer’s laptop.
SG: That’s the same struggle we all have.
420 Steve Gibson

GH: I think, for me, I am interested more in real-time performed graphics. The
image quality is important, but it’s not as important as responsiveness and
also getting good framerate, just getting good, natural motion. And if the
image quality has to suffer a bit because of it, for me, that’s fine. So I’ve
always kind of followed that.
SG: You can’t really follow that though, if you’re putting it into a theatre.
GH: Agreed! That’s why we captured all the performed gestures, then re-
rendered it to full resolution clips in non-real-time, then edited it into the
film. It was a hybrid performed-post-rendered project.
SG: Great. That’s really interesting. Okay, so here’s a last question. It’s a ‘future’
question, so you may not be able to answer. What are the future develop-
ments you’re planning for Derivative, TouchDesigner or any other venture?
It might be interesting gigs that you have coming out.
GH: I see only a limited distance in the future because we are building a prod-
uct with currently available technology and hunches of what people will
want in the near future, which in itself is a lot! We’re always trying to make
TouchDesigner better for more people in the short term. Now it is used in
schools to teach graphics, real-time media arts and even math and science.
It’s used in interactive art projects, media playback systems, show control,
visualisation projects, puppeteering in theme parks and more. It’s great to see
the synergy of all these fields mixing up – this synergy totally drives us, and
it is great that one product can serve all these needs, and that people can go
off and build their own applications.
In 2005 it was almost unthinkable that a non-programmer could make
their own video mixer product, but now you can actually build your own.
And that idea of building your own interactive applications with a proce-
dural node-based system, high rendering quality, great video playback and
all that. It opens so much up.
Going forward, TouchDesigner has been evolving to interface with a great
variety of input devices and protocols to get live data into and out of Touch-
Designer. Like Oculus VR and Leap Motion, robotic machinery, Arduinos,
DMX lighting control, trackers and all that. That’s natural – TouchDesigner
is designed to f low and process data through it. More web interaction and
more web interoperation as well, and TouchDesigner in the cloud.
I would very much like to be performing more stuff, but as we were dis-
cussing earlier, I spend 95% of my time developing the tool and 5% trying
it out. I actually like building tools, getting them out to people and letting
them take it from there, seeing what they do with it, and informing what
our development team builds next. I personally prefer performing in situ-
ations where there’s zero expectation about what you are about to do, just
turn up, plug some gear in, work with the people around you and experi-
ment. So I like the more guerrilla-style situations.
SG: Some people have like a master plan of what they want to do. You guys have
obviously come a long way right now, to the point where it’s not like you
 Interview 3: Greg Hermanovic 421

need to solve that many problems. At some other companies that are really
just starting up. They need to solve every problem. So they really need to
have a set of milestones for the future.
GH: Yeah. I think we’ve reached a point where the infrastructure of Touch-
Designer is pretty broad and open. It embodies 3D, audio, compositing, user
interface building, data interoperability between TouchDesigner and other
systems. That’s a great starting point. Going forward, we find it most excit-
ing simply keeping pace with the expanding horizons of the TouchDesigner
community.

Notes
1 SideFX Official Website, www.sidefx.com/company/about-sidefx/.
2 Norman McLaren, Synchromy, National Film Board of Canada Vimeo page, 1971, https://
vimeo.com/29399459.
3 Alex Di Nunzio, Cmusic, Musica Informatica: Computer music history and more Web-
site, 2010, www.musicainformatica.org/topics/cmusic.php.
4 Omnibus Animation Demo Reel (1985), VintageCG youtube page, 2009, www.youtube.
com/watch?v=K18ZcE2t1Kw.
5 Ryan Ulyate, David Bianciardi, Judith Crow, and Greg Hermanovic, “Interactive Dance
Club ’98: A Legend in the Making!,” in ACM SIGGRAPH 2018 Panels (SIGGRAPH ’18)
(New York, NY, USA: Association for Computing Machinery), Article 3, 1–2. https://
doi.org/10.1145/3209621.3214887.
6 Peter Mettler, The End of Time, Vimeo on-demand page (paywall), 2018, https://vimeo.
com/ondemand/eot.
19
INTERVIEW 4: MARKUS
HECKMANN, TECHNICAL
DIRECTOR, DERIVATIVE;
PROGRAMMER FOR CARSTEN
NICOLAI AND OTHERS
Steve Gibson

Introduction
Markus Heckmann is a software developer and programmer who works for
Derivative. He has collaborated with alva noto/Carsten Nicolai, Matt Thibideau,
Whitevoid and many others. In this interview he discusses the ideas behind these
collaborations and how he helped to design idiomatic Live Visuals interfaces for
their various projects. He also talks about his own work as wuestenarchitekten.

STEVE GIBSON (SG): You work at Derivative on Touch Designer now, having
moved from Germany to Toronto. Could you describe your background, and
what led you to Live Visual software research and Live Visuals performance?
MARKUS HECKMANN (MH): I started after school, pretty much when I went
to University in Germany to a Technical University, which are focused on
engineering degrees. And the course that they had was called Media Tech-
nology. It was a new course, I think I was in the second class that took this
course. That was in 1998, and they tried to get into the whole media thing.
That’s probably the best way to describe it, because it seemed that they really
didn’t know either what it was going to be yet. So you were exposed to TV
studios, and live recording there. A little bit of everything mixed in with
the typical engineering degree, and stuff like math, electrode technology,
or electronics programming. But of course, student life has lots of parties,
and during the time back then more and more people were performing Live
Visuals at these parties. This probably might be a personal thing, but what
I found always distracting there was that there was content in the visuals.
The use of advertising snippets or news snippets made me stare at the screen
literally, not being able to enjoy the party essentially.
SG: So you were distracted?

DOI: 10.4324/9781003282396-24
 Interview 4: Markus Heckmann 423

MH: I got distracted by it. And during my studies, I did two internships and met
a guy called Sven Gareis from Berlin, he is part of monitor.automatique.1 And
they had, they had different approach to it. They built their own tools in
Macromedia Director.
SG: I remember it well . . .
MH: They were ripping apart pixels essentially, making it a little bit more
abstract. The approach was to more accompany the music, and establish a
feel around that, rather than trying to bring ever faster, changing, recogni-
sable images that might bring a meaning with them. And from that point,
he encouraged me actually to go into the field of VJing. Soon afterwards,
through a small article in 3D World the magazine, I found out about Deriva-
tive and the software that they were creating. This sounded exactly in my
direction, as the software was promoted as being completely generative. You
could create content that goes with the music, that doesn’t necessarily bring
issues to the table. Then I did an internship here in 2002–2003 and I used
the internship also, of course, to learn the software, and then to use it for
myself in my VJing, which I did extensively.
Through that I also met Stefan Kraus in Weimar, and I eventually actu-
ally switched University to the Bauhaus in Weimar to study with him, or to
be closer to him. He was part of a group called MXZehn.2 And they defined
VJing very specifically as video not film. So they took a departure there.
They were called MXZehn or MXTen, referring to the Panasonic mixer, and
they used this mixer just in a feedback loop, so no content. It was a pure
instrument, because if you played it, you could create abstract imagery and
colours on the screen. If you stopped playing, it would white out.
Those two [Gareis and Kraus] were probably the most inf luential people
that brought me into the world of VJing, and they also informed my style
as well. Soon after that I left University without ever finishing, and started
a job here [at Derivative] because the offer came up. And that’s the back-
ground there.
SG: That’s really interesting. Because I can see the abstraction playing out in
your own work.
You’re [the] technical director at Derivative, and obviously, you work
intimately with the development of TouchDesigner. Can you tell me what
your key goals, strategies or methods are for producing and programming
such a vast and complex, interactive system?
MH: I think there’s multiple inf luences there. That’s the one thing that is very
beneficial being in such a small company, and such an open company as
well, is that some of the development is driven by personal interest. There is
development driven through projects that you help on or completely deliver,
and there is inf luence through user input. Looking at the outside world:
what is the world currently doing? What do they need? Of course, you
try to attract outside people, not only your user base, but you’re talking to
everybody, trying to get them to use our software. The development tries to
424 Steve Gibson

please all these different inf luences, and you have to weigh what makes most
sense, and what is achievable? Sometimes it’s about what is it nice to have: is
it necessary? Can you achieve goals without x? And that’s mainly it actually.
SG: So it’s about finding the key thing that’s necessary for either a project or the
software itself? I mean, TouchDesigner is quite vast and complex. So I think
what’s interesting about it is it seems to be able to be made into a number of
different things, as opposed to say a normal piece of VJ software.
MH: You try to cover as many fields as possible. When the first versions of
TouchDesigner came out, the interface and the whole feel of the software
was strictly directed towards VJs. It has left that path quite dramatically
more and more, becoming a content creation tool for interactive projects.
And of course, this is nice, because Greg is practising this and I love practis-
ing this myself: taking it into any dirty club and just VJing with it, while
somebody else might run huge installations that runs in offices or at fairs.
One more thing is actually you judge trends that are currently on the
market, and you try to see if those are very short-term trends that shouldn’t
be implemented, or if those are long-term trends that will last for a long
time, that will stay around?
SG: That’s a tricky one.
MH: It’s tricky. Although you can tell from the advertising market fairly
nicely . . .
SG: To a degree . . . Greg [Hermanovic] and I had a discussion around virtual
reality. Everybody in the early 2000s said, “Virtual reality is pretty much
dead, the helmet kind of killed it.” Now look at what’s happening. It came
back with a vengeance. I think it became kind of uncool for a while for vir-
tual reality: “that’s so 90s.” You know what I mean?
MH: Yes, some things, of course, come back.
SG: I think it was not really very realisable using the technology in that era. I
think that’s why it was dumped, but now it’s much more facile.
MH: It’s also an accessibility and practicality issue. With TouchDesigner, our
task is to find possibilities for users in the short term to develop these things
themselves, and bring it into the software. So providing connections in the
software, where they can hook in and develop it themselves.
SG: Isn’t it because there is a kind of modular system right within TouchDe-
signer where you can add things in?
MH: Yes, there’s that and then there’s a port to any kind of interface protocol,
where we try to interface with lights via DMX, or with other hardware
using MIDI, TCP IP, any of that. You have a wide array of possibilities to
talk to TouchDesigner or communicate with it.
SG: Your work as a live visualist and programmer has involved collaboration
with a number of very prominent musicians, interaction designers and art-
ists, including alva noto,3 Matt Thibideau,4 Whitevoid 5 and many others.
Could you refer to one or two of these, and outline your involvement in
these projects?
 Interview 4: Markus Heckmann 425

MH: The three you mentioned are all a little bit different. They’re distinct, in
how my work is structured. So they’re good examples of all the three paths
that my work takes. Starting with alva noto (see Figure 19.1): working with
him on his projects, also informs my aesthetic, and how I see aesthetics as
well. He has had a huge inf luence on that. He’s the artist in this project and
I’m the programmer. I can experience my creativity through translating his
view on something, and putting it into an installation, into practice. And
this is actually a very rewarding process, because I learned a lot about how
other artists see the world, and I learned a lot about process, how they get to
the finished project.
SG: Did he have a lot of instructions for you or a set programme that he wanted
you to use?
MH: He would give me very precise instructions on what is to be displayed.
Literally very precise, and there’s not much way I can go astray from these
instructions. Other things are more ideas that he would communicate, and
you work on these ideas until you meet the expectation there. I always feel
that this is for me a great learning experience right there. It’s inspiring work:
fulfilling a vision of an artist or completing the vision of an artist. Because
he takes so much with you from that.
SG: So it fed into your own work later?
MH: Yes, I believe so. It very much informs what I do in terms of distilling
things down into simplicity. I would hope that it informs my aesthetic work
as well.
SG: What about the other artists?
MH: On the opposite side, the next one would be the Matt Thibideau collabora-
tion that we did, which is a much more freeform thing (see Figure 12.3 in
Chapter 12). It’s more like a traditional jam actually, where Matt produces
the music, and he would give me some information in the form of MIDI

FIGURE 19.1 alva noto, Conception photo 2012, Elektra 13 Musée d’art contemporain
de Montréal. Used by permission.
Source: Galerie EIGEN + ART/Berlin [email protected]
426 Steve Gibson

notes for example. It gives me some sense of structure. I’m taking that and,
in a way, live coding the performance. I have a base structure, and with an
open system, that’s great to do. Because you build a new branch, you work
on it for a bit, and then you splice it back into the main branch and have
that displayed. You can go through a couple practices. And that’s what we
would do. You set themes essentially, that you agree on, and then you go
from there. With every practice, you get more comfortable with the system
and can go freeform from there.
SG: So you’re responding very much in the moment, even though there are trig-
ger points.
MH: Yeah, definitely. Although I have to say personally, of course, on a large
stage it’s harder to do that.
SG: Especially if you’re doing it at Mutek, which has a fairly discerning crowd.
MH: And the jam sessions usually last longer than the performance itself. It gives
you a little bit more time to get into it. Which is something that I actually
love about club visuals, you have the first two hours to just get into the
rhythm of things. And then you’re good for another two hours.
The third one, the Whitevoid project, that’s the kinetic lights (see Fig-
ure 19.2). This is a purely engineering project. There is there’s creativity in
solving problems. It’s the creativity then of the person who’s programming it
that we’re building in the application. Otherwise, it’s very technical.

FIGURE 19.2 Christopher Bauder’s and Robert Henke’s GRID at Fête des Lumières
Light Art Festival 2013 in Lyon. Whitevoid KineticLights employing
TouchDesigner. Image by Christopher Bauder. Used by permission.
Source: Markus Heckmann [email protected]
 Interview 4: Markus Heckmann 427

SG: Your aesthetic input to that was absolutely minimal?

MH: Yes, aesthetics and node layout and structuring the networks, nothing in
the actual output. My input is very limited there because it is their project,
and they form the shape of the actual installation. In the end I’m just provid-
ing the tools for them to be able to do that.
SG: You worked several years on the installation of Carsten Nicolai’s unidis-
play, and the later adaptation of that into a performance event. Could you
describe the process of developing a visual work for the installation, and how
you adapted that for live performance?
MH: Raster-Noton6 – Carsten Nicolai, Olaf Bender, and Frank Bretschneider –
were here in Toronto, and we invited them to come over to the office. I have
a little bit of a personal connection to them, being born in Leipzig, which is
very close to Chemnitz where they come from. I was exposed to Carsten’s
work, especially through galleries in Leipzig. It was a probably a little bit
random, to be honest, to be able to actually bring them into the office, and
have them look at our software solution, which we thought might be very
fitting for what they were doing.
After bringing Raster-Noton into the office, Carsten invited me to a
residency in Florida for a week as a guest, not as an artist, just as a guest to
work with him closely there and see what we could do with TouchDesigner
for his purposes. And this eventually led to trying to come up with a perfor-
mance that we didn’t really have a concept for. But when he saw the node
system of Touch, he was basically pointing at it and saying, “That’s what I
want.” So from there we came up with the idea to use our node system, and
our user interface as the visual itself. And then it was a matter of transform-
ing the parts of his performance into visual displays in a tree, with the tree
visualising all the points, such as how sound is converted into colour.
SG: So you have a formal system for doing that development. Is that right?
MH: We developed this system with them then, yeah.
SG: So there was a very explicit relation between the sound and the image
generated?
MH: Yeah. That one to one relation, which drove that. And from there Carsten
has this whole branch of works, which are collections of certain things. Like
there’s a grid index, where he assembled all kinds of grids, such as any kinds
of grids that can be formulated in math terms. And then he’s been doing this
display for univrs and unitxt. It’s also an index of different ways to visualise
sound. And this continued into more topics with unidisplay for displaying
time and perception, effects on perception, pretty much how can you trick
perception? And lastly unicolor, which we did in Japan, was centred on the
effects of colour onto the screen (see Figure 19.3). A third project was real-
ized in 2015, called unitape.
SG: So “uni-pretty-much-everything”?
MH: Yes, that’s right. All collections of effects. And this being a screen display,
the node system lent itself to this approach, because you can look at things
closely, but you can also look at the whole collection at once.
428 Steve Gibson

FIGURE 19.3 Carsten Nicolai, unicolor, 2014, DLP-projectors, DMX-LED lights,

projection screen, mirrors, computer, sound, bench with loudspeakers,
dimensions variable. Exhibition view, as part of the exhibition ‘City
and Nature’, Sapporo International Art Festival, Sapporo Art Museum,
Japan 2014. Photography: Julija Goyd. Courtesy Galerie EIGEN+ART
Leipzig/Berlin and Pace Gallery. © VG Bild-Kunst, Bonn 2021. Used
by permission.
Source: Galerie EIGEN + ART/Berlin [email protected]

SG: Your own personal work has an overarching interest in abstract geometrical
forms, deliberate avoidance of realism, or any video-based imagery. I could
be wrong on that, but I don’t remember seeing any video-based imagery in
your work at all. Is this a personal preference? Or do you consider yourself
part of a movement or an aesthetic grouping that favours the abstract over
the real?
MH: Initially, as I described, this was a pure gut feeling, literally not able to stand
the onslaught of images on myself. Later on, meeting other people who fol-
lowed the same path, that got me going more into that topic [abstraction].
And then looking even further, learning about it more, studying, looking
into past artists you find that it’s a very old movement.
SG: In film history, Stan Brakhage . . . .
MH: Fischinger . . .
SG: Norman McLaren, although he has some realistic elements in some of his
films . . .
MH: A big inspiration was Peter Kubelka.7 He has these f licker films, there’s
white and black fields essentially, black being silence and white being every
 Interview 4: Markus Heckmann 429

frequency available. And I find that as a very clean representation of music

or everything. You have these two extremes. Just having these f licker films. It
felt to me that this was the cleanest expression of what I was actually after.
And that’s from . . . I don’t even know, but that’s extremely old as well.
SG: I think those were 1950s or 1660s.
MH: I saw that Ars Electronica had an exhibit See that Sound or something like
that. That was a really, really nice exhibition that taught me a lot about the
history of this whole thing, and how many people have also tried to step
away from pure content, and instead interpret music visually.
SG: Away from representation of a towards abstraction?
MH: Yes.
SG: There’s an interesting counter argument to that. There are many people who
have made it, but particularly with VJing, when it becomes very abstract or
geometrical, it’s kind of like wallpaper. I think there are instances where it
does become that. What would you say that that? How do you avoid just
being kind of a shiny, beautiful thing in the background?
MH: Yeah, definitely. And it cannot become just beautiful, it needs to have an
effect on the partygoer. And there we come to a to a point where you could
say that VJing becomes almost light. You’re not trying to be pretty? Well,
light maybe has a pretty effect as well, but it’s more like an energising effect
or an accompanying effect. I believe also, the imagery that you’re putting on
the wall, needs to have the right rhythm to it. It needs to be loud, or very
dynamic. And that’s where I think you would avoid becoming wallpaper.
And this would be a criticism on the general simple use of visuals in clubs:
that pleasant imagery, like landscapes, or the most beautiful shots can be
completely off topic from the feel of the music, and be just wallpaper.
SG: That especially happens, I think, when you’ve got a scenario that often does
happen in clubs: the DJ shows up and there’s a house VJ, and the house VJ
may not know that DJ’s music, but may have a deep archive of images that
might be completely unrelated to what that DJ is doing. I have to say I have
a bit of a problem with that personally.
MH: And the same of course happens with completely abstract imagery as well. I
probably have found myself in that position as well. Not being able to relate
to the music.
SG: Let’s say you did something for someone who is very explicitly political. It
might not work out because they will probably need some realist content.
MH: Yeah, definitely.
SG: It’s an important thing to think about: the appropriateness of things for the
musical content?
MH: Yes. If I think of another Raster-Noton artist Atom,8 for example, he does
all these visuals himself, and they are very realistic, and perfectly fitting.
You don’t even want to mess with that. It’s exactly what his music needs.
SG: You have an ongoing artistic career as wuestenarchitekten9 which includes
work for gallery exhibition, such as RegX,10 that you did at ArtSpin here
430 Steve Gibson

in Toronto. Could you describe the connection between your work with
Derivative or your Live Visuals work and that work?
MH: The name wuestenarchitekten informs the content. I just picked that name,
and it was difficult for everybody to say, and I thought that’s perfect. But
the underlying thought is that wuesten is the desert. And combined with
architects, ‘desert architects.’ The thought of for me was that the des-
ert resembles complete entropy, complete chaos, the sand being complete
noise. And out of this noise you form your content, your installations.
You pick from, you filter the noise into what you want to say. Going with
that, I’m trying to take out as much as I can, and come up with a single
statement.
SG: So it is kind of minimalist in a way?
MH: Yes, very much. And how it connects to my work with Derivative? It’s
probably around a corner. Because it goes through the VJing experience,
that very much was shaped also by the work here [at Derivative], and by the
work with clients. But then there is there’s probably a need to do more. To
get away from just the party crowd, to actually try to have people look at
something . . .
SG: To actually pay attention to it for a while?
MH: Yeah. And so it’s inf luenced by what I do as a VJ. And what I learned about
what has been done before, and what I have read makes these projects and
shapes them? But I don’t think there’s a direct connection between Deriva-
tive and my [installation] work. That is natural life.
SG: I have maybe a more follow-up question, do you borrow concepts from your
software development in these art exhibitions?
MH: Basically, this is probably true, actually. That’s where I feel there’s con-
nections between them. I should start this way: I like the display of things
connected with science. I find that easy, I find that clear. It’s not a muddy
terrain. Life can be muddy already, so I try to steer clear of that actually, by
using scientific concepts. And working with developers here gives me the
opportunity to learn about certain scientific concepts, concepts in computer
vision, math, that I myself might not be familiar with, but through them, I
actually learn about it, and I can use these concepts or abuse these concepts
in my own work (see Figure 19.4 for an example of Markus Heckmann’s
installation work).
SG: So the kind of hard or scientific things that you do a Derivative, whether
they’re math or other problems that you have to work out, that’s what’s
informing your artistic installations?
MH: Yeah.
SG: That’s interesting, because that sounds very formal. There’s been a whole
discussion because art went through modernism, which ended up in a period
of quite formal, structural work. Especially if you look at music there was
serialism, which tried to organise music on very mathematical terms, and
then that was kind of thrown out the window as being too strict. Then you
 Interview 4: Markus Heckmann 431

FIGURE 19.4 Markus Heckmann, Transduktor, audio-visual installation with Plexiglas

box, fog, projection, and sound, 2015–2016. Used by permission.
Source: Mail conversation with artist available on request.

have the wild, crazy 1980s and 1990s, where anything went. Now I kind of
see a rejection of that anything goes, a return to a kind of formalism, with-
out necessarily being as strict as it was, say in the 1950s and 1960s.
MH: I’m probably borrowing from that, where I try to come up with a rule set
that an installation is created from.
SG: So you do use explicit rule sets?
MH: I try to yeah. Sometimes with the installations you filter it down to be so
simple that the rules that you had are not even visible anymore, because you
distilled it down to a more hilarious simple thing.
SG: What do you consider to be the core conceptual, formal, artistic, techno-
logical, political, or any other values that inform the work that you do? In
your case of technological values, conceptual, artistic values might be the
key ones.
MH: Yes, probably mostly technological values. There might be sometimes a
slight criticism of economic or political values in my work, where I play
off trends that I see in art, because I might not agree with them, especially
looking at interactivity and information visualisation. Otherwise it mostly
comes down to technologically-driven ideas. The word scientific is always
very large. And I don’t want to overstate this: I’m not even close to being
a scientist. I more scrape the surface and pick up certain aspects that I find
interesting and reuse those in my work.
432 Steve Gibson

SG: Well you said previously that science, and I’m assuming science content,
informs your work. So without you necessarily being a scientist, you’re
interested in things that are going on in science?
MH: Yeah.
SG: So then that’s kind of informing how you do things? In what way do you use
sort of science in your work?
MH: You mentioned the RegEx installation earlier. Just to explain what that was.
It’s a very analogue installation essentially. Twenty two-by-fours [lengths
of wood], mounted on a wall painted white, and then projected on with
an ever-changing pattern. The two-by-fours were mounted vertically, and
each two-by-four had its own colour, and very limited colour range. And
the creation of the colours was done through a rule base that picked up new
values from a set of colour ranges essentially.
SG: And what was that triggered by?
MH: Time.
SG: So it just ran through time? There wasn’t necessarily any user input?
MH: No user input, no. That’s the one thing I try to steer away from, from: this
direct interactivity, where the user can see or knows precisely what they’re
doing. The interaction, in my opinion, should be between the viewer and
the piece. And not between the viewer and the device. And so I usually roll
that out and just let it run.
SG: So it’s kind of ambient?
MH: Ambient, yes. It’s actually funny because I had a work where I took just
data off the web. So a shopping list from university for chemicals for their
chemical lab, essentially random data points. I shaped that into a map that
looked like sea chart with moving parts, everything was moving along. The
funny part that I observed when having that on display was that people were
constantly pushing on the screen . . .
SG: Expecting something was going to happen, or was happening?
MH: Nothing happened. First of all, you could see all the fingers smudges on
the screen afterwards. But secondly, there was actually an expectation for
it. I guess because it was a map, people tried to interact with it. That was
interesting. I found it interesting that it was precisely not interactive. But
everybody interacted with it, or a large portion of people interacted with it.
SG: This will be an obvious question in some ways, but how is the use of par-
ticular software and hardware inf luenced your production of Live Visuals?
MH: I see TouchDesigner as a tool, as a hammer or chisel that’s given to me. And
I have the feeling that I can express myself with it on the computer side of
things enough that I don’t actually need other software tools.
SG: So you limit yourself to using TouchDesigner?
MH: Yes, I know my limitations there. I know what I can do. And probably the
output is shaped in these borders actually.
SG: So it really forms what you do as a live visualist. What hardware do you
generally use with it?
 Interview 4: Markus Heckmann 433

MH: That goes from screens, projectors, lights, to lasers.

SG: Do you use any interface devices? Like any kind of control system?
MH: Sometimes I have started using MIDI devices, I have a little FaderFox that
I’m using. It’s nice practical, small. In a live setting, like when VJing, I am
not using any interfaces anymore, but just using the interface in TouchDe-
signer, because I’m doing live coding essentially.
SG: So that really directs your way of working. It’s very hands on.
MH: Yeah.
SG: What earlier movements and Live Visual histories have inf luenced your own
work?
MH: Lately definitely Oskar Fischinger for sure. Kubelka definitely inf luenced
me as well. Lately, I’ve been reading about New Tendencies (NT)11 from
Zagreb in the 60s and 70s (see Figure 19.5).12 Their use of computers in art
would be deeper inf luences.
SG: How have they inspired you?
MH: To an extent aesthetically. They went through a time that I never went
through, where a medium was introduced into their artwork. They had a
chance to really explore, and I find that inspirational, the exploration of a
new medium. I feel it’s very important for me to understand their thought
processes and why they did certain things, and what they did to evolve from
that. For that I tried to understand the medium itself.
SG: In a way we’re entering a kind of middle phase in computing and the arts,
because the early phase is really over: let’s be generous and say from the
1960s to the 1990s. So now we’ve kind of learned to master the tools.
MH: For the last 60 years, it was new media.
SG: And can we call it new media now?
MH: [Laughs] We have a pretty much shown that this field is established, and
you find it in major museums, galleries, it’s there. It’s not new anymore. But
as painters would study their history, I really feel that to evolve further, I
need to understand the roots of it [digital art], that’s why I draw on New
Tendencies’ work.
SG: Are there any current live visualists that you admire and why?
MH: One of my earliest inf luences, MXZehn, continues to be an inspiration to
me, because they have continued on their path. They work on their rules,
and of course, they evolve and change, they become more complex, but
they are still an inspiration to me because you can watch that evolution.
And they are involved in a lot of research into what you what you can do
with computers, with display technology, or lights or more. They open up
possibilities as well that can take you away from the screen. They break
down the screen and use light more as a medium, and evolve live VJing
culture in this way.
SG: What are your future plans for Live Visual productions?
MH: That’s similar in that it equates to where my current inspiration comes from.
And I withheld the names until now. At Mutek I saw Nonotak perform.13
434 Steve Gibson

FIGURE 19.5 New Tendencies 4 poster, Ivan Picelj, 1969. Used by permission.
Source: [email protected]/http://picelj.com/contact/
 Interview 4: Markus Heckmann 435

I think they’re a French/Japanese duo. They managed to break the screen,

personally I think very nicely. The performance there was a triangle that
they were inside, and they projected from the back and from the front onto
a kind of gauze material, in which the back projector interacts with the
front projector. Of course, because the projection shoots through the gauze,
it becomes sculptural. I see light as a sculptural thing. And so that’s actually
where my interest is moving towards: to use light more, or use a projector as
the light source or use light as the source.
SG: And my final grandiose question is, where do you see the future of Live
Visuals going over the next 10 to 50 years? So longer term.
MH: I think we’re seeing it currently. And there are also we could mention
Nonotak again, dissolving the screen. And I don’t think about projection
mapping. That is, I believe, something else, that’s different. It’s just a dif-
ferent screen to synch content on essentially. And when looking at work
by Kimchi and Chips14 – they’re British-Korean, Elliot Woods and Mimi
Son – they did an installation called Light Barrier.15 It’s an array of mirrors,
and by computing how the light hits the mirrors they can create these light
shapes in fog that’s f loating over. And I think that’s really interesting because
you really move into space with your imagery. And that’s actually some-
thing I would foresee getting more attention, really dissolving the screen
completely and moving into space. And that’s actually the long-term thing
I would like to see as well. But it totally depends on the club culture. Of
course, VJing is so much connected to the club culture.
SG: I mean, you see clubs expanding into some strange places. We did a show at
Zouk Singapore, and instead of having screens, they had LEDs all over the
ceiling and walls. I have to say created quite a particular effect.
MH: It’s how your environment is: and you go to this place for this environment.
I still see the VJ also using the space that’s given to them as a free room to
interpret and do whatever they want. While if you do these big installations,
fixed installations, then it’s probably more curated content then I would
guess. I also believe that probably film and video will also be still on the wall
in 50 years. It just works, works with music.

Notes
1 Monitor.automatique Official Webpage, http://monitor.automatique.de/e/frame.html.
2 MXZehn Audiovisual Design Official Webpage, https://mxav.net/.
3 alva noto Official Webpage, www.alvanoto.com/.
4 Matt Thibideau Resident Advisor Page, https://ra.co/dj/mattthibideau.
5 Whitevoid Official Webpage, www.whitevoid.com/.
6 Raster Official Webpage, https://raster-media.net/about-raster.
7 Peter Kubelka Wikipedia Page, https://en.wikipedia.org/wiki/Peter_Kubelka.
8 atom, Artist Page, Raster Official Website, https://raster-media.net/artists/atom?c=5.
9 Markus Heckmann Official Webpage, https://project1.net/.
10 Markus Heckmann, RegEx, 2011, Markus Heckmann Official Webpage, https://project1.
net/works/regex.
436 Steve Gibson

11 Nove tendencije/New Tendencies was an exhibition/conference series of computer art

held in Zagreb between 1961 and 1973, https://monoskop.org/New_Tendencies.
12 Noémie Jennifer, “Turns Out 1960s Yugoslavia Was a Hotbed for Computer Art,”
Vice, August 23, 2015, www.vice.com/en/article/z4qqaj/turns-out-1960s-yugoslavia-
was-a-hotbed-for-computer-art.
13 Nonotak Studio Official Webpage, www.nonotak.com/.
14 Kimchi and Chips Official Webpage, www.kimchiandchips.com/about/.
15 Kimchi and Chips, Light Barrier, 2014, Kimchi and Chips Official Webpage, www.
kimchiandchips.com/works/lightbarrier/.
20
INTERVIEW 5: PETER METTLER,
DIGITAL AND LIVE CINEMA
ARTIST
Steve Gibson

Introduction
Peter Mettler is known for a diversity of work in image and sound mediums –
foremost for his films such as Petropolis and Gambling, Gods and LSD – but also as
a photographer and ground-breaking live audio-visual performer. In this inter-
view he discusses his use and development of Live Visuals software as a means to
tap into a f luid improvised creative process, both for feature film production and
for live performance collaborations with music artists such as Fred Frith.

STEVE GIBSON (SG): You’ve worked as both a filmmaker and a Live Visual artist.
In your experience, what are the fundamental differences between the two
mediums? How do you adapt your film-based work for the live arena or for
Live Visuals projects?
Peter Mettler (PM): Well, my background is essentially coming from music and
filmmaking. And I’ve spent 30-odd years making films as compositions, but
I’ve always been interested in process and especially improvisation, and kind
of putting myself out into an environment and not knowing exactly what’s
going to happen. In other words, not working by script, but rather working
by themes.
SG: So even in your work that’s intended to be screened rather than performed
that’s the case?
PM: Yes, even in the let’s call them classic films, which are not so classic in a
traditional sense. In those films I really tried to work associatively by theme,
and in practice, go out into the world and piece things together until they
resonate as a composition. But of course, that all takes a long time, it can take
years. And a lot of it, especially in editing, is very belaboured. Meanwhile,
you’re considering the structure and the intentions and the themes and the
DOI: 10.4324/9781003282396-25
438 Steve Gibson

narrative arcs and all these things, and you’re really working it out. As much
as you can inject improvisation into the shooting process, it’s always been
hard to do that in an editing process, because you’re editing bit by bit by bit.
So I’ve always been envious of musicians who are able to improvise, have
an instrument, and have their whole body of knowledge and awareness and
just speak with their instrument. That was one of the things that got me into
image mixing: when the technology became such that you can start to do
things like that. It was actually when I was making the film Gambling, Gods
and LSD, which was a very big project (see Figure 20.1). It spanned 10 years,
and it had a lot of extra footage in it. At that point, I started to take the things
that I had shot in digital form or in tape form, and put them on different
cassette tapes and DVDs, and I had maybe eight sources running. I would
just start them all and they’d be running and I’d have analogue mixers like
an MX 50, and various things like that kind of chained hierarchically. We’d
basically improvise and create layers of mixing and editing on the spot.
Before that, I had played in a musical group where I’d done the same thing
with audio tapes. So I was familiar with that kind of process in a sonic envi-
ronment, and now was starting to translate it into a visual environment. I
was using pretty intensive hardware, I had a big arrangement with a lot of
patching. I did my first shows with Fred Frith, who’s an amazing musician
and improviser.1 That was kind of the birth of it, and then I kept following
ways to make that possible. Then I discovered Motion Dive, which is a very
basic Japanese software.2 It’s like an AB mixer, you can have a group of clips

FIGURE 20.1 Peter Mettler, Gambling, Gods and LSD, 2002, Film still. Director, Peter
Mettler. Used by permission.
Source: Mail conversation with artist available on request
 Interview 5: Peter Mettler 439

here of your clips there and combine them together as two tracks. I had that
as one of my sources and I still had all the tapes and DVDs and sounds I was
mixing as well going through to create the final output. Then I found another
software called Isadora,3 and I had a friend build an interface along the logic
I liked, which is actually very much a sound mixing logic that goes by chan-
nels. Under each channel you have all the ways that you can affect that par-
ticular channel, and then you have other channels that combine your previous
channels. I’d be working with four simultaneous channels usually combining
images as they f lowed. And now of course you can control so many aspects
[that you couldn’t at that time]: you can control all the things you can do in
Photoshop, and you can control speed and direction and all sorts of things.
Then I met Greg [Hermanovic], who was introduced to me by Tom
[Kuo]. And for the last eight years we’ve been working on Mixxa, and that is
really getting to the point that it is a f luid performance instrument, designed
along the lines that I like to perform with.
You asked about the difference between film and performance. With
performance it’s about not being able to think too much. I really like the
idea of kind of mining the subconscious, and also to take all these images
that are coming at you on a daily basis, and make them your own in a way,
to recontextualise them to work associatively in the moment, and create an
experience that is one kind of an exploration of your own codes and patterns
and recognitions. But also giving an audience an experience where they can
function like that: where it’s not a storytelling device, it’s really a meditative
immersion into images and sounds, where they can relate in their own ways.
If I do a show, often after the show I’ll have ten conversations and every-
body says something completely different about it, “that meant that and that
meant that,” but it wasn’t my intention.
SG: Sometimes when you’re improvising as a live visualist you aren’t even aware
of the combination of images that is happening. Like you said, if you get
into the moment, you’re often thinking about the rhythm as it relates to the
music, and then the content might slip out of your mind in a way, which,
like you said, would never happen when you’re editing a film.
PM: Yes, you consider a lot, and you can be focused, like you say, purely on a
rhythmic aspect, yet the content is what other people are watching, or the
colour or. . . . I like that, I like being pushed to an edge where I’m kind of
out of control, but I feel f luid. It’s like music. If you play a piece of music, it
doesn’t all have meaning: it has emotion, and it creates states. it’s an internal
world that you’re expressing outwardly. From that, I think I can learn lots
of different things.
There is another aspect. The other big thing that attracted me to it [Live
Visuals performance] is the social aspect of it, because I’m finding more and
more as films get made, whether they’re commercial or art films, they tend
to get shown, and the group of people watching them does not connect to
each other. You know, the extreme example being [watching a film] at a
440 Steve Gibson

shopping mall, where you drive in a car, go watch this thing in a mall, and
walk by the clothes shops and go home. What I like about, for example
rave culture or certain kinds of performance culture, is that it’s a social
context where people are talking and interacting. Of course, if there’s dance
involved that’s another way that people interact and see something and hear
something. That live social aspect is really attractive to me, and I’d like to
bring it in more in the future.
SG: Much of your work as a filmmaker is quite political and focuses on real
world issues, such as in your film Petropolis.4 With a few notable exceptions,
and I mention VJ Greenaway and The Light Surgeons as examples, much
of the Live Visual’s world is dominated by abstract geometric imagery that
shies away from concrete meaning, politics or social issues. What’s your
view of how meaning should be addressed in a Live Visuals context?
PM: Personally, I like the idea in visual mixing of moving in and out of abstrac-
tion. You’ll find I source a lot of imagery that’s my own, but also other
people’s. It can also be an instructional video for a coffee machine. It doesn’t
matter, all these kinds of images that have been made as part of our language
are game for the mix. I like presenting an array of these images where we
recognise them for what they are, but then they start to interact with each
other and in a way decay into an abstraction in almost a hallucination. Or
they decay into something that’s completely materialistic, like you’re aware
that you’re watching a material being created. It’s also recontextualising
things that we’re very familiar with, and perhaps giving them new meaning.
Personally, that’s what I’m interested in. More than pure abstraction. I find I
can only last so long in pure abstraction before I don’t relate to it anymore, it
doesn’t move me anymore. I need shifts back and forth. That’s pretty much
how I’ve been working.
SG: If I look at some of your performance works, or even the video mix works
like the Framemixes pieces (see Figure 20.2),5 the source material always
seems to me to be video or have a reference to the real, even if it becomes
really abstracted, which is for me, personally more interesting than just deal-
ing with geometrical forms. Do you see that as central to the way you do
things?
PM: Yes, I do. I’m interested in in even going further with it, because it’s this
kind of crossroads between what you could call a musical experience or a
sensuous experience, and maybe a narrative or representational experience.
They’re both completely valid to me, and as I say, I like going between the
two worlds, because I think that’s part of how we register our environment,
how we speak in words, yet we communicate emotionally. So those things
are happening in parallel.
I think what happens is that a lot of the gear that’s out there allows you to
do things in abstraction, so it’s very easy for lots of people to just remain on
the abstract level. If you compare it to electronic music, it’s very easy to set
up a bunch of beats and layer them, but there’s something of a soul lacking.
 Interview 5: Peter Mettler 441

FIGURE 20.2 Peter Mettler, Framemixes still – from the image mixing array at Iluzjon
Cinema, Warsaw, Poland, 2013. Used by permission.
Source: Mail conversation with artist available on request.

It [pure abstraction] just doesn’t connect in the same way. I mean, there’s a
lot of beautiful pure abstraction, it’s just for me personally I can’t sit in that
world for too long before I want a counterpoint that makes the abstraction
even more amazing.
SG: In your film The End of Time you included a long sequence that was made
using the HD video mixing tool Mixxa in TouchDesigner. Could you out-
line the process you use to create the sequence?
PM: That sequence was very much applied to the specific film The End of Time
(see Figure 20.3). One of its main themes obviously is our awareness of time,
our perception of time as a construct. And of course, film is a time-based
medium: it has a beginning, middle and end. It reinforces our perception
of beginnings and endings, and arcs and that kind of thing. And the film is
more or less traditional in that sense, but at a certain point towards the end,
it loses the conventions. I mean it seems to end three, four or five times.
And then there’s this “Mixxa section,” as I called it while we were making
it, which really compresses time. It layers a lot of the things that we’ve seen
in the film. It’s kind of like a possible representation of our mind in that
moment of watching the film as it’s referring forwards and backwards, and
in the moment, and layering ideas and jittering them and so on.
I thought it was interesting to use that form in this film, in this time-
based film. Also thinking of it historically, because the first recorded images
were basically a camera was planted and something happened in front of
it, and then the train coming into the station, as in the Lumiere Broth-
ers’ Arrival of a Train at La Ciotat.6 Then that evolved to, for example, the
442 Steve Gibson

FIGURE 20.3 Peter Mettler, The End of Time, 2012, film still. Director, Peter Mettler.
Used by permission.
Source: Mail conversation with artist available on request.

discovery of close ups on people, so you have wide shots and tight shots, and
the language of the medium turned. In that way it became popularised, and
narratives became very popular, and they still dominate filmmaking. What
if in the future we get so used to being online and referencing images so
quickly, as we already do when we wander around on the internet, that we
actually start to layer things and only need very, very short bits of images to
comprehend something and then move on to the next thing? So what was
once a train coming into a station for five minutes as a single shot, now is
just a couple frames mixed with a thousand other things? So that sequence
[in The End of Time] was a suggestion of how time is speeding up, how our
perception of information is getting faster and faster, and how we start to
layer it.
SG: I guess it has a kind of structural purpose too: appearing at the end of the
movie. It wouldn’t work at the beginning obviously.
PM: Yeah. “Where does this come from?” Although I started another film in a
similar way. The beginning of Gambling, Gods and LSD is kind of a mine
of images. It’s about a three-minute passage, but every frame is a different
image: so 24 frames per second, every frame is a different image, and then
they’re layered 10 deep. So it’s really a miasma, and within that miasma you
perceive little traces of things. For that film, that was kind of a suggestion of
the unconscious in terms of an image bank. A lot of the stuff in that miasma
was regular television, or images from everywhere, as well as the images of
the film you’re about to see. And that was before I was doing any of this like
mixing stuff. That was laboriously done in editing, but it’s thematically related
 Interview 5: Peter Mettler 443

[to live mixing]. In terms of the process of making the sequence for The End
of Time using the Mixxa software, you have to perform it. You can’t edit it.
SG: You can perform it 20 times until it’s the right one.
PM: Exactly. So that’s what I did. I performed, I looked at it, I thought “oh, I’d
like to change that.” I performed it again, and just kept performing it until
I had it right.
SG: That’s very much like what a musician would do.
PM: Yeah, and then I tweaked the music or the sound once the image was locked
there.
SG: I have a bit of a sub question to this. Could you explain why you chose to
use a Live Visuals tool on a feature film? You’ve partly answered that, but
maybe more directly, what did this bring to the rendered film that couldn’t
be achieved with conventional editing software?
PM : I think in a general sense what I find interesting about doing both –
continuing to do classical film as well as image mixing – is that I learned a
lot from image mixing in surprising ways. Because you’re sort of forced to
an edge of reaction, you start to see ways of looking at things, juxtapositions
that you can then incorporate into your more classical filmmaking. In this
instance [The End of Time], there’s very literally a performance dropped into
a classical film. I think the logic of association, and the looseness of being
able to work in a more expressionist manner, inf luences the more tradi-
tional, classical filmmaking throughout. It’s like in theatre, when people go
into improv to find out nuances about their character, or discover interac-
tions with each other. Or if you’re with a camera out in the world, you just
let the world inf luence how you’re shooting, as opposed to imposing things
onto it. It’s in a similar vein.
SG: That’s a good analogy.
Your series of pieces entitled Framemixes are intriguing blends of elec-
tronic music, montage and found footage. A number of them seem to ref-
erence earlier film and video history. Number five7 seems to be related to
expressionism. Number three,8 “Stan Brakhage style” experimental film.
Were these historical illusions deliberate in these pieces and what was your
inspiration for making them?
PM: You’re kind of asking the question like it was a regular film. And if it was
a regular film, I would probably answer those questions, addressing that. I
guess the big difference, which is part of what interests me about mixing,
is that I try to start from a place where I don’t explain to myself why I’m
doing things.
SG: So it’s the same as in your live mixing, it’s intuitive in a way?
PM: It’s really just trying to react to imagery. I know The Cabinet of Dr. Caligari
has a huge history in expressionism, but I’m not really putting it there because
I want to say something about Caligari and that form. It just becomes part
of our collective language. Of course, that varies from individual to indi-
vidual, but I’m more interested in what it elicits. And I mean [in Framemix
444 Steve Gibson

#3] it’s actually Caligari and Jesus Christ. So it’s pretty loaded. To me that
mix was about a kind of ecstatic moment, because it’s that character with his
eyes closed, and you see motion around him and abstract forms. As he opens
his eyes, the whole thing gets a lot busier and denser with patterns and stars
around him, and this sort of kitschy Jesus Christ backlit box – which was the
original thing that I found – is f lashing. And it’s kind of like an epiphany. So
if there was a title for it, and I gave it meaning, it would be something like
that: Epiphany. It’s less a reference to either religion or expressionism.
SG: And what about number three? It struck me as more abstract than most of
your other ones.
PM: It’s a combination of things that are chosen, almost on like a tactile level more
than on an informational level. But you can start to think about the begin-
ning of time, with the lava and CERN, this incredible piece of technology
that we’ve created to spin particles and crash them and determine things about
the beginning of time. You know, you can riff on it, but I’m reluctant to say it
means this or that. One thing I found interesting about the mirroring of lava
[used in Framemix #3] is something that’s organic in nature, and f lowing and
soupy muddy in nature, when you fold it upon itself, you start to see – through
symmetry – character, you see faces and things that look like gods, and tribal
masks and all this kind of thing. It made me think about tribal culture before
imagery, how some of their stuff really looks like that, and if maybe that
somehow came from a mirroring effect of their own discovery. Because if you
mirror a lot of things in nature they become symmetrical and start to have
two eyes, a mouth and a nose and they start to look human in a way. A lot of
the totems you see are otherworldly, they look human and they look like they
come from nature. And that was interesting.
SG: Framemix #3 (Figure 20.4) includes some astonishing and beautiful morph-
ing effects in the middle section. Could you describe how you achieve these
effects?
PM: This is actually documentary footage that I shot in India, at a festival or a
ritual that takes place once every 100 years. And that involved these men
getting dressed up with bright orange body paint, and big complex head
dresses that are ornate. So they’re already this incredible kind of visual
design, and basically I just mix that together with fire burning and with
Busby Berkeley 9 swimming choreographies, where the swimmers actually
start to look like chains of DNA. And I added this heart-throbbing orange
in the background. So the four videos are being layered, but they’re also
being controlled in terms of their very slight feedback that creates a diffusion
that blends the layers into each other. So it looks like one whole shape [made
up] of the different elements.
SG: You’ve worked with a wide variety of musicians, both in the live context and
on film projects. Could you describe how you work with musicians in live
performance, and explain your process of collaboration?
 Interview 5: Peter Mettler 445

FIGURE 20.4 Peter Mettler, Framemix #3 still – from the image mixing array at
Iluzjon Cinema Gallery, Warsaw, Poland, 2013. Used by permission.
Source: Mail conversation with artist available on request.

PM: Yes, it’s really varied. I also like to think of this tool [Mixxa in TouchDe-
signer] as something that allows you to do many different things. Just as
with a piano, it doesn’t mean you’re playing the instrument the same way
each time. I can give you a description of a few different relationships I
have with musicians. The one that I’ve cherished and fostered the most has
been with Fred Frith, who I’ve also worked with prior to live performance
where he composed music for me, or did music for me for films. And he’s
an incredible improviser.
SG: I’ve seen him live a few times.
PM: So you know, he’s just extraordinary. And we don’t talk about what we do,
almost nothing.
SG: It’s like a [ John] Cage thing. You show up to the gig?
PM: Yeah, right, exactly. So basically we set up our stuff, he’s got his world
of guitar, going through electronics and loopers and things, with pickups
all over his guitar, and I’ve got my system. I also mix sound with him. So
mixing sound and image and we go! He may start a sound, I may start an
image, and then we start reacting to each other and we see what happens.
So that’s the freest, scariest, and most full of surprises. We’ve also done a
show which was an homage to Andreas Züst, a collaborator of mine who
passed away, who also was a big collector and meteorologist. We did a show
called Meteorologies,10 and I filmed a lot of things related to meteorology out
of his book collection: all these old drawings, paintings, and graphics from
hundreds of years back depicting weather and weather phenomena, as well
his own art, which has a certain relationship perhaps to what we’re doing
446 Steve Gibson

with image mixing. He would photograph objects or other photographs,

and then paint over them with mushroom pigment, and they would decay.
So that was quite nice, because I was using like imagery that already had
a certain sense of decay, mixing it further with this process. And that had a
bit of an arc although there were just basically libraries of things you could
use through the different segments of the performance. So the performance
could still vary pretty dramatically.
SG: Did you rehearse that, or did you still just to show up to the gig?
PM: We never had a rehearsal, but we did a few gigs of that [piece]. So some of
those became things we referred to as we went forward. And then on another
extreme would be I worked with some classical people here in Toronto. We did
something by Shostakovich, and something by George Crumb.
SG: Which George Crumb did you do?
PM: It was Vox Belaenae (Voice of the Whale).11,12 It was a quartet, and they had a
score. I had their score, and I made a visual score for myself. Tom [Kuo] was
assisting me because at that time I needed more arms than I had to drop the
clips into place so that they could be mixed at the right moment. That was like
a score of clips, and I was performing more or less the same thing each time,
but I was feeling it with the musicians, because they also don’t play exactly
the same thing in each performance. The timeframes change. So that’s like a
scored piece, but still using it [Mixxa] as a live instrument. I virtually never
play a mix. I don’t see the point. It’s always something I’m doing on the spot.
SG: But it was a repeatable performance, that would probably be done the same
way twice?
PM: Yes, the same elements, but you know, maybe the violin solo has a different
intonation and is a bit longer, that kind of thing.
SG: Slightly reordered, but in the same form?
PM: Exactly. So that’s another extreme from “techno, party, rave all night long.”
I did a lot of that in Europe, and to me, that’s good fun. I love a lot of the
music. I love the social aspect. I myself am dancing while I’m playing. And
it’s kind of loose. You know, it’s not like all eyes are on the screen, obviously.
SG: I think that probably clubgoers are more forgiving of poorly timed visuals,
more than poorly timed dance music. That means you could be a lot looser.
PM: Yes. And you can experiment, and not worry too much. And then people
come and tell you funny things. And people have requests. Sometimes it’s
pretty funny. “Can you bring that back? That frog face, or whatever.” Then
I worked also with Biosphere, and for the show we did, he’s basically play-
ing compositions that he has, but he has the f lexibility of mixing the tracks
as we go along.13 So it’s a set of stuff he had already, and he’s playing a set
piece, but he has f lexibility within that, and I’m more following him. So
there’s varying degrees of [responsiveness], where the musicians are actually
responding to the image, or I’m just responding to the music. And then in
some instances I’m also mixing sound which becomes part of the music (see
Figure 20.5 for a photo of Peter Mettler performing as a live visualist).
 Interview 5: Peter Mettler 447

SG: How does working in the Live Visuals arena differ from your experience of
using music or commissioning a score for a feature film?
PM: My work in feature film, vis-a-vis music, is unorthodox to begin with,
because I usually cut my own music, even if it comes from other people’s
tracks, parallel to the editing process. I’m not a conventional music user to
begin with. Sometimes, for example, I’ve worked with Fred [Frith], who’s
done both composing music for film, which he does amazingly well, but
also I had him just improvise to an edit that I’d done. And it was quite
funny, because he was playing for a specific sequence and part of the film in
Switzerland that lasted about 10 minutes, and we talked about what kind of
music he would improvise. And then he just kept playing for the rest of the
duration of the film, which turned into this journey through India, which
was a whole other 45 minutes, and he created this beautiful piece of music
just by watching and reacting to the film. Then I took that piece, and basi-
cally it gave me a bed of elements to work with for the whole movie that
I used as themes, and I layered in parts and reconstructed what he had
done there.
SG: So in some ways, kind of like applying a Live Visuals logic to the soundtrack?
PM: Exactly. You see the kinship in the two processes in different ways.
SG: The last somewhat grand question here. In most of your work, there’s a
presence of the spiritual, either explicitly in the imagery or in the mood

FIGURE 20.5 Peter Mettler, live performance – YoshtoYoshto at Videoex Zurich

2015 with Jeremy Narby and Franz Treichler. Used by permission.
Source: Mail conversation with artist available on request.
448 Steve Gibson

created by a particular choice of images and the ambient soundtrack. How

have different concepts of spirituality affected your work, and how do you
feel they’re manifested in the experience of your work? Or is this a mis-
reading of your work, and are you more interested in creating a sense of
wonder towards the natural and human world, or is there even something
else at work?
PM: How do I interpret that, because I don’t really know what spiritual means?
SG: If I compare your work, for example, to Markus Heckman’s work, yours
strikes me as much more spiritual. It has spiritual implications, whereas his
is very formal, very geometrical and abstract.
PM: So when you say spiritual, you mean . . .?
SG: I am thinking the wide meaning of that word . . .
PM: Like there’s a human element, a human consciousness element?
SG: Yeah, or even just a hint of ritual. I see that in almost everything I’ve viewed
of yours.
PM: When I hear that word, I guess what it makes me think of is the psyche, or
the world of perception from the inside. And I think my films and the mix-
ing, are constantly addressing that: the kind of psychology of perception and
the emotional states that we experience, and how that ties into what we see,
or how what we see affects our emotional states. And that includes dream
logic, desire, beauty, wonder, ritual, trance. Trance is a word I haven’t used
in this whole interview, but trance is incredibly important. Because trance
is sort of a state where you start to see things heightened, and you start to
go beyond just registering things with labels and language, and that’s really
exciting to me. And I guess that ventures into the idea of spirituality.
SG: I don’t mean to suggest that you have an agenda that is spiritual, but that just
seems to me that the content has spiritual implications.
PM: I guess that’s because I’m working probably more from an emotional centre.
My world is keyed into nature, and growth, and cycles and things that the
entire body of senses picks up on. But it’s not to say that I can’t imagine some
people feel that in crystal crystalline structures of abstract mathematics, and
for them that can be a spiritual experience in in a different way. It’s just a
different access.

Notes
1 Peter Mettler and Fred Frith, Meteorologies, 2012, Peter Mettler Official Webpage, www.
petermettler.com/meteorologies.
2 Digital Stage, “Motion Dive,” www.digitalstage.jp/en/.
3 Isadora, Troikatronix Official Webpage, https://troikatronix.com/.
4 Peter Mettler, Petropolis: Aerial Perspectives on the Alberta Tar Sands, 2009, Peter Mettler
Official Webpage, www.petermettler.com/petropolis.
5 Peter Mettler, Framemixes, 2011–12, Peter Mettler Official Webpage, www.petermettler.
com/framemixes.
6 Louis Lumière, Arrivée d’un train (a la Ciotat) (Arrival of a Train at La Ciotat), MoMA
Learning, www.moma.org/learn/moma_learning/louis-lumiere-arrivee-dun-train-a-la-
ciotat-arrival-of-a-train-at-la-ciotat-1895/.
 Interview 5: Peter Mettler 449

7 Peter Mettler, FRAMEMIX #5 for 24 Image Magazine (2012), Peter Mettler vimeo page,
https://vimeo.com/55547152.
8 Peter Mettler, Music by Gabriel Scotti and Vincent Hanni, FRAMEMIX #3, Ciné-
matheque québécoise (2012), Peter Mettler vimeo page, https://vimeo.com/53460862.
9 Busby Berkeley (1895–1976) IMDB page, www.imdb.com/name/nm0000923/.
10 Peter Mettler and Fred Frith, Meteorologies, 2012, Peter Mettler Official Webpage, www.
petermettler.com/meteorologies.
11 George Crumb, Vox Balaenae, Wikipedia page, https://en.wikipedia.org/wiki/Vox_
Balaenae.
12 Peter Mettler, Live performance with the Art of Time Ensemble, composer George
Crumb, Vox Balaenae, 2007, Peter Mettler Official Webpage, www.petermettler.com/
vox-ballanae.
13 Peter Mettler, Biosphere Mix, Live image mix performance with music by Biosphere,
The Royal Cinema for Hot Docs, Toronto, 2013, Peter Mettler Official Website, www.
petermettler.com/biosphere-mix.
AFTERWORD
Steve Gibson

Live Visuals: Where To?

The book, whilst informed by the Real-Time Visuals Research Network (2013
to present), was written during a period of considerable upheaval, as the coro-
navirus pandemic waxed and waned for almost two years (and still continues at
the time of writing this). There is no doubt that the pandemic presented some
serious issues for Live Visuals, audio-visual performance and live cinema (as well
as pretty much all of the performing arts community). Venues closed, performers
were stuck at home and performance ground to a halt pretty much worldwide
(with a few notable exceptions, such as in New Zealand). There were attempts
to hold virtual concert events, and many performers converted their studios into
sites for streaming performances, or even just studio recording sessions; however,
it is clear that the sense of communal audience ‘presence’ was at the very least
unsatisfying for most of those virtual events.
Some face-to-face performance events began to be run in spring 2021, includ-
ing well-publicised test events in Liverpool: two club nights, a live concert and
a business festival.1 Some artists made tentative steps to reintroduce touring and
one-off events. Some festivals attempted a return in summer 2021; however, it
was clear that performance had gone through a serious crisis,2 and there can be
no doubt that there will need to be some significant rethinking about the presen-
tation of Live Visuals, live audio-visuals and music events. It may be that virtual
reality (VR) can offer some insights into a more communal form of distanced,
virtual and/or streamed performance if a way can be established for events to be
shared in the ‘metaverse’ with a group of like-minded audience members. Even
in that case, though, the expectation that all audience members would have a VR
headset is more than a bit of a stretch.
DOI: 10.4324/9781003282396-26
 Afterword 451

It is notoriously difficult (and usually foolhardy) to attempt to predict the

future, but one interesting area that continued to thrive during the COVID-19
pandemic was the outdoor projection event, particularly events occurring at
festive seasons. An example shown in Figure 21.1 was taken by the author at
Edinburgh Royal Botanic Gardens Christmas event during the height of the
COVID-19 pandemic in early January 2021. Given the confirmation that out-
door transmission of COVID-19 is very rare, it is plain that outdoor events such
as this are relatively safely achievable and may present further opportunities
beyond the limits of the time-limited festive light and projection display.
Some Live Visual producers have focussed their attention on the public arena,
creating displays that can be used outdoors all year round. Organisations such as
Limbic Media (Canada),3 Squidsoup (UK)4 and Klip Collective (United States)5
have made careers out of doing just that. This was certainly the case prior to the
pandemic as well, but this sort of work has obvious resonance post-COVID and
provides an example of how live audio-visuals can be brought into the public
domain safely and with a better communal sense of audience presence than is
offered by virtual and streamed performance.
All of this is not to say that Live Visuals and audio-visual performance in
more traditional venues such as music clubs and performance venues will not
make a return. They plainly will and already have done so, though with certain
restrictions being somewhat inevitable depending on the laws of the specific
country that they are occurring in. As this book has illustrated in very real terms,

FIGURE 21.1 Light display and projection mapping at Edinburgh Royal Botanic
Gardens, 2 January 2021. Photo by Steve Gibson.
Source: Author’s own image.
452 Steve Gibson

Live Visuals has a long history, and the performed image has survived world
wars, massive cultural shifts and previous pandemics. The experimental use of
technology to perform audio-visuals in real time is a demonstrably wide field
that encompasses so many disciplines and possible venues for distribution that
clearly it will survive in one form or another into the 21st century. This book
has charted the trajectory of this diverse movement from the classical world to
the present, and the future of the performed image will be written by succeeding
generations of artists, designers and technologists. We can confidently predict
that this will be full of both unexpected surprises and new forms of immersion
in the audio-visual that will transport us to previously unexperienced places.

Notes
1 BBC News, “Covid: No Detectable Spread of Virus after Liverpool Pilot Events,” www.
bbc.co.uk/news/uk-england-merseyside-57249289, May 26, 2021.
2 Alexandra Jones, “‘I’m Not Ready for Other People’s Sweat to Drip on Me’: Will Club-
bing Survive the Pandemic?,” Guardian Online, September 17, 2021, www.theguardian.
com/music/2021/apr/17/im-not-ready-for-other-peoples-sweat-to-drip-on-me-will-
clubbing-survive-the-pandemic.
3 “Art Against the Ordinary,” Limbic Media Official website, https://limbicmedia.ca/.
4 Squidsoup official website, www.squidsoup.org/.
5 Klip Collective Official website, www.klip.tv/.
INDEX

16mm film 64, 76, 96, 200, 386–387, Arcangel, Cory 178, 179
393–394 architecture iii, 2, 4, 58, 68–69, 203, 252,
35mm film 54–56, 64, 91, 386, 392 279, 281, 313, 315, 335–349, 364
3D animation 102, 127, 366, 409, 411 Arisona, Stefan 122, 122, 202
Aristotle 11–13, 47, 135, 138
Ableton Live 53, 211, 297, 299, 331, 404, Arkaos 110, 123, 293
413, 416 Ars Electronica 2, 173, 338, 339, 363, 429
abstraction: abstract expressionism 166, Art & Language 174, 180
171, 187, 398; in art 121–122, 127; in audio-visual performance 1, 4, 123, 125,
audio-visuals 105, 157, 398–399, 423, 198, 204, 222, 229, 240, 250–251,
428–429, 440–441; in film 56, 58, 195 258–261, 273, 289, 295–307, 329, 356,
acid house 90, 96–97, 101, 105 400, 450–451; and art 121, 315; history
aesthetics 194, 270, 279, 359, 425; and of 24, 35, 46, 48, 54, 71, 83–84, 106,
advertising 92; and art 109, 118–119, 196, 244; presence in 4, 274, 331; and
165–167, 171, 174, 179–181, 186–187, technology 3, 77, 112, 207n19, 208,
195, 197, 202; and Live Visuals 2, 210
41, 49, 54, 89, 91, 95, 99, 103–106, augmented reality (AR) 188, 188, 206,
127–128, 194, 198–199, 200–201, 336, 338, 342
204–206, 355–356, 360, 369, 393–394, avant-garde 296; and art 115, 127,
426, 427–428; and music 21, 214; 164–167, 170–171, 173–174, 176, 186,
Relational Aesthetics 175–176, 316–317; 188; and audio-visuals 41, 267, 276,
and technology 64, 74, 78, 159–160, 296; and film 51; and popular culture
194–195, 224–225, 227, 345 83–85, 407
affordance 194–195, 208, 214, 223–224,
353, 366 Barber, George 93, 94
Alberti, Leon Battista 3, 309–310 Barthes, Roland 114–115, 121, 174, 197
Aldighieri, Merrill 63, 85–86, 96–97, 123 Baudrillard, Jean 129, 176–177
algorave 205, 205 Bauhaus 151–152, 202, 336, 423
Allen, Christopher 2, 95, 127, 301, 303–306, Belson, Jordan 49, 73, 74, 79
392–406 Benjamin, Walter 176–177, 179
Anderson, Laurie 63, 77, 83–84, 86, 103, Berghain 280–282
304 Bishop, Bainbridge 29–31
Aphex Twin 360, 361 Bishop, Claire 167, 176, 179, 186, 189
454 Index

Björk 360, 366, 373n48 DJ(ing) 77, 96–97, 105–106, 109, 115,
Bourdieu, Pierre 358–359 196, 222, 254, 294, 297–299, 329, 345,
Brakhage, Stan 58, 74–75, 92, 153, 241, 392, 429
247, 293, 408, 428, 443 DMX 126, 420, 424, 428
Brand, Stewart 113–114 drum and bass 94–96, 105, 113, 412–414
bricolage 165, 188 dub 115, 355
Burroughs, William S. 84, 115 Duchamp, Marcel 115, 167–168, 169, 173,
186, 197
Cage, John 62, 65, 115, 149, 151, 153, 445 Duvet Brothers 92, 93, 95
camera obscura 3, 336
capitalism 166, 172, 175–179, 273, 276, Eclectic Method 92, 201
284n44, 354, 366, 370n2 Eco, Umberto 115, 174, 197
Cascone, Kim 159, 161 Eggeling, Viking 51, 152–153, 152
Castel, Louis-Bertrand 1, 18–29, 34–35, Eisenstein, Sergei 49–51, 89, 92, 157, 170,
43, 136, 249, 293 196, 296–297, 332
CD-ROM 103, 107 electronic dance music (EDM) 102, 187,
Chemical Brothers 97, 99, 105, 366, 355, 362
372n45 Emergency Broadcast Network (EBN)
Chion, Michel 157–158, 241, 247 90, 95, 99–100, 104–106, 111, 123,
Chladni, Ernst 142, 143, 146 202, 289, 293
clavecin oculaire (ocular harpsichord) iii, 1, expanded cinema 1–2, 63–67, 74, 77, 257,
18–27, 27, 29, 43, 249 293, 301, 355, 377–382, 386, 389
Coldcut 90, 101–106, 196, 196 Exploding Plastic Inevitable, The 63,
colour music 1, 9, 14, 18–21, 24–26, 28, 71–72, 72
31–35, 46–48, 135–138
colour organ 1, 29–34, 30, 41–43, 49, Final Cut Pro 393, 395
135–136, 200, 290–296 Fischinger, Oskar 41–43, 46, 49–52, 58,
colour-tone analogy 9, 14–21, 24–31, 34 62, 73, 149, 153, 293, 296, 433
conceptualism 109, 121, 127, 186 Fluxus 3, 63, 75, 115, 160, 167–168, 176,
contemporary art 356, 398; and media art 368, 370n9; and audience participation
110, 125–128, 164–166, 170–176, 179, 313, 381; and happenings 69–72; and
186–189; and nostalgia 118; and post- television 195, 198, 202–204
conceptualism 121, 161, 180 formalism 125–127, 164–166, 171–173,
Courchesne, Luc 308–309, 315, 320–321, 186–188, 431
321, 324 Frith, Fred 437–438, 445–447
COVID-19 (Coronavirus) 5, 112, futurism 130, 167
119–121, 129, 270, 277, 282, 283n9,
362–363, 365, 370, 450–451 Gesamtkunstwerk 140, 202–204, 208, 304,
cybernetics 67, 74, 114, 116–117, 119–120, 308
172, 185–186 Gibson, J.J. 194–195
Gibson, Steve 122–123, 122, 202, 329–330,
Dadaism 63, 95, 115, 118, 167–168, 330
171–172, 356, 370n9 Gibson, William 98, 119
Davies, Char 308–309, 313, 322, 322–326 Godspeed You! Black Emperor 200, 386
Dawood, Shezad 121, 308, 325–328, Grau, Oliver 4, 309
331–332 Greenaway, Peter 300–303, 301, 380, 440
DeadMau5 126, 361, 367, 371n27 Guattari, Félix 111, 175
Debord, Guy 170–171, 176
Debussy, Claude 47, 139–140 Haçienda 92, 97, 280, 364
Deleuze, Gilles 111, 175, 183 happenings 69–72, 167
Derivative 2, 123, 126, 290, 298, 407–421, Hawtin, Richie 198, 204, 416; Plastikman
422–423, 430 290, 367, 415–416, 417
Derrida, Jacques 114–115, 174 Hayles, Katherine 116–117, 121, 172
D-Fuse 2, 204 Heckmann, Markus 53, 126, 298–300,
digital signal processing (DSP) 110–111, 123, 299, 416, 422–435
242–243, 289, 291–294, 303–305, 404 Helmholtz, Hermann von 140–142, 141
 Index 455

Hermanovic, Greg 2, 290, 407–421, 424, Krueger, Myron 74, 78–79, 312–313, 319
439 Krüger, Johann Gottlob 25–26
Hexstatic 104–106, 104, 196 Kubelka, Peter 428, 433
Hill, Tony 2, 63, 77, 301, 377–391; Floor Kwan, Alan 308, 325–327, 326
Film/2nd Floor Film 75–76, 76, 377–381,
378, 382, 384; Point Source 390–391 Lanier, Jaron 282, 313, 332n11, 369
hip-hop 64, 101, 105, 115, 355–356, 392 Lászlò, Alexander 43–44, 44
Houdini 407–413 Le Corbusier 58, 63, 68, 68–69, 370n7
human-computer interaction (HCI) 161, Light Surgeons 2, 77, 123–127, 172,
210, 217, 228–230 297, 300–306, 380, 392–406, 440;
SuperEverything* 95, 177, 203, 203,
Ikeda, Ryoji 125, 160–161, 180–181, 247, 303–306, 306, 392, 399, 399, 403–405,
248, 254, 358 405
immersion 3, 110, 119, 269–270, 272, liquid light shows iii, 1, 49, 63, 73–74,
275, 281, 309–312, 329–332, 349, 439, 146, 200, 291–293, 383
452; immersive environments iii, 2–5, Lissajous, Jules 142, 145, 155
77–79, 103, 110–112, 126, 130, 170, live cinema 2, 4, 84–86, 109, 128, 203,
183, 257–259, 307, 308–310, 315–316, 206, 219, 289, 292, 297, 300–306,
319–327, 329–332, 354, 364–368, 366, 306, 331, 392, 396–405, 399, 450; and
389–391, 393–395; and live cinema expended cinema 74–77, 301, 378–382;
304–306; and projection mapping and scratch video 95, 100, 107, 303;
341; and virtual reality 311–313, 322, and second screening 198, 199
326–328 live coding 4, 161, 205–206, 209–212,
installation 68–71, 74–75, 95; art 215, 219, 229–230, 258, 426, 433
installation 110, 117, 170, 180, 181, 183, Lozano-Hemmer, Rafael 79, 121,
280, 335, 344; audio-visual installation 315–319, 332, 338, 339; Relational
63, 77–78, 102–106, 123–127, 161, 167, Architecture 110, 130, 176, 308, 315–317,
178, 204, 246, 254, 294, 298–300, 308, 318, 319, 331
312–316, 318–324, 338, 345, 347, 348, Lucier, Alvin 243–244
353, 365–368, 400–401, 424–427, Lye, Len 41–42, 49, 52–54, 58, 64–67,
430–432, 431, 435; f ilm/video 153
installation 52, 76, 76, 85, 99, 379–382,
386–389, 389–390; VR installation Macromedia Director 112, 115, 423
327–329, 332 MadMapper 110, 123, 291
instrument design 26–32, 43–44, magic lantern 3, 28, 336, 337
240–262 Magnusson, Thor 211–212, 211, 228,
interactive art 4, 69, 76–78, 125, 310, 240–242, 251
313–316, 331, 420 Marxism 166, 172, 284n34
Interactive Dance Club 411–414, 412 Massive Attack 203, 356, 357
Isadora 110, 418, 439 materiality 92–93, 94, 121, 177, 353–354,
364
jazz 52, 56, 72, 85, 168, 294, 392, 396, Max (MSP) 112, 155, 161, 292, 297
407, 410 McLaren, Norman 42, 49, 52, 54–58, 92,
Jordà, Sergi 215, 229–230 148, 153–154, 249, 332; Blinkity Blank
Joshua Light Show, The 54, 63, 73–74, 73, 54–56, 57; Dots 54–56, 249; inf luence
90, 96, 200, 293, 407 of 62–67, 92, 92, 293, 296–298;
Synchromy 54, 57, 408, 416, 428
Kandinsky, Wassily 47–51, 138–140, 149, McLean, Alex 211–212, 251
152 McLuhan, Marshall 174, 195
Klang farbe (‘sound-colour’) 140–141 Messiaen, Olivier 47–49, 141–142
Klee, Paul 49–51, 50, 152 Mettler, Peter 4, 437–448; The End
Klip Collective 110, 113, 123–127, 124, of Time 417, 419, 441–443, 442;
339–340, 451 Framemixes 440, 441, 443–444, 453
Knife, The 361, 366, 371n27 MIDI 53, 86, 112, 201, 207n15, 244,
Konx-Om-Pax 358–359, 359 299, 404, 409, 418, 424–425; MIDI
Krauss, Rosalind 118, 175, 179 controllers 44, 46, 53, 77, 98, 110, 201,
456 Index

206, 291–293, 297, 304, 411, 414, 416, photography 165–168, 175–177, 182, 197,
433 283n8, 345, 360, 395
minimalism 168, 171, 186, 189, 356–357 Pink Floyd 73, 83–84, 336
Ministry of Sound 277, 364 Pop Art 3, 63, 71, 75, 167, 171–172, 186,
Mixxa 417–419, 439, 441, 443, 445–446 189
modernism 41–44, 46, 62–64, 66, 69–71, post-conceptualism 109, 121–122, 130,
130, 138, 165–166, 174, 353, 430 161
Modul8 43, 79, 110, 123, 290–292, 292, 331 postmodernism 109, 115, 166, 170–175, 186
Moholy-Nagy, László 148, 159–160, 202 post-punk 85, 96
montage 51, 81, 89, 92–93, 122, 157, 170, Prodigy 97–99, 106
196, 296–297, 300, 443 projection mapping iii, 2–3, 74, 110,
Moog, Bob 66; Moog synthesizer 65, 66, 122–127, 203, 291, 317, 339, 345, 349,
79, 408 368, 435, 451
motion graphics 91, 99, 101, 194, 289 punk 83, 85, 97, 105, 197, 366
MTV 81, 86, 89, 96, 100, 104–105, 120, Pythagoras iii, 10–11, 11, 47
195
multimedia 77, 100–104, 107, 115, 140, Raster-Noton 398, 427, 429
157, 173, 180, 198–199, 304–305, 392; rave 86, 89–90, 95–99, 98, 101, 105–107,
multimedia performance 63–64, 71, 107n8, 127–128, 201, 285n55, 355,
83–85, 167, 202–203, 228, 304 363, 440
music video 42, 52, 63–64, 79–81, 86, Reactable 213, 213, 215–216
89–92, 100–105, 195–196, 201, 296, real-time visuals iii, 329, 450
360, 384 remix 101–103, 104, 116, 122, 167, 294,
musique concrete 55, 62, 64, 67, 157, 295, 355, 369, 396
355–356 repurposing 67, 73, 75–76, 122, 165, 168,
MUTEK Festival 123, 276, 299, 363, 416, 365
426, 433 Residents, The 85, 103, 107
MXZehn 110, 123–125, 423, 433 Resolume 43, 110, 123, 290
Rhoades, Jason 117, 181, 181
Nelson, Theodor H. (Ted) 115, 121 Richter, Hans 51, 152–153
Newton, Isaac 10, 14–21, 18, 20, 26, 47, Rimington, Alexander Wallace 31–34,
136–137, 136, 140 32, 41, 43, 47, 137, 200, 293, 296
Niblock, Phill 243, 247, 257 Ritter, Don 308–309, 323–325, 325, 327
Nicolai, Carsten 110, 126, 130, 398, 422,
427–428, 428; noto, alva 46, 53, 77, Sandin, Daniel 78–79, 80, 155
125–127, 296–298, 300, 307, 358, 422, saudade 113, 118–119, 129
424–425, 425 Schaeffer, Pierre 55, 62, 64, 157–158, 241
Novak Collective 123, 269 Scheinwerfer 54, 123, 290, 292–296, 295
Schoenberg, Arnold 47, 50, 140–141
Oculus 270, 281, 313, 323, 325, 331, 420 scratch video 1, 64, 67, 81, 83, 86, 90–107,
Oneohtrix Point Never 118, 357–358, 358 94, 115, 128, 195–196, 200–201, 289–293,
openFrameworks 224–225, 232 303
Open Sound Control (OSC) 299, 409 Scriabin, Alexander 41, 45–47, 45, 49, 58,
Orbital 97–99 196, 296–297
Osborne, Peter 121, 180–183, 187 semiotics 96, 252
Sensors_Sonics_Sights 122, 249, 251, 297
Paik, Nam June 69–71, 70, 74, 78–81, 91, Severed Heads 63, 81–82, 96
97, 172–173, 195, 257 Shaw, Jeffrey 63, 79, 83, 308, 313–316,
Panasonic MX10 and MX50 89, 395, 418, 314, 331, 335, 338
423 Sherwin, Guy 63, 75–77, 254, 256, 301
Papanek, Victor 352–353 SIGGRAPH 363, 411–422, 412
parametric visualisation 4, 240, 243, 246, Silicon Graphics 322, 412
249, 252–253, 256, 259–261 Single Wing Turquoise Bird 54, 63, 200,
performance art 69, 84, 171, 267, 398 293
Pfenninger, Rudolf 148–149, 160 Snow, Michael 74, 293, 408
 Index 457

SoftImage 322–323, 408 VDMX 290, 304–305, 404

Sónar 106, 363 Velvet Underground (band) 63, 69, 71–73,
Son et lumiere 335, 337 72, 200, 298
Squidsoup 123, 183, 319, 451 Velvet Underground (venue) 345, 346
Stockhausen, Karlheinz 62, 64–65, 69, VHS (video) 64, 89–92, 95–97, 100–101,
115, 139, 154 105–106, 394–395, 418
suprematism 170–171 video synthesizer 1, 63–64, 70, 70–71,
surrealism 115, 166–172 79–82, 80, 97, 208, 212
Sutherland, Ivan 311–312, 312, 319, 322 virtual reality (VR) iii, 1, 4–5, 110, 122,
synchresis 158 165, 206, 229, 272, 280, 309–312,
synesthesia 4, 41, 44–47, 50, 74, 138–141, 331, 332, 332n11, 369, 424, 450; and
329–331 immersion 130, 322–329; presence in
systems art 116, 168, 171–174, 185–186, 4, 270, 282; VR hardware 338, 342,
189 420; VR installations 78, 113
visual music iii, 1–3, 24, 39n86, 42–43,
Tanaka, Atau 122, 249, 251, 259 46–58, 62–67, 81, 138, 149, 152, 157
tangible interfaces 4, 209–210, 212–213, VJ(ing) iii, 1–3, 109–115, 114, 122–123,
215, 230 168, 185, 195–197, 222, 254, 276,
tastiera per luce (luce) 41, 45–46, 45, 289–290, 293–301, 301, 304–307, 329,
296–297 346, 358, 392, 396, 407, 410, 414–416,
techno 90, 96–99, 105, 107, 111, 118, 280, 423, 429–430, 433–435; historical
355, 414 antecedents of 54, 71, 74, 77, 81–82,
telepresence 270–271 85–86, 96–107, 115, 123, 167, 315; VJ
theosophy 49, 139–140, 149 hardware/software 79, 92, 110–112,
Theremin, Leon 43, 47–48, 154 123, 201–203, 208, 290–292, 297,
theremin 48, 297, 402 302–303, 331, 424
Thibideau, Matt 298, 299, 422, 424–425
Threnoscope 211, 212, 251 Warhol, Andy 63, 69, 71–72, 72, 91, 172,
Tobin, Amon 125, 204; ISAM 123, 203, 178, 179, 200, 298
274 Wheatstone, Charles 142, 144, 310, 311
TouchDesigner 2, 53, 110, 123, 126–128, Whitevoid 298, 422, 424, 426, 426
290, 298–300, 407–421, 423–427, Whitney, James 49, 58, 66–68, 73, 101,
432–433, 441, 445 154, 195
transmedia 280, 308–309, 331 Whitney, John 49, 58, 62, 66–68, 73, 136,
Transmediale 173, 363 154–155, 195, 293
Wodiczko, Krzysztof 335, 338
U2 100, 106, 180, 202, 305, 392 wuestenarchitekten 422, 429–430
Universal Everything 120, 120
user-centred design 217–218, 230–232 Xenakis, Iannis 49, 58, 68, 151

Vanderbeek, Stan 74, 383 Youngblood, Gene 63, 67, 74–75; Expanded
Varèse, Edgard 47–48, 58, 62, 65, Cinema 2, 74, 117, 139, 173–175, 379,
68–69, 68 383, 407
Vasulka, Steina 77, 244, 244, 247, 249, 257
Vasulka, Woody 77, 249 Zappa, Frank 63, 73, 73, 84