Applications

Editor: Mike Potel

http://www.wildcrest.com

Applications of Computers to Dance __________________

Tom Calvert
Simon Fraser
University
O f all the art forms, dance has probably been the
slowest to adopt technology. In part, this reflects
the reluctance of dancers and choreographers to let
human movement—began to work with Cunningham
and other choreographers to explore this approach. The
result was the evolution of Life Forms into a tool known
anything get between them and the live kinesthetic as Life Forms Dance and now as DanceForms
Lars Wilke experience. But it also reflects the fact that useful tools (http://www.danceforms.com). This tool lets the chore-
Credo have been slow to develop in a market with limited com- ographer try out ideas before ever meeting with live
Interactive mercial opportunity. dancers. None of DanceForms’ features would surprise
Probably the first article suggesting the application those working in human figure animation; however, it’s
Rhonda Ryman of computers to dance was published by A. Michael Noll customized for dance. Similar features exist in Curious
University of in Dance Magazine in January 1967, although New York Labs’ Poser, Alias’ Motion Builder and in general-purpose
Waterloo choreographer Merce Cunningham discussed this at animation systems such as Alias’ Maya and Discreet’s 3ds
about the same time.1 These initial discussions sug- max. However, most of those applications are opaque to
Ilene Fox gested visualizing dance with animated stick figures and the typical dance choreographer or dance teacher.
Dance Notation using these visualizations to plan choreography. Soon As Figure 1 (page 8) shows, DanceForms provides the
Bureau other choreographers suggested using computers to choreographer with a stage, where figures can be
compose and edit dance notation scores and then to placed. While it’s possible to use any body with an artic-
translate these scores into animation. Dance notation is ulated skeleton (for example, a spider, a horse, and so
analogous to music notation, but is less widely known— on), the standard bodies provided are male and female
the “Labanotation Primer” sidebar provides a primer on human figures for ballet and modern dance. After select-
the notation system most widely used in North America. ing a figure or figures, a keyframe animation approach
As computer technology has developed and become builds up the movement for each of the dancers. A stu-
less expensive, many artists have found ways to use it to dio window provides a flexible set of controls to refine
enhance their performances with interactive multime- the posture of individual figures, including 3D poten-
dia. This has included incorporating computer-generated tiometers for joint angles. Limbs and limb segments can
images and sound with live dance performance and be directly manipulated with the mouse, with or with-
using sensors that let the live dancers’ movements con- out inverse kinematics (the IK algorithm calculates joint
trol imagery, sound, and a wide variety of special effects. angles of a limb as the user positions the endpoint). It’s
This overview describes some of the current applica- also possible to import motion capture data using one
tions of computer graphics to dance including visualiz- of the standard formats.
ing choreography, composing, editing and animating As the user adds keyframes, their sequence can be
dance notation, and enhancing live performance. viewed and edited in the score window, which is essen-
tially a timeline. The partially composed piece can be ani-
Visualizing dance and planning mated at any time in the stage window, using a variety of
choreography body representations including stick figures, bounding
When composing a dance, a choreographer typically boxes, outlines, or mesh surfaces. Alternatively, the piece
works with one or more dancers to physically set the can be rendered without perceptible delay and played in
movement of each dancer over time. For any significant the 3D performance window.
piece, this can take weeks or months. The choreogra- A wide variety of choreographers—including
phy requires not only the dancers’ time but also access Cunningham and Danny Lewis—have used this chore-
to appropriate rehearsal space in a dance studio. This is ography package. Educators in K–12 and postsecondary
expensive and dance companies work on restricted bud- institutions also use it—particularly innovative educa-
gets. A computer tool that could be used to plan the tional users include the states of Kentucky in the US and
choreography would help minimize the time needed for of New South Wales in Australia. While the tool has
live rehearsal. merit in its own right as an approach to thinking about
In 1991, the founders of Credo Interactive—the devel- movement in 3D and across time, it helps with resources
oper of Life Forms, a software package for planning by letting students in more common computer labs plan

6 March/April 2005 Published by the IEEE Computer Society 0272-1716/05/$20.00 © 2005 IEEE
 Labanotation Primer line divides the left side of the body from the right. Symbols
Labanotation is a system for analyzing and recording to the left of the center line refer to the left-hand side of the
human movement. The original inventor, Rudolf von Laban body, symbols to the right of the center line to the right-
(1879–1958) was an important figure in European modern hand side of the body. Some body parts must be identified
dance. He published this notation first in 1928 as by a symbol, see Figure C.
Kinetographie. Several people continued the development
of the notation and Hutchinson Guest has written the (1) (2) (3) (4)
classic text.1 In the US, the New York-based Dance
Notation Bureau supports Labanotation and archives many C Symbols indicating body parts: (1) head, (2) face, (3) hands,
Labanotation scores (http://www.dancenotation.org). and (4) front of left shoulder.
Each Labanotation symbol gives four pieces of
information. First, the symbol’s shape indicates the
direction of movement (see Figure A). Finally, the symbol’s length indicates duration of the
movement. The staff is read from the bottom up; moving
Forward ahead in time (see Figure D). The tick marks on the center
line divide the time into counts and the horizontal lines
Left forward Right forward correspond with the bar lines in the music. Movements
diagonal diagonal written on the same horizontal line occur simultaneously;
movements written one above another occur sequentially.
Measure numbers and dancers’ counts appear to the left of
Side left Side right
the staff.
Place

Left back Right back

diagonal diagonal

Back

Right leg gesture

Left leg gesture
A

Right support
Direction symbols.

Left support

Upper body

Right hand
Right arm
Left hand
Left arm
Torso

Head
Next, the symbol’s shading shows the level of a
movement; diagonal strokes for high, a dot for middle, and
blackened for low (see Figure B).

Left side Right side

(1) (2) (3) of body of body

B Symbol shading: (1) forward high, (2) place middle, and (3)
right side low. D The staff.

Third, the symbol’s placement on the column on the staff Reference

indicates the part of the body that is moving. A 1. A. Hutchinson Guest, Labanotation: The System of Analyzing
Labanotation staff represents the human body; the center and Recording Movement, Taylor and Francis, 1987.

choreography that is later performed in scarce dance tent provides the teacher and the student with a rich
studios. They can then make better use of the time spent environment in which to explore dance animation.
with the dancers. Complementing these modern dance and ballet moves
DanceForms is suited as a tool in dance education, content collections, Ryman has published a tutorial-
introducing students to a variety of dance movements based practical guide to Dance Forms 1.0 (http://www.
and allowing them to explore new ways to expressively dcd.ca) offering a hands-on approach to learning the
combine those movements. Rhonda Ryman at the program with step-by-step exercises.
University of Waterloo has created comprehensive con-
tent including dictionaries for ballet (Ballet Moves II) Composing and editing dance scores
and modern dance (Modern Dance Moves); both are The Labanotation primer in the sidebar shows that a
published by Credo Interactive (http://www. score consists of symbols on a vertical staff. The symbol’s
danceforms.com). Ballet Moves II details hundreds of meaning depends on its shape and shading, vertical
positions and steps as codified in the English and length, and placement in one of the staff columns. Several
Russian schools of ballet, as well as select samples from word-processing-type editors support the composition
the Italian school. Figure 2 shows an example. This con- and editing of Labanotation scores. These include

IEEE Computer Graphics and Applications 7

Applications

1 DanceForms
provides
(a) a stage
window for
(a) (b)
composing
multiple
dancers,
(b) a studio
window for
creating partic-
ular body posi-
tions, (c) a score
window to
show how each (c)
dancer moves
over time, and
(d) a rendered
performance
window.

(d)

■ AutoCAD-based Calaban, which is particularly use- Macintosh-based 2D graphics editor specializes in creat-
ful for creating high-quality printed scores (http:// ing Labanotation scores (a Windows version is being
www.bham.ac.uk/calaban/frame.htm); developed). It was developed at the Dance Department
■ AutoCAD-based Labanatory, which aims to let users of the Ohio State University by George Karl, Scott
search a score (http://www.labanatory.com/); Sutherland, and David Ralley under the direction of Lucy
■ X-Windows-based Labanotation LED developed in Venable. LabanWriter lets the user create one or more
Australia (http://www-staff.it.uts.edu.au/~don/ staffs on the virtual page and provides a palette of
pubs/led.html); Labanotation symbols that can be selected and placed on
■ MacBenesh for Benesh notation, developed by Ryman the staff in the columns representing the appropriate
and her colleagues at the University of Waterloo and body part. Direction and turn symbols can be stretched
now maintained and distributed by DanceWrite in the vertical direction to indicate their duration.
(http://members.rogers.com/dancewrite/index.html); Modifier symbols can be selected and placed in proximi-
and ty to the main symbols or columns that they modify. The
■ Benesh Notation Editor developed by the Benesh development of a score is illustrated in Figure 3.
Institute in London for editing multistave notation LabanWriter treats symbols strictly as 2D graphi-
scores (http://www.benesh.org/frames.html). cal objects that fall loosely into two subclasses:
stretchable and fixed sized. Stretchable symbols
These editors work well, facilitating the writing and have associated time durations, such as direction
more importantly the editing of scores. However, the symbols and turn symbols, while fixed size symbols
most widely used editor is the LabanWriter are normally modifiers of columns, measures, or
(http://www.dance.ohio-state.edu/labanwriter). This other symbols, or are used to annotate the score.

8 March/April 2005
Column and timing information is
not explicitly stored with the sym-
bols. The symbols are organized
into columns and measures only
insofar as their Cartesian posi-
tions indicate. LabanWriter has
no knowledge base for parsing the
score and nothing prevents the
user from entering complete non-
sense.
The LabanWriter file (.lw4) is an
ASCII file that stores only enough
information to reproduce the
arrangement of the graphical sym-
bols just described. Each record in
the file contains the type, position,
size, color, special fonts, and other
information related to the drawing
of the symbols. The order of symbols
within the data stream is the one
laid down by the user. No attempt is
made to sort the symbols in any 2 Ballet Moves II provides a dictionary of ballet positions and movements, including this
other way. (A detailed description of palette of basic positions in the English school.
the file format is available at
http://www.dance.ohio-state.edu/
labanwriter/LW4/LW44FileFormat.
html). LabanWriter can also print the Labanotation
score, or produce a 2D raster image in one of three pop-
ular graphic file formats (.png, .pict, or .jpeg).
LabanReader, also developed at the Ohio State
University, is a software program for selective view-
ing of notation symbols in a dance score. It can be
used to focus on particular aspects of movement for
teaching or to display various kinds of patterning for
movement analysis. LabanReader is suited for use
with scores created in LabanWriter 4.0 or later.
As mentioned previously, LabanWriter and other
notation editors fulfill a useful function. However, to
date, the developers have not accounted for knowledge
of the score’s structure (syntax or semantics). In con-
trast, almost all word processors check spelling and
many make suggestions on grammar. The notation edi-
tors so far have not applied rules to check the structure
of what has been composed—they are merely drawing
systems for placing symbols on a staff. Another issue is 3 Composing a score with LabanWriter. The user selects
their inability to search a score for a particular move- from the main menu (above left) to see pop-up menus
ment pattern (Labanatory has some searching capabil- containing families of symbols or signs (above right)
ities). One reason that notation editors are so limited is that can be placed on the three-lined vertical staff.
that no consistent, unambiguous way exists of repre-
senting human movement with a machine-readable
ontology or grammar of the kind that can be created in Credo Interactive, has led the development of
XML, for example. LabanDancer, a program to translate LabanWriter
scores into 3D animation.
Translating dance notation into
animation Creating a composite score
Not everyone in the dance community can read nota- The first stage in translating LabanWriter files is to
tion, and writing dance scores is a specialized skill. A convert the stream of graphical symbols into a set of
program to translate notation into animation (and vice composite data records that have meaning in the
versa) could make the notation more accessible to Labanotation context and to populate a data structure
choreographers, dancers, and students. The New York- that is a more direct analog of the Labanotation score.
based Dance Notation Bureau, in collaboration with In this process, graphical symbols are spatially sorted
Simon Fraser University, University of Waterloo, and and assigned to columns and measures based on the

IEEE Computer Graphics and Applications 9

Applications

symbol position relative to the staff origin. Modifier sym- er, explicitly specify the path to be followed in carrying
bols are simultaneously collected and later associated out the gesture. The path can be deduced once the start-
with the symbols or columns that they modify based on ing and ending orientations are known. LabanDancer
their proximity (an inherently error-prone operation). uses an inverse kinematics algorithm and applies con-
The composite score thus derived contains a map of straints to ensure that articulated limbs carry out the
animation channels; each channel is associated with a movement in an appropriate plane. For some move-
different limb segment or support (in most cases this ments it’s necessary to add intermediate keyframes as
corresponds to a column in the Labanotation score). The additional constraints to ensure that articulated limbs
data records within each channel are ordered by time, do not move inappropriately, such as passing through
and each record contains all possible modifier informa- other body parts, for example.
tion. Ancillary information, such as timing, paths, or The parser that interprets gestural commands is sim-
relationships is also stored in the data structure outside ple at a high level—as noted, final limb orientations are
of the channels. In a final pass, the data record timing is usually explicitly specified. However, at a detailed level
adjusted to reflect the real timing of the gestures accord- the parser is extremely complex since it must ensure that
ing to context-dependent rules. every movement is appropriate to the context. In addi-
tion to the constraints discussed previously, other con-
Kinematic model textual issues include the dance style—for example, a
The LabanDancer application uses a deformable, foot might be pointed in ballet, but not in folk dance—
polygonal mesh model driven by a hierarchical skele- and the timing conventions used in Labanotation—
ton. Keyframe animation channels control joint angles although a gesture is written within a beat, the usually
in the skeleton, and four inverse kinematic chains drive understood timing is for it to begin moving before the
the arms, legs, feet, and hands. The software uses the beat and arrive on the beat. This can vary depending on
IK using analytical methods (IKAN) algorithm, which is the context. For example, arm gestures must be syn-
specialized for controlling human-like limbs with 7 chronized with leg gestures, which in turn are con-
degrees of freedom and has a deterministic rather than strained by the support symbols—enough time must be
iterative solution.2 IKAN also has the advantage of allow- left to complete the leg gesture before it contacts the
ing control over the elbow or knee joints, which is impor- ground for the next support, and a leg cannot gesture
tant for achieving all possible poses specified by the until weight is transferred to the other leg partway
Labanotation score. Keyframe channels control the end through the step.
effectors, representing foot and hand position and ori-
entation. These channels can interpolate position either Interpreting support changes
linearly, or spherically, a necessary feature in creating In Labanotation the concept of support and support
believable paths between poses. A 3D Bezier curve—the change is the basis for all locomotion. The notation
parameters of which a footsteps algorithm determines— shows which limb supports the body over a period of
controls the global position of the root of the hierarchy time and the direction (if any) of the movement. Support
(that is, the pelvis). changes from foot to foot combined with forward move-
Currently, LabanDancer provides four separate mod- ment, for example, result in a forward walk or run. Thus
els, two female and two male, customized for modern the notation does not explicitly specify the flexion and
and classical dance. Users can easily create more mod- extension of the limbs concerned, leaving it to the intel-
els representing different dance traditions by using com- ligent dancer to recognize which movements are neces-
mercially available modeling programs. sary to achieve support change in a specific direction.
One approach to the animation of support changes
Three command types would be to store keyframe sequences for all possible
The composite score is parsed into three streams. The transitions; however, the number of sequences increas-
first interprets gestures—these are non-weight-bearing es exponentially with the movement’s complexity.
movements of any body part. The second interprets sup- LabanDancer uses a general approach based on van de
port changes (including locomotion), and the third Panne’s footprints algorithm for animation of human
involves other issues such as repetition of a sequence, locomotion.3 This algorithm calculates an optimum path
paths, use of floor plans, and so on. for the center of gravity of a biped based on the place-
While much of Labanotation is explicit—it objective- ment and timing of the footsteps and the geometrical
ly specifies the orientation of limb segments at a partic- configuration. Once the path of the center of mass is
ular time—there are numerous instances where the known, keyframes for the feet are generated, based on
Labanotation is implicit—relying on notator and dancer the position and timing of footsteps and any restrictions
knowledge to deduce the movement in the context of imposed on the flight phases of the foot. The foot
the piece in question. Thus, the translator program must keyframes drive the goal positions for the IK chains dri-
include a knowledge base from which movements can ving the legs. Important in the choice of this algorithm
be interpreted in their context. was the fact that it could take changes in level into
account, something that is important in Labanotation.
Interpreting gestures Also, the algorithm seamlessly handles the transition
The Labanotation symbols generally indicate unam- from walking to running, jumping, or hopping, and can
biguously the start time, end time, and final orientation handle locomotion along a curved path.
of limb parts involved in a gesture. They do not, howev- To implement this approach it’s necessary to find a

10 March/April 2005
way to generate the timing and position of the footprints
from the composite Labanotation score. The footprints
algorithm specifies the footsteps in absolute position
and relative (to the last) time, while Labanotation spec-
ifies support changes in absolute time and relative (to
place) position. The translation from one system to the
other is relatively straightforward.

Turns, pivots, and other commands

Turn symbols within the support columns of the
Labanotation score indicate a turn or pivot. The foot-
steps algorithm that we use does not adequately take
this type of motion into account, and thus some modi-
fications were required. Also, many other Labanotation
symbols do not translate into either gestures or changes
in support. These include techniques to handle multi-
ple dancers, the use of floor plans, symbols to indicate
repetition of a sequence, and so on. The parser recog-
nizes these symbols and has procedures to implement
the corresponding movement.
4 LabanDancer interface lets the user follow the animation of a
LabanDancer prototype Labanotation score. Using the score or animation cursor the user can go
The first LabanDancer prototype has been imple- back and forth, comparing the movement with the notation.
mented for Windows and Mac OSX. Figure 4 shows the
user interface. Fairly standard controls allow the user to
adjust the viewpoint by rotating the stage on all three human movement. Notation systems, such as
axes and by providing for zooming in and out. In addi- Labanotation, do this at a conceptual level, but since the
tion to totally interactive control with the mouse, fixed notation itself does not have an unambiguous machine
views can be chosen or the camera can follow the machine-readable representation, the notation is not
dancer. The display can show the dancer’s footprints, enough. The composite score data structure developed
defining the path of the movement. Also, a sound out- for LabanDancer is a pragmatic attempt to address this
put provides a metronome-like tock sound to indicate problem. A more principled approach must start by
each beat. An important interface feature is the display defining an ontology or grammar that captures the nec-
of a simultaneous graphic of the original Labanotation essary syntax and semantics.
score on the left of the screen. A cursor moves up the In April 2004, the Dance Notation Bureau and the Ohio
score as the animation progresses. The user can drag State University sponsored a workshop where 29 experts
the cursor on the score or on the timeline below the ani- in Labanotation, dance, human movement, and technol-
mation to move backwards and forwards through the ogy met to discuss adding intelligence to programs that
animation sequence. compose, edit, and animate Labanotation. This would
add understanding of a symbol’s attributes, such as which
Future body part is moving, timing, and so on, and enable new
This human animation software currently translates advances in the use of technology and notation. (See the
numerous types of movement including supports, ges- meeting report at http://dancenotation.org/DNB/
tures, turns, circular paths, and contracted limbs. Future news/report.html.) The consensus at the workshop was
versions will be extended to handle all aspects of a score that there is a need for an interlingua—that is, an unam-
and in addition, LabanDancer will be written into a file biguous representation of movement that can map to dif-
that can be read into DanceForms or other more gener- ferent notation systems and editors, animations, and
al programs for human figure animation. This file can motion-captured dance data.
then be edited and combined in a variety of ways with Like all efforts to develop international standards, this
other sequences from animation, motion capture, or is a challenging problem. Following the development of
LabanDancer translations of other Labanotation scores. MusicXML to provide structure for musical scores, there
A more complete description of LabanDancer can be has been some initial discussion of a possible LabanXML
found elsewhere.4 as proposed by Minako Nakamura and her colleagues
in Japan5 or a more comprehensive DanceXML as pro-
Current research and future posed by Simon Fraser University graduate student
developments Jonathan Hatol. It’s easy to see how XML can help cap-
There are many things that can be done to enhance ture the high-level structure of Labanotation, but it’s
and extend the computer technology used to animate more difficult to capture the more detailed context-
and visualize dance, plan choreography, edit and ani- dependent structure.
mate notation, and enhance performance. However, a When a usable ontology-based representation is devel-
recurring issue in many of these developments is the oped, it will then be necessary to create interfaces to the
need for a unique, unambiguous way to represent existing representations used in notation editors in sys-

IEEE Computer Graphics and Applications 11

Applications

Computer Graphics and Interactive suspended from a grid and spiral in response to
Technologies in Live Performance visitors. Vision-sensing-software eyes let the
There are many examples of the use of all forms computers localize the visitors’ positions in space.
of multimedia in conjunction with live dance Different visual sequences are projected on each of
performances. In some cases a preplanned the screens. The visuals, which experiment with
animation sequence or digital video with body movement, have a unique digital look and
appropriate sound is played as a backdrop and live feel that the artists created in the video shooting
dancers interact with the preprogrammed display. process. One image captures a person slowly
A more technically challenging system is required jumping and falling and other images show bodies
when it senses the live dancers’ movements and in mud and water—Figure A gives a sense of this
modifies the imagery and/or sound. Another complex environment.
situation arises when digital video, possibly with Another exploratory project is Whispers
immersive or stereo display, links two distant (http://whisper.surrey.sfu.ca/index2.html) by
locations for rehearsal and/or performance. All Thecla Schiphorst and others, which involves
kinds of other variation are possible. physiological sensors, small wearable computing
One ambitious example is the Trajets devices, wireless computer communication, and
(http://www.trajets.net/) interactive installation handheld technologies embedded in evocative
codirected by Susan Kozel and Gretchen Schiller, and playful garments worn by the participants. It
with development by Rob Lovell and Scott Wilson. maps physiological data onto linked and
Here, 12 trapezoid-shaped, motorized screens are networked devices worn on or close to the skin
and in garments. In other words, it involves
collecting data from the bodies of participants and

Courtesy Roth-Kimura Architects, Holualoa, HI

communicating the participants’ responses to the
system and other participants through
visualization and sonification techniques.
One venue where much of the latest work is
displayed is the biannual Monaco Dance Forum
(http://www.monacodanceforum.com). The
forum is an international dance meeting and
festival that offers dance performances,
multimedia installations, exhibitions, video dance
screenings, and symposiums.
Another source for work up to 2000 is the
A View of the Trajets interactive installation. Digital Performance Archive (http://dpa.ntu.ac.uk/
dpa_site/).

tems for animating dance and in motion capture systems. Kinematics Techniques for Anthropomorphic Limbs,”
The great advantage of this new representation will be Graphical Models, vol. 62, no. 5, 2000, pp. 353-388;
that it will define all movement explicitly and not implic- http://hms.upenn.edu/software/ik/ik.html.
itly, as with present systems. It will be possible to search 3. M. Van de Panne, “From Footprints to Animation,” Com-
easily for movement patterns and to use this power in puter Graphics Forum, vol. 16, no. 4, 1997, pp. 211-223.
choreography, as well as many aspects of dance schol- 4. L. Wilke et al., “Animating the Dance Archives,” Proc. 4th Int’l
arship. This development can also be useful in other Symp. Virtual Reality, Archaeology and Intelligent Cultural
applications that require the analysis of human move- Heritage (VAST), Eurographics Assoc., 2003, pp. 91-99.
ment such as animation for games or the representation 5. M. Nakamura, “Text Representation of Labanotation Data
of motion-captured human movement data used in the for Computer Based Motion Analysis,” presented at the
clinical diagnosis of locomotor problems. ■ World Dance Assoc./Int’l Council of Kinetography Laban/
Congress on Research in Dance Int’l Conf., 2004; http://
www.imb.is.ritsumei.ac.jp/~hachi/hachi_e.html.

References Readers may contact Tom Calvert at [email protected].

1. M. Cunningham, Changes/Notes on Choreography, F. Starr,
ed., Something Else Press, 1968. Readers may contact Mike Potel at potel@
2. D. Tolani, A. Goswami, and N. Badler, “Real-Time Inverse wildcrest.com.

12 March/April 2005