History of HCI

Key systems, people and ideas

Matthias RAUTERBERG
Eindhoven University of Technology (TU/e)
The Netherlands
[TIP: in presentation mode links to additional sources are active]
History of Computer Technology

Digital computer grounded in ideas from

1700’s & 1800’s

Computer technology became available in

the 1940’s and 1950’s

see further: History of Computing

History of HCI
© M. Rauterberg, TU/e 2
Konrad Zuse (1910-1995)
In 1936 Konrad Zuse, absolutely set apart from the academic world, started
constructing an automatic machine to solve calculation problems for
designing plane wings; these analyses forcing him to long and repetitive
calculations. In his living-room and with the mere aid of few tools, he first
produced a binary mechanical memory, to which he soon connected a
mechanical calculation unit as well as a programming unit controlled from old
movie films punched by hand. He called this model V1 (Versuchsmodell 1),
but subsequently changed this name into Z1 in order not to confuse it with
the flying bombs having the same name! Having become aware of the poor
liability and slowness of this machine, in 1939, Zuse prepared a second one,
called Z2, characterized from a still mechanical memory but with a relay-
operated electromechanical calculating unit.
In the following years, Zuse accomplished a real electromechanical working computer, Z3, which was submitted in 1941 to
an audience of engineers and scientists, raising great interest. Not yet satisfied, living in a Berlin continuously bombed and in
which it was difficult to find even food, Zuse constructed Z4 with a mechanical memory (relays were now unfindable): the
machine was ended in 1944. Zuse arrived in protecting Z4 from the destructions of war and from the hands of Allies, hiding it
in a cellar in the small Bavarian village of Hindelang. Once tranquillity returned again, Zuse transferred in to the Swiss
Federal Institute of Technology (ETH), in Zurich, were it remained working for 15 years. Up to 1951, this machine remained
the only working computer in continental Europe.

The history of Zuse is emblematic for at least five reasons: 1) his contribution was completely original, as he was very
isolated from the rest of the world but even from the German research activity; 2) the fact that he has conceived a binary
representation of figures which is that adopted from all the modern computers; 3) the fact that he has independently
achieved an architecture which was already suggested by Babbage; the invention of the first programming language
(Plankalkul, 1943-45); 5) the extremely practical and simple way of facing the problem: the estimated cost of Z2 is 6.500 US
$, only.

© M. Rauterberg, TU/e 3
Z3 (1941)
Zuse’s Z3 was the world’s first reliable working machine for very complicated
arithmetic calculations, which was freely programmable and was based on
a binary floating point number and switching system.

Konrad Zuse in front of his reconstructed Z3

© M. Rauterberg, TU/e 4
Eniac (1943)
– A general view of the ENIAC, the first all electronic numerical integrator
and computer in USA.

From IBM Archives.

© M. Rauterberg, TU/e 5
Mark I (1944)
•The Mark I paper tape readers.

From Harvard University Cruft Photo Laboratory.

© M. Rauterberg, TU/e 6
von Neuman Architecture (1946)

Instructions and data are stored in the same memory

for which there is a single link (the von Neumann
bottleneck) to the CPU which decodes and executes
instructions.
The CPU can have multiple functional units.
The memory access can be enhanced by use of caches
Johann (John) von Neumann
made from faster memory to allow greater bandwidth (1903-1957)
and lower latency.

J. Presper Eckert Jr. and John Mauchly were the first to develop the von Neuman architecture. John von
Neumann wrote "First Draft of a Report to the EDVAC" describing the ideas of a stored memory computer. The
complicated story is described in the wonder history of computers "Engines of the Mind" by Joel Shurkin.

© M. Rauterberg, TU/e 7
Princeton Architecture (1946)

A course on computer design based on

the von Neuman concept was run by the
Moore School in 1946, and was attended
by British and American professionals.
Also in 1946, von Neuman, Herman
Goldstine and Arthur Burks published a
comprehensive report of their work at
Princeton's Institute of Advanced Study
Electronic Computer Project where they
had established themselves following
their involvement with the Moore School.
The report detailed the operation and
architecture of their work on digital
computers, and has been described as the
blue print for ‘modern' digital computing.

© M. Rauterberg, TU/e 8
Mainframe Computers

Stretch (1961)
A close-up of the Stretch technical control panel

IBM SSEC (1948)

From IBM Archives.

© M. Rauterberg, TU/e 9
Enabling Technology

1904 Sir John Ambrose 1947 the first Transistor was

Fleming invents the 1958 Jack Kilby invented the
invented by Bardeen, Brattain integrated circuit at Texas Instruments
vacuum tube and diode. and Shockley in the Bell Labs and got the Noble Prize. Comprised of
in the USA. only a transistor and other components
on a slice of germanium, Kilby's
invention, 7/16-by-1/16-inches in size,
revolutionized the electronics
industry.
Intel's 4004 IC chip, generally acknowledged as the world's first "microcomputer on a chip," was originally
designed in 1969-70 for the Busicom (Japan) 141-PF desktop calculator.

© M. Rauterberg, TU/e 10
Historical overview

1945 Memex

1969 Flex
1973 Alto
1974 Bravo

1974 IBM portable computer

1983 Apple Lisa
1984 Apple Macintosh
1987 MicroSoft Windows
© M. Rauterberg, TU/e 11
Vannevar Bush (1890-1974)

“As We May Think” - 1945

Atlantic Monthly

“…publication has been extended

far beyond our present ability to
make real use of the record.”

© M. Rauterberg, TU/e 12
Vannevar Bush (1945)

Postulated Memex device

Can store all records/articles/communications
Large memory
Items retrieved by indexing, keywords, cross
references
Can make a trail of links through material
etc.
Envisioned as microfilm, not computer

© M. Rauterberg, TU/e 13
Memex design sketch (1945)

© M. Rauterberg, TU/e 14
J.R. Licklider (1915-1990)

1960 - Postulated “man-computer symbiosis”

Couple human brains and computing machines
tightly to revolutionize information handling
“The hope is that, in not too many years, human brains and
computing machines will be coupled together very tightly and that
the resulting partnership will think as no human brain has ever
thought and process data in a way not approached by the
information-handling machines we know today.”

An MIT psychoacoustrician named J.C.R. Licklider took and immediate and intense interest in computer after Clark demonstrated
the TX-0 to him. Licklider applied his background in psychology to research how people interacted with computers, and he
became known as an expert in human-computer interaction. People at ARPA took notice and offered Licklider the job of director
for their new Information Processing Techniques Office (IPTO). He accepted the position as the founding director and continued
his research in human-computer interaction.

© M. Rauterberg, TU/e 15
Vision/Goals (1945-1995)

Immediate Intermediate Long-term

•Time sharing •Combined speech •Natural language
•Electronic I/O recognition, understanding
•Interactive, real- character •Speech recognition
time system recognition, light- of arbitrary users
•Large scale pen editing •Heuristic programming
information
storage and
retrieval

© M. Rauterberg, TU/e 16
Mid 1960’s

Computers too expensive for individuals

-> timesharing
– increased accessibility
– interactive systems, not jobs
– text processing, editing
– email, shared file system

© M. Rauterberg, TU/e 17
DEC PDP-1 (1961)

As the world's first commercial

interactive computer, the PDP-1
was used by its purchasers to
pioneer timesharing systems,
making it possible for smaller
businesses and laboratories to
have access to much more
computing power than ever
before.

© DEC Inc.

© M. Rauterberg, TU/e 18
Ivan Sutherland (1938-)

SketchPad: 1963 PhD thesis at MIT

– Hierarchy - pictures & subpictures
– Master picture with instances
– Constraints
– Icons
– Copying
– Light pen as input device
– Recursive operations

© M. Rauterberg, TU/e 19
Sketchpad (1963)

Input device: Light pen used

on cathode ray tube.

Graphical objects could be

drawn and modified through
constraints.
Object oriented modell.
Ivan Sutherland using the console of the TX-2 at MIT
Copy and paste.

© MIT Lincoln Lab

© M. Rauterberg, TU/e 20
Douglas C. Engelbart (1925 - )
 Engelbart invented the mouse at
Stanford Research Labs in 1964.
 Landmark system/demo:
– hierarchical hypertext,
multimedia, mouse,
high-resolution display, windows,
shared files, electronic messaging,
CSCW, teleconferencing, …
– Augment/NLS system [NLS: oN Line
System]

© M. Rauterberg, TU/e 21
The First Mouse (1964)

Knee control

Douglas Engelbart

Years before personal computers and desktop information processing

became commonplace or even practicable, Douglas Engelbart had invented
a number of interactive, user-friendly information access systems that we
take for granted today: the computer mouse, windows, shared-screen
teleconferencing, hypermedia, groupware, and more.

© M. Rauterberg, TU/e 22
Augment/NLS (1968)

Augment/NLS Features:

2-dimensional display
text editting, by two
persons from different
consoles, at the same
time.

Links.

Video-conferencing.

Mouse.

© M. Rauterberg, TU/e 23
Alan C. Kay (1940-)

Dynabook - Notebook sized computer loaded

with multimedia and can store everything
Personal Computing
Desktop Interface
The FLEX software

© M. Rauterberg, TU/e 24
FLEX & Dynabook (1969)

Computer should function like a

living organism.

Kay‘s Ph.D was about FLEX, an

early object orientated language.

First idea of a book-sized

Computer.
Use of graphic rather than text.

Alan Kay developed the Dynabook

at Xerox PARC.
© Xerox PARC
© M. Rauterberg, TU/e 25
Theodor (Ted) H. Nelson (1937-)

 Computers can help people, not just

business
 Coined term “hypertext”

Ted Nelson originally invented the word "hypertext" for "non-

sequential writing". His long-standing interest in all things related
to HT became the Xanadu project. The Xanadu Operating
Company was owned for a while by Autodesk, but later dropped.

© M. Rauterberg, TU/e 26
Nicholas Negroponte (ca 1938-)

 MIT machine architecture & AI

group (1969-1980s)
 Ideas:
– wall-sized displays, video
disks
– AI in interfaces (agents),
speech recognition,
multimedia with hypertext
© MIT MediaLab, Boston

© M. Rauterberg, TU/e 27
Personal Computers (PC)

Late ‘70’s
Apple II
Z-80 CP/M
IBM PC
Text and command based
Word processing
Spreadsheets
© M. Rauterberg, TU/e 28
Input/output devices
Input Output

Early days connecting wires lights on display

paper tape & punch cards paper
keyboard teletype

Past keyboard scrolling glass teletype

+ cursor keys character terminal
+ mouse bit-mapped screen
+ microphone audio

Today data gloves + suits head-mounted displays

computer jewelry ubiquitous computing
natural language autonomous agents

© M. Rauterberg, TU/e 29
IBM (1974)

Mark-8
The Mark-8 was an Intel 8008 based
machine with 256 bytes RAM. It was
introduced in July 1974, and 1000-
2000 were produced. It was the first
portable computer to really be
marketed, and had no ROM. The
market value of a Mark-8 was a round
$12,000.
The machine pictured right was the
precursor to the IBM 5100 machine. It
was introduced in 1975, and was very
costly. It was IBM's first entry into the
microcomputer market.
© IBM

© M. Rauterberg, TU/e 30
IBM (1975)

IBM 5100
IBM 5100, introduced in September
1975, was IBM's first portable
computer.
The 5100 was just one of several
portable computers IBM made before
the Personal Computer (PC).
The 5100 model was followed by the
5110, the 5120, the Datamaster, and
then finally the 5150 PC.

From http://www.blinkenlights.com/pc.shtml © IBM

© M. Rauterberg, TU/e 31
IBM (1981)

IBM PC
Product shot of IBM Personal Computer (5150),
introduced in 1981;
Features: monitor, keyboard, and pin-feed printer; b/w.
The operating system (OS) was by Microsoft, who
licensed it to IBM as PC-DOS.
Although not necessarily the best machine by
technological standards, IBM's expertise and the fact
that the IBM PC actually looks and feels like a
professional computer system made the IBM PC and the
numerous PC clones extremely popular. They have
evolved into today's so-called Wintel (Windows + Intel)
computer systems, used world-wide. © IBM
From http://www.blinkenlights.com/pc.shtml

© M. Rauterberg, TU/e 32
DOS (history)

1980, April: Tim Patterson begins writing an operating system for use with Seattle Computer Products' 8086-based
computer. Seattle Computer Products decides to make their own disk operating system (DOS), due to delays by Digital
Research in releasing a CP/M-86 OS.
1980, August: QDOS 0.10 (Quick and Dirty OS) is shipped by Seattle Computer Products. Even though it had been created
in only two man-months, the DOS worked surprisingly well.
1980, September: Tim Patterson shows Microsoft his 86-DOS, written for the 8086 chip.
1980, October: Microsoft's Paul Allen contacts Seattle Computer Products' Tim Patterson, asking for the rights to sell
SCP's DOS to an unnamed client (IBM). Microsoft pays less than US$100,000 for the right.
1980, December: Seattle Computer Products renames QDOS to 86-DOS, releasing it as version 0.3. Microsoft then bought
non-exclusive rights to market 86-DOS.

1981, February: MS-DOS 1.0 runs for the first time on IBM's prototype microcomputer.
1981, July: Microsoft buys all rights to DOS from Seattle Computer Products, and the name MS-DOS is adopted.
1981, August: IBM announces the IBM 5150 PC Personal Computer, featuring a 4.77-MHz Intel 8088 CPU, 64KB RAM,
40KB ROM, one 5.25-inch floppy drive, and PC-DOS 1.0 (Microsoft's MS-DOS), for US$3000.

1982, May: Microsoft releases MS-DOS 1.1 to IBM, for the IBM PC. It supports 320KB double-sided floppy disk drives.
Microsoft also releases MS-DOS 1.25, similar to 1.1 but for IBM-compatible computers.

1988, June: Microsoft releases MS-DOS 4.0, including a graphical/mouse interface.

© M. Rauterberg, TU/e 33
Bill Gates (1955-)

William (Bill) H. Gates is chairman and chief software architect of

Microsoft Corporation, the worldwide leader in software, services and
Internet technologies for personal and business computing. Microsoft
had revenues of $25.3 billion for the fiscal year ending June 2001, and
employs more than 40,000 people in 60 countries. In his junior year,
Gates left Harvard to devote his energies to Microsoft, a company he
had begun in 1975 with his childhood friend Paul Allen. Guided by a
belief that the computer would be a valuable tool on every office
desktop and in every home, they began developing software for
personal computers. Gates' foresight and his vision for personal
computing have been central to the success of Microsoft and the
software industry.
Under Gates' leadership, Microsoft's mission has been to continually advance and improve
software technology, and to make it easier, more cost-effective and more enjoyable for people to
use computers. The company is committed to a long-term view, reflected in its investment of more
than $4 billion on research and development in the current fiscal year 2001.

© M. Rauterberg, TU/e 34
© source http://www.microsoft.com/billgates/bio.asp
MS DOS (1981)

Microsoft DOS (Disk Operating

System) is a command line user
interface.
Microsoft releases MS-DOS 1.0
to IBM, for the original IBM PC
in 1981.
In 1982 MS-DOS 1.1 supports
320KB double-sided floppy disk
drives.
Microsoft also releases MS-
DOS 1.25, similar to 1.1 but for
IBM-compatible computers with
720 KB floppy disk drives.

© Microsoft Inc.

© M. Rauterberg, TU/e 35
PCs with GUIs

Xerox PARC - mid 1970’s

– invention of the first PC: ALTO
– Local processor, Bitmap display, Mouse
– Precursor to modern GUI
– LAN - Ethernet

© M. Rauterberg, TU/e 36
 Xerox Alto (precursor to the Star)

Alto applications:
Bravo WYSIWYG text editor.
BravoX Mesa implementation of
Bravo, an ancestor of Microsoft Word.
Laurel Electronic mail program.
Neptune Disk file manipulation
program, sort of like sweep.
Works like the Font DA mover
program on a Macintosh.
Press Document printing program.
Sil Drawing program.

From Xerox Alto Archive

© M. Rauterberg, TU/e 37
Xerox Star - ‘81

First commercial PC designed for “business

professionals”
Desktop metaphor, pointing, WYSIWYG

First system based on usability engineering

© M. Rauterberg, TU/e 38
Xerox Star (1981)

© M. Rauterberg, TU/e 39
Xerox Star (GUI)

© M. Rauterberg, TU/e 40
Xerox Star (history)

Commercial flop
– $15k cost
– closed architecture
– lacking key functionality (spreadsheet)

© M. Rauterberg, TU/e 41
Apple II (1977)

Built in 1977, the Apple II was based on Wozniak's

Apple I design, but with several additions. The first
was the design of a plastic case--a rarity at the time--
which was painted beige. The second was the ability
to display color graphics--a holy grail in the industry.
The Apple II also included a larger ROM, more
expandable RAM (4K to start), and 8 expansion slots. Steve Wozniak (1950-)
It had integer BASIC hard-coded on the ROM for
easier programming, and included two game paddles
© Apple Inc.
and a demo cassette for $1,298. In early 1978 Apple
also released a disk drive for the machine, one of the
most inexpensive available. The Apple II remained on 1979, February
the Apple product list until 1980. Apple Computer releases DOS 3.2.
1979, July
Apple Computer releases DOS 3.2.1

© M. Rauterberg, TU/e 42
Apple Lisa - ‘82

Based on ideas of Star

More personal rather than office tool

Still expensive!

Conceptual success, but

commercial failure
Steve Jobs (1955-)
co-founder Apple Computer Corporation
© M. Rauterberg, TU/e 43
Apple Lisa (1983)

Named for one of its designer's daughters, the Lisa

(pictured left) was supposed to be the Next Big
Thing. It was the first personal computer to use a
Graphical User Interface (GUI). Aimed mainly at
large businesses, Apple said the Lisa would
increase productivity by making computers easier
to work with. The Lisa had a Motorola 68000
Processor running at 5 Mhz, 1 MB of RAM two
5.25" 871k floppy drives, an external 5 MB hard
drive, and a built in 12" 720 x 360 monochrome
© Apple Inc.
monitor.
http://www.apple-history.com/lisa.html

© M. Rauterberg, TU/e 44
Apple Lisa (applications)

LisaWrite: word processor

LisaCalc: spread sheet
LisaGraph: charts
LisaList: an outline builder, idea manager
LisaProject: project scheduler
LisaDraw: drawing program
(predecessor to Mac Draw)
LisaTerminal: modem communications
software. © Apple Inc.

© M. Rauterberg, TU/e 45
Apple Macintosh - ‘84

Aggressive pricing - $2500

Not trailblazer, smart copier
Good interface guidelines
3rd party applications
High quality graphics and laser printer

© M. Rauterberg, TU/e 46
Apple Macintosh (1984)

Released in January of 1984, the

Macintosh was the first affordable
computer to include a Graphical User
Interface (GUI). It was built around
the new Motorola 68000 chip, which
© Apple Inc. was significantly faster than previous
processors, running at 8 MHz. The
Mac came in a small beige case with
a black and white monitor built in. It
came with a keyboard and mouse,
and had a floppy drive that took 400k
The early Mac team members (1979) consisted of Jeff Raskin, Brian
Howard, Marc LeBrun, Burrell Smith, Joanna Hoffman and Bud
3.5" disks--the first personal computer
Tribble. to do so.

© M. Rauterberg, TU/e 47
Apple Macintosh (GUI)

© Apple Inc.

© M. Rauterberg, TU/e 48
Apple Macintosh (history)

Problem of the Mac: Only 128k main memory

Well written applications: MacWrite and MacDraw

Mac 512k, Mac512ke and Mac Plus were

introduced to save the Mac

New applications: Pagemaker, Word Excel

© M. Rauterberg, TU/e 49
MS Windows (1987)

Release of the IBM PC AT in Windows planed 1983 was

August 1984 running at 6MHz released on August 11, 1987

Windows 1.01
was a large
disappointment!

© Microsoft Inc.

© M. Rauterberg, TU/e 50
Microsoft Windows (history)

Steve Jobs from Apple Inc. complained about the

stolen Mac OS’s interface design.

Bill Gates from Microsoft Inc. replied:

“Hey Steve, just because you broke into Xerox’s
house before I did and took the TV doesn’t mean
I can’t go in later and take the stereo.”

Microsoft Windows 2.03 was released January 1988.

Microsoft Windows 3.1 was released in 1992.

© M. Rauterberg, TU/e 51
Oberon at ETH (1985)

Oberon is the name of a modern integrated software

environment. It is a single-user, multi-tasking system that runs on
bare hardware or on top of a host operating system. Oberon is
also the name of a programming language in the Pascal/Modula
tradition. The Oberon project was launched in 1985 by Niklaus
Wirth and Jürg Gutknecht at the ETH in Zurich. Although the
Niklaus Wirth (1934-)
project was originally targeted towards in-house hardware, the
language and system have now been ported to many computer
platforms. Oberon is the first worldwide modeless software
system: completely modeless graphical user interface with
support for graphical primitives; both overlapping and tiled
windowing systems supported concurrently; possibility of
Jürg Gutknecht
configuring the working environment.

© M. Rauterberg, TU/e 52
Ben Shneiderman (1947-)
Dr. Shneiderman is the author of ‘Software Psychology: Human Factors in
Computer and Information Systems’ (1980) in which he coined the term
direct manipulation. Later he wrote the influential text book ‘Designing
the User Interface: Strategies for Effective Human-Computer Interaction’
(1987, third edition 1998).
Ben Shneiderman has written over 200 articles and published several
books, including Elements of FORTRAN Style: Techniques for Effective
Programming (with Charles Kreitzberg, 1972); and Hypertext Hands-On!
An Introduction to a New Way of Organizing and Accessing Information
(with Greg Kearsley, 1989). He has also edited numerous articles and
several books, including Directions in Human/Computer Interaction
(1982) and Sparks of Innovation in Human-Computer Interaction (1993).

Ben Shneiderman was a Professor in the Department of Computer Science, Founding Director (1983-
2000) of the Human-Computer Interaction Laboratory, and Member of the Institutes for Advanced
Computer Studies and for Systems Research, all at the University of Maryland at College Park.

© M. Rauterberg, TU/e 53
Historical Overview (1945-1995)

[source: Brad A. Myers (1998). A brief history of human-computer interaction technology. Interactions, vol 5(2), pp. 44-54]

© M. Rauterberg, TU/e 54
 Mobile Phone (1954)
Growing up in Chicago, Martin Cooper earned a degree in electrical engineering at the
Illinois Institute of Technology. Later hired by Motorola in 1954, he lead a group of
research team to develop the first ever portable phone, the Motorola Dyna Tac, which
stands for Dynamic Adaptive Total Coverage. Weighing 1089 gram, the first
commercially viable version of the Dyna Tac was released in 1983. It measures 9 x 5 x
1.75 inches in size with 30 circuit boards. Unlike handsets today, there was no display
available and the only feature was call, dial and listen (what else would you expect?).
The heavy batteries can only withstand 35 minutes of talk time and need a long 10 hour
recharge.
Martin Cooper
Some important years in the mobile history,
1955 introducing the worlds first whole automatic mobile-phone system.
1972 A global system is presented. Covers all the oceans of the world.
1978 introducing the worlds first person searching-system with a number-display.
1981 The world’s first automatic and boundless mobile-phone system.
1986 First time when you can transfer computer services via a mobile system.
Dyna Tac 1988 The pocket-phone is introduced.

© M. Rauterberg, TU/e 55
What is Virtual Reality (VR)?
In 1989, Jaron Lanier, CEO of VPL, coined the term virtual reality to bring all of the
virtual projects under a single rubric. The term therefore typically refers to three-
dimensional realities implemented with stereo viewing goggles and reality gloves.

Myron Krueger (1991):

….The term (virtual worlds) typically refers to three-dimensional realities
implemented with stereo viewing goggles and reality gloves.

George Coates (1992):

Virtual Reality is electronic simulations of environments experienced via
head mounted eye goggles and wired clothing enabling the end user to
Myron Krueger
interact in realistic three-dimensional situations.

P. Greenbaum (1992):
Virtual Reality is an alternate world filled with computer-generated images that respond
to human movements. These simulated environments are usually visited with the aid of
an expensive data suit which features stereophonic video goggles and fiber-optic gloves.

© M. Rauterberg, TU/e 56
Dimensions to define VR

Vividness
(richness of an environments representation)
• breadth (visibility, audibility, touch, smell)
• depth (quality, fidelity)

Interactivity
(extend to which a user can modify form and content of a mediated
environment)
• speed (update rates, time lag)
• mapping (text, speech, gestures, gaze, complex behavior patterns)

© M. Rauterberg, TU/e 57
Classification of VR and other Media

From Jonathan Steuer

© M. Rauterberg, TU/e 58
History of VR (technological milestones)

1956 Sensorama (Morton Heilig)

3D visuals, vibration, stereo sound, wind, smell, little interaction.
1961 Headsight System (Philco Corp.)
Head Mounted Display (HMD), head tracking, remote video camera,
telepresence.
1965 The Ultimate Display (Ivan Sutherland)
Stereoscopic HMD, computer generated images, tracking, visually
coupled system.
1967 Grope (University of North Carolina)
6 degree of freedom force feedback.
1977 The Sayre Glove (Sandin, Sayre, DeFanti Univ. Illinois)
Gesture recognition.
1987 Virtual Cockpit (British Aerospace)
head and hand tracking, eye tracking, 3d visuals, 3D audio,
speech recognition, vibro tactile feedback.

© M. Rauterberg, TU/e 59
Morton Heilig
In the late '50s, a quiet man named Morton Heilig began
designing the first multisensory virtual experiences. He
developed something called the Sensorama. Resembling
one of today's arcade machines, the Sensorama combined
projected film, audio, vibration, wind, and even
prepackaged odors, all designed to make the users feel as
if they were actually in the film rather than simply
watching it. Since real-time computer graphics were many
years away, the entire experience was prerecorded, and
played back for the user.
Although he was a gifted and visionary inventor, Heilig
was less successful as a businessman. He was unable to get
funding for his Sensorama machines, and they were never
manufactured. Fortunately, he didn't give up there;
Heilig had an idea that would later prove to be the basis for
an entire industry: the first Head-Mounted Display
(HMD), which he patented in 1962.

© M. Rauterberg, TU/e 60
Sensorama (1956)

Sensorama is a simulator that gives one

person at a time an illusion of reality. It is a
semi-portable automatic machine that can be
plugged in anywhere electricity is available.
The illusion of reality is achieved by
providing the viewer with a wide range of
sensory information. All of this information is
perfectly synchronized with the picture
(aromas, wind, and vibrations change
instantly). All the control information is on
one piece of film. Sensorama is completely
automatic. The viewer activates it by
depositing a coin or pushing a button
(depending on application).

© M. Rauterberg, TU/e 61
Headsight System (1961)

(© nVision)

first HMD (1961) Sutherland & Sproull (Harvard, 1967) Datavisor 80

© M. Rauterberg, TU/e 62
Ultimate Display (1965)

Ivan Sutherland is a pioneer in the field of computer

graphics and in 1965 he described 'The Ultimate
Display', which included interactive graphics and
force-feedback devices.
In 1968, he described a prototype virtual reality
system in his paper 'A head-mounted three-
dimensional display'.
But it was a team at the NASA Ames Research
Center who really opened up the possibilities of
virtual reality worlds with their Virtual Interface
Environmental Workstation (VIEW), developed
during the 1980s as a training system for future
astronauts.
© NASA Ames Research Center

© M. Rauterberg, TU/e 63
Grope (1967)

Haptic Display Grope III

The Force-Feedback Project, which began in 1967, first
focused on the development of a system to support
scientific visualization in the area of molecular docking,
the Docker application. This application provides graphic
(wire-frame) representations of molecules and their inter-
atomic forces to allow a user to adjust the relative position
and orientation of molecules while searching for minimum
energy binding sites.
A series of systems have been developed, evolving from a
2-D system, through a 3-D system and a 6-D system for a
simple docking task, to a full 6-D molecular docking
system called GROPE-III. These later systems have
employed a modified Model E-3 Argonne Remote
Manipulator (ARM).

© Fred Brooks, University of North Carolina

© M. Rauterberg, TU/e 64
Sayre Glove (1977)

Early contributions to computer graphics included performances with real-time

graphics accompanied by music and the use of its hardware and software for
creating the computer animation of the Death Star "schematic" for the first Star
Wars.
In 1976, based on an idea by colleague Rich Sayre, DeFanti and Sandin developed
an inexpensive, lightweight glove to monitor hand movements; the Sayre Glove
provided an effective method for multidimensional control, such as mimicking a
set of sliders.
Projects in the 1970s through mid-1980s centered on video game technology, real-
time computer animation on microcomputers, and interactive multimedia
installations.

© University of Illinois at Chicago

© M. Rauterberg, TU/e 65
Digital Data Entry Glove (1985)

Digital Data Entry Glove, T. G. Zimmerman & Y. Harvill

 first glove-like device (cloth) onto which numerous touch, bend,
and inertial sensors were sewn.

 measured finger flexure, hand-orientation and wrist-position,

and had tactile sensors at fingertips.

 orientation of hand tracked by video camera; required clear

line-of-sight observation for the glove to function.

 designed as alternative to keyboard; matched recognized

gestures/hand orientations to specific characters, specifically to
recognize the Single Hand Manual Alphabet for the American
Deaf; circuitry hard-wired to recognize 80 unique combinations
of sensor readings to output a subset of the 96 printable ASCII
characters; a tool to “finger-spell” words.

 finger flex sensors, tactile sensors at the fingertips, orientation

sensing and wrist-positioning sensors; positions of sensors were
changeable.

 US Patent 4,542,291: Zimmerman & Harvill, Optical Flex

Sensor, September 17, 1985 [ACM CHI paper 1987]
© VPL Research, Inc.
© M. Rauterberg, TU/e 66
VIEW (ca.1985)
The VIEW configuration included a
head-mounted display, head and hand
tracking, speech recognition, three-
dimensional audio output, and a tracked
and instrumented glove.
The glove was the interface through
which the user could interact with the
virtual world. A graphical representation
of the glove moved around the virtual
world in response to the user's hand
movements. The glove had fibreoptics
embedded in it and these detected
changes in finger positions, while a
separate motion sensor detected the
position of the hand.
The computer recalculated the
coordinates of the glove's image based
its movements © NASA Aerospace Lab.

© M. Rauterberg, TU/e 67
Virtual Cockpit (1987)

© British Aerospace Lab.

© M. Rauterberg, TU/e 68
Input Devices (overview)

Sensor Devices
1. Spatial Position/Orientation Sensors
• 2DOF (Mouse)
• 3DOF (Microscribe, FreeD Joystick)
• 6DOF (Polhemus Fastrack)
2. Directional Force Sensors
• 5 DOF (Spacemouse)
• 2 DOF (Joystick)
3. Gesture Recognition
• Data Gloves
4. Eye Tracking
5. Speech Recognition Systems

© M. Rauterberg, TU/e 69
Input Devices (1)

Directional Force Sensors

SpaceMaster

© M. Rauterberg, TU/e 70
Input Devices (2)

Gesture Recognition

Dextrous Hand Master, Exos SUPERGLOVE, Nissho Cyberglove , 5th Dimension

© M. Rauterberg, TU/e 71
Input Devices (3)

Spatial Position/Orientation Sensors

Polhemus InsideTrack MicroScribe FreeD Joystick

(Magnetic Tracking) (Mechanical Tracking) (UltraSonic Tracking)

© M. Rauterberg, TU/e 72
 Input Devices (4)

Visual Haptic Workbench

The Visual Haptic Workbench consists of five hardware

components.
The dominant hand of the user experiences haptic feedback
from the PHANToM, and the subdominant hand navigates
through a menu interface via Pinch glove contact gestures.
Head tracking is done with a Polhemus Fastrak receiver
mounted on a pair of Stereographics CrystalEyes LCD shutter
glasses. The subdominant hand can also be tracked with a
separate receiver to facilitate more complex interaction
[see also http://haptic.mech.nwu.edu/intro/gallery/] paradigms. The audio subsystem gives the user additional
reinforcement cues to clarify
© M. Rauterberg, TU/e 73
Output Devices (1)

NASA HMD (ca 1985)

© M. Rauterberg, TU/e 74
Output Devices (2)

Four-sided CAVE (1992)

An Immersive VR Environment

© Electronic Visualization Laboratory

University of Illinois at Chicago

© M. Rauterberg, TU/e 75
Output Devices (3)

Six-sided CAVE (1998)

The KTH Six-sided CAVE built by TAN GmbH.

In October 1998 TAN finished the world's

first 6-sided TAN VR-CUBE™ for the Royal
Institute PDC/KTH in Stockholm/Sweden.
© KTH, Stockholm This type of VR-CUBE™ complete encloses
the user from all sides.

© M. Rauterberg, TU/e 76
Output Devices (4)

CyberSphere (1998)
The scientists Eyre and Eureka in VR-Systems UK
have been researching a CyberSphere, a device,
which consists of a large, translucent sphere
containing the user.
The images are distortion-corrected and then
projected on the surface of the sphere, allowing the
user a full 360 degree field of view.
It also allows the user to move around in the world,
by walking inside the ball, which will move in
response to the users movements.

© VR-Systems, United Kingdom

© M. Rauterberg, TU/e 77
Ubiquitous Computing (1991)
Ubiquitous computing is just now beginning. First were
mainframes, each shared by lots of people. Now we are in the
personal computing era, person and machine staring uneasily at
each other across the desktop. Next comes ubiquitous
computing, or the age of calm technology, when technology
recedes into the background of our lives.
Mark Weiser is the father of ubiquitous computing (1991).
[Mark Weiser, "The Computer for the Twenty-First Century”, Scientific American, Mark Weiser (1952-1999)

pp. 94-10, Sept. 1991]

What Ubiquitous Computing Isn't

Ubiquitous computing is roughly the opposite of virtual reality. Where virtual reality puts people inside a
computer-generated world, ubiquitous computing forces the computer to live out here in the world with
people. Virtual reality is primarily a horse power problem; ubiquitous computing is a very difficult
integration of human factors, computer science, engineering, and social sciences.

© M. Rauterberg, TU/e 78
Apple’s Newton (1992)
Apple Computer's Newton line of personal
digital assistants (PDA) began as CEO
John Scully's pet project in 1992. Since
then, seven different Newton models were
released before "iCEO" Steve Jobs came
back to Apple: in 1998 he quickly axed
the newly formed subsidiary responsible
for the Newton line, Newton, Inc.
A small, hand-held device with pen-input,
personal organizational functions, and
communication capabilities becomes the
first shipped Newton product by Apple Inc.
Although intelligent software is an
essential element, this product is viewed
as a smaller, cheaper, stripped-down
version of the initial Newton concept that
can be mass produced.
© Apple Computing Inc.

© M. Rauterberg, TU/e 79
PARCtab (1993)

The PARCtab is most easily operated

with two hands: one to hold the tab,
the other to use a passive stylus or a
finger to touch the screen.
But since office workers often seem to
have their hands full, we designed
the tab so that three mechanical
buttons fall beneath the fingers of
the same hand that holds the tab,
allowing one-handed use.
The device also includes a piezo-electric
speaker so that applications can
© Xerox PARC generate audio feedback

© M. Rauterberg, TU/e 80
Unistroke alphabet
Techniques for handwriting recognition
have improved in recent years, and
are used on some PDAs for text entry.
But they are still far from ideal since they
respond differently to the unique
writing characteristics of each
operator.
Xerox PARC have experimented on the
PARCTAB with Unistrokes, which
depart from the traditional approach
in that they require the user to learn
a new alphabet---one designed
specifically to make handwriting
easier to recognize
© Xerox PARC

© M. Rauterberg, TU/e 81
Half-QWERTY (1993)

Typing With One Hand Using Your

Two-handed Skills
Half-QWERTY is a one-handed
typing technique, designed to
facilitate the transfer of two-
handed typing skill to the one-
handed condition

© Matias, MacKenzie & Buxton, Toronto

Bill Buxton

© M. Rauterberg, TU/e 82
PalmPilot (1996)

The PalmPilot has a lot

functionality.
This device fits with its
pocket size into one
hand.
There is a communication
channel via IR to the PC.
Small, and a reasonable
price
© Palm Computing Inc.

© M. Rauterberg, TU/e 83
PalmPilot alphabet

Input similar to
“natural” alphabet
not user specific
minimize the user’s
learning and
adapting costs
© Palm Computing Inc.

© M. Rauterberg, TU/e 84
Newton MessagePad 2x00 (1997)

The final incarnation of the Newton OS came

packaged in the MessagePad 2100, an internet-
ready, heap-increased version of the MessagePad
2000. The 161.9Mhz StrongArm processor, which
Palm is just beginning to port PalmOS to in 2002, is
the true reason the Newton 2X00 line is still one of
the more functional PDAs of these days.

When the MP2K was first released in 1997, unlike

the PalmPilot, they were meant to work to laptop
functionality rather than act as a digitized datebook.
This means that you can browse the Web in full
grayscale (and perhaps Javascipt), number crunch
on spreadsheets, use graphing calculators,
wordsmith in a functional word processor, and so on.
Movies have been made to run on Newtons and
Mp3 syncing with iTunes has been implemented.

© Apple and Newton, Inc.

© M. Rauterberg, TU/e 85
Brian Shackel (ca 1920-)
Brian Shackel is Emeritus Professor of Ergonomics at Loughborough
University and the founder of the HUSAT Research Institute. He is
one of the pioneers in applying human factors and ergonomics to
computer systems. After graduating from Cambridge and following
military service, he set up EMI’s human factors group and worked
on many systems and product user interfaces. He joined
Loughborough University of Technology and set up HUSAT in 1970.
He has been an advocate international standards committees and
publishing widely. He is the founder and first chairman of the IFIP
Technical Committee “Human-Computer Interaction” in 1989.
The prestigious BRIAN SHACKEL AWARD is associated with each IFIP TC13 INTERACT
Conference, usually biennial, and is to recognise the most outstanding contribution in the form
of a refereed paper submitted to and delivered at the INTERACT conference.

“From early days (cf Licklider & Clark, 1962) the need for larger displays has been emphasised; but just when it seemed, in the late
1980s, that full page and larger displays would come with lower prices, the focus in the industry turned to portability and we moved
backward to smaller screens. While there was some improvement, larger screens (eg 21 inch CRT displays) are still not available at
an acceptable price; as long ago as 1977 Kay & Goldberg (1977) in their Dynabook concept specified a display the size of a full
paper page, but I know of no portable laptop, let alone notebook, which has an A4 page size screen.” (Brian Shackel, 2000)

© M. Rauterberg, TU/e 86
Tim Berners-Lee (1955-)
BORN June 8, 1955, in London
1976 Graduates from Queen's College, Oxford
1980 While at CERN, writes "Enquire"
1989 Proposes global hypertext project called "WorldWideWeb"
1991 The Web debuts on the Internet
1993 University of Illinois releases Mosaic browser
1994 Joins M.I.T. to direct the W3 consortium
Tim Berners-Lee is
considered to be the
1999 Today nearly 150 million people log on to the Internet via
founder of the
WWW World Wide Web.
“First of all, let's get clear the difference. The internet is a collection of computers, which was put together during the 1970's.
When I proposed the Web in 1989, the internet had been around for 15 years. You could use e-mail, you could store files on ftp
servers, and people could access them, but it was very complicated. The web was the step to make accessing a remote document
just one click. The internet spread really quite slowly. It started in research, moved into universities, and many people only heard
about it when the web became available as an easy way to use it. “ (T. Berners-Lee, 1999)

© M. Rauterberg, TU/e 87
World Wide Web (1990)
In Europe, researchers at CERN (the European Laboratory for Particle Physics) were struggling with their own computer networking
problems. Throughout the system people used different techniques, protocols, and equipment, making communication between
computers very complex. In 1980, Tim Berners-Lee, a consultant at CERN, wrote a program called "Enquire-Within-Upon-Everything,"
enabling links to be made between any point in the system. Nine years later Berners-Lee wrote "Information Management: A Proposal:"
Instead of standardizing the equipment or software, they created standards for data, and a universal addressing system. That way any
document on the Internet could be retrieved and viewed. In 1990, CERN was the largest Internet site in Europe. Over the next year or
two, the proposal was circulated and revised, resulting in an initial program being developed that was dubbed the World Wide Web. At
least one expert has called the Web a "side effect of CERN's scientific agenda." In 1992, the World Wide Web was demonstrated and
distributed, and browser software was released throughout and beyond CERN. That November there were about 26 reliable Web servers.

All you needed to use the Web was a browser. The early browsers were functional but not especially "user-friendly." A young
programmer at the National Center for Supercomputing Applications (NCSA) named Marc Andreesen created a new graphical Web
browser. This was pleasing to the eye and easy to use -- just point-and-click. Users didn't need to know any programming or even any
Internet addresses. It also made it fairly simple for users to add their own material to the Web. Andreesen and his coworkers called this
browser Mosaic, and released free versions for Windows and Macintosh in August of 1993. Interest in the Web -- especially commercial
interest -- exploded with the arrival of Mosaic. By October there were more than 200 Web servers, and at the end of 1993, Mosaic was
being downloaded from NCSA at a rate of 1,000 copies per day. By June 1994, there were 1,500 Web servers.

In July 1993, there were 1,776,000 hosts in 26,000 domains; by July 1996, there were 12,881,000 hosts in 488,000 domains. In July
1996, there were 3,054 Internet service providers and projections of Web user sessions rising to 15.79 billion in the year 2000.

© M. Rauterberg, TU/e 88
Historical Overview: Robots
1818 Mary Shelley, Frankenstein
1827 Joseph Atterly, A Voyage to the Moon
1863 Jules Verne, A Journey to the Center of the Earth
1865 Edward S. Ellis, The Steam Man of the Prairies
1870 Jules Verne, 20,000 Leagues Under the Sea
1895 H.G. Wells, The Time Machine
1920 Karel Capek, R.U.R. (first use of the word "robot")

1921 - The term "robot" was first used in a play called "R.U.R." or
"Rossum's Universal Robots" by the Czech writer Karel Capek.

1941 - Science fiction writer Isaac Asimov first used the word "robotics"
to describe the technology of robots and predicted the rise of a powerful robot industry.

1948 - "Cybernetics", an influence on artificial intelligence research was published by Norbert Wiener.

1956 - George Devol and Joseph Engelberger formed the world's first robot company.

1959 - Computer-assisted manufacturingg was demonstrated at the Servomechanisms Lab at MIT.

1961 - The first industrial robot was online in a General Motors automobile factory in New Jersey. It was called
UNIMATE.

1963 - The first artificial robotic arm to be controlled by a computer was designed. The Rancho Arm was designed as a
tool for the handicapped and it's six joints gave it the flexibility of a human arm

© M. Rauterberg, TU/e 89
UNIMATE (1961)

UNIMATE, the first industrial robot,

began work at General Motors. Obeying
step-by-step commands stored on a
magnetic drum, the 4,000-pound arm
sequenced and stacked hot pieces of die-
cast metal. The brainchild of Joe
Engelberger and George Devol,
UNIMATE originally automated the
manufacture of TV picture tubes.

© M. Rauterberg, TU/e 90
Wabot-1 (1973)

The WABOT-1 was the first fun-scale anthropomorphic robot

developed in the world at Tokyo's Waseda University under
Ichiro Kato.
It consisted of a limb-control system, a vision system and a
conversation system. The WABOT-1 was able to
communicate-with a person in Japanese and to measure Ichiro Kato
distances and directions to the objects using external receptors,
artificial ears and eyes, and an artificial mouth.
The WABOT-1 walked with his lower limbs and was able to
grip and transport objects with hands that used tactile-sensors.
It was estimated that the WABOT-1 has the mental faculty of a
one-and-half-year-old child. WABOT-1 consisted of the
WAM-4 (as its artificial hands) and the WL-5 (Its artificial
legs).
WABOT-1

© M. Rauterberg, TU/e 91
SONY’s AIBO (1999)
1999 marked a turning point for the world of entertainment when Sony
introduced the electronic robot AIBO in Japan. AIBO, referred to as the
Entertainment Robot means “companion” and is an acronym for Artificial
Intelligence RoBOt.
Sony has developed two versions of AIBO, the first run ERS-110 and the newer
ERS-111. Both of the models have proven to be a large success. When the first
5,000 ERS-110 models were introduced 3,000 sold in Japan within twenty
minutes on the Internet. The 2,000 models that remained were made available
exclusively for the United States and were sold within four days on the Internet.
Demand for the new AIBO ERS-111, which has a 64-bit microprocessor and 32
bytes of memory, rose immensely. Ten thousand ERS-111 AIBO models were
made available despite receiving 135,000 orders. “The digital ‘bot is more
dexterous than its predecessor: In addition to all the usual tricks—heeling or
chasing a ball—it does a little dance and waves a front paw on hearing its name.
Speech-recognition software lets it learn up to 50 commands” (International
Business, # 3709, page 170). The cost for AIBO in the United States has
remained thus far steady at $2,500 with the option to purchase a $500 performer
kit. AIBO is attempting to become popular all over the world, however, 90% of
AIBO purchases still come from Japan.

© M. Rauterberg, TU/e 92
SDR-3X (2000)
Much like its predecessors Aibo and Aibo II, Sony's 50cm-tall, 50kg prototype SDR-3X has been designed to
entertain. It can perform a variety of relatively high-speed, autonomous movements, including walking at a
speed of 15 meters a minute and dancing to a tune with a quick tempo. It's equipped with speech- and image-
recognition functions.
The SDR-3X gave a demonstration by walking at a high speed, moving its body like a gymnast, dancing to a
disco tune with a quick tempo and kicking a ball into a goal net as instructed by voice.
The SDR-3X model employs the same OPEN-R architecture as Sony's autonomous entertainment canine robot
"AIBO." Similar to AIBO, the SDR-3X model recognizes human voices and images. It has an 18,000-pixel CCD
color camera in its head area. The SDR-3X model can maintains its balance in the upper half of the body by
twisting its body and moving its arms, thereby realizing stable walking movements with alternating feet.
The SDR-3X model has 24 joints, with actuators for each joint that help move the joints. It has two joints in the
neck, two in the trunk, four in each arm and six in each leg. Two 64-bit RISC processors enable real-time control
of the joints to realize autonomous movement.
Sony's original real-time operating system called "Aperios" is used for the SDR-3X.
The biggest challenge for a humanoid robot is to keep its body in full balance. The SDR-3X keeps balance by
moving its arms and twisting at the waist to counteract the yaw-axis moment, the force needed to turn the body
right and left, which is generated from the lower half of the body every time the robot takes one step forward in
the high-speed walking movement.
And the robot's posture is controlled in real-time, to prevent it from falling over. It uses a variety of information,
such as the angle of the floor, gathered from contact sensors in the torso section as well as from a "dual-axis
accelerometer" and a "dual-axis angular rate sensor" in the waist section, whenever the robot walks up slopes
and moves its whole body.
It can (1) move forward or backward and walk sideways at a speed of up to 15 meters a minute; (2) turn left or
right when walking (with the maximum 90 degrees of freedom for each step forward); (3) get up from the
position of lying on its stomach or its back; (4) stand on one leg (possible even on inclined ground); (5) walk on
a bumpy road; (6) kick a ball; and (7) dance to a wide range of tunes.

© M. Rauterberg, TU/e 93
Augmented Reality (AR)

Core aspects:

User sees real environment; combines virtual with real.

Supplements reality, instead of completely replacing it.

Photo-realism not necessarily a goal.

© M. Rauterberg, TU/e 94
AR (historical overview)

•Early 1990’s: Boeing coined the term “AR”.

Wire harness assembly application begun.

•Early to mid 1990’s:

UNC ultrasound visualization project

•1994: Motion stabilized display [Azuma]

•1994: Fiducial tracking in video see-through

[Bajura / Neumann]

© M. Rauterberg, TU/e 95
AR at Boeing (1990)

The term "augmented reality" was coined at

Boeing in 1990 by researcher Tom Caudell.
He and a colleague, David Mizell, were asked
to come up with an alternative to the
expensive diagrams and marking devices then
used to guide workers on the factory floor.
They proposed replacing the large plywood
boards, which contained individually designed
wiring instructions for each plane, with a
head-mounted apparatus that would display a
plane's specific schematics through high-tech
eyeware and project them onto multipurpose,
reusable boards. Instead of reconfiguring each
plywood board manually in each step of the
manufacturing process, the customized wiring
instructions would essentially be worn by the
worker and altered quickly and efficiently Tom Caudell
through a computer system.
© Boeing Inc., USA
© M. Rauterberg, TU/e 96
Motion-stabilized AR Display (1994)

This system is the first

motion-stabilized AR
display that works
outdoors and achieves
tighter registration
than any previous
outdoor AR system.

© Ronald Azuma,
HRL Laboratories

© M. Rauterberg, TU/e 97
Fiducial tracking (1994)

The AR tracking system of Bajura and

Neumann consists of a robust vision
landmark tracker, inertial gyro sensors, and
the complementary fusion filter described
above. The sensor module contains a CCD
video camera (Sony XC-999 with 6mm
lens), and three orthogonal rate gyroscopes
(GyroChip II QRS14-500-103, from
Systron Donner), which are tightly covered
by a foam block to provide shock
protection and a stable temperature
environment from the sensors. The video
camera provides a 30 Hz video stream,
while the three gyroscopes are sampled at
1kHz via a 16-bit A/D converter (National
Instruments DAQPCI-AI-16XE-20).
© University of Southern California
© M. Rauterberg, TU/e 98
MIT Wearables (1996-)

What's a Wearable?
To date, personal computers have not lived up to their
name. Most machines sit on the desk and interact with
their owners for only a small fraction of the day. Smaller
and faster notebook computers have made mobility less of
an issue, but the same staid user paradigm persists.
Wearable computing hopes to shatter this myth of how a
computer should be used. A person's computer should be
worn, much as eyeglasses or clothing are worn, and
interact with the user based on the context of the situation.
With heads-up displays, unobtrusive input devices,
personal wireless local area networks, and a host of other
context sensing and communication tools, the wearable
computer can act as an intelligent assistant, whether it be
through a Remembrance Agent, augmented reality, or
intellectual collectives.

© M. Rauterberg, TU/e 99
Digital Desk (1991)

Pierre Wellner

The DigitalDesk is built around an ordinary physical desk and can be used as such, but it has extra capabilities. A video camera is
mounted above the desk, pointing down at the work surface. This camera's output is fed through a system that can detect where the
user is pointing (using an LED-tipped pen) and it can recognise the documents that are placed on it. The more advanced version
also has a computer-driven projector mounted above the desk enabling electronic objects to be projected onto real paper
documents -- removing the burden of having to switch attention between screen and paper and allowing additional user-interaction
techniques. [invented and built by Pierre Wellner, Xerox EuroPARC, ACM CHI paper]
© Xerox EuroPARC, UK
© M. Rauterberg, TU/e 100
ImmersaDesk (1994)

The ImmersaDesk is a drafting-table

format virtual prototyping device.
Using stereo glasses and sonic head
and hand tracking, this projection-
based system offers a type of virtual
reality that is semi-immersive.
The ImmersaDesk features a 4x5-foot
rear-projected screen at a 45-degree
angle. The size and position of the
screen give a sufficiently wide-angle
view and the ability to look down as
well as forward. The resolution is
1024 x 768 at 96Hz.

© University of Illinois at Chicago

© M. Rauterberg, TU/e 101

Build-It (1996)

design team with experts

from different disciplines
are planning together
integration of
different expertise
intuitive interaction style
with a tangible object
‘brick’
an example for a natural
user interface

© ETH, Zurich

© M. Rauterberg, TU/e 102

Cubby (1997)

Cubby, a desktop VR system,

addresses the problem ‘ unification of
the display and manipulation spaces’.
With a pen-like input device, the user
can operate on virtual objects where
they appear.

Unlike many VR systems, Cubby is

well-suited to precision manipulation
tasks. Possible areas of application
include surgical simulation and
computer aided modeling. Invented and built by Tom Djajadiningrat
© Technical University Delft, The Netherlands

© M. Rauterberg, TU/e 103

Virtual Workbench (1998)

Kent Ridge Digital Labs (KRDL), Singapore

© Kent Ridge Digital Labs

© M. Rauterberg, TU/e 104

DynaWall & CommChairs (1998)

The size of the DynaWall opens a new

set of human-computer interactions.
It is possible that information objects
can be taken at one position and
put somewhere else on the display
or thrown from one side to the
opposite side.
Dialog boxes always appear in front of
the current user(s).
User interface components are always
© GMD IPSI, Germany at hand, etc.

© M. Rauterberg, TU/e 105

Horizontal 3D Displays (1998)

© CMD, Uppsala University

© M. Rauterberg, TU/e 106

Speech and Voice in User Interface

Application areas

• Control and Data input in “hands-busy” environments

• Feedback in visually limited environments

• System control over telephone-line

• Control for impaired or disabled people

© M. Rauterberg, TU/e 107

Speech Signal Processing (1949)

J. Dreyfus-Graf (1949).
“Sonograph and Sound Mechanics”.
The Journal of the Acoustic Society of America, 22, pp. 731-739
This was not a real speech recognizer but an oscilloscope that would put
the beam at a different spot depending of the content of the speech.

Raymond Kurzweil: "[Bell's] insights into separating the speech signal into different frequency
components and rendering those components as visible traces were not successfully
implemented until Potter, Kopp, and Green designed the spectrogram and Dreyfus-Graf
developed the steno-sonograph in the late 1940s. These devices generated interest in the
possibility of automatically recognizing speech because they made the invariant features of
speech visible for all to see."

© M. Rauterberg, TU/e 108

Speech Recognizer (1952)

The first real pattern matcher was developed at AT&T Bell Labs:
Davis, K., Biddulph, R. & Balashek, S. (1952).
“Automatic recognition of spoken digits”.
The Journal of the Acoustic Society of America, 24(6), 637-642.

In 1952, as the US government-funded research began to gain momentum, Bell Laboratories

developed an automatic speech recognition system that successfully identified the digits 0 to 9
spoken to it over the telephone. Major developments at MIT followed. In 1959, a system
successfully identified vowel sounds with 93% accuracy. Then seven years later, a system that
had a vocabulary of 50 words was successfully tested. In the early 1970’s, the SUR program
yielded its first substantial results. The HARPY system, at Carnegie Mellon University, could
recognize complete sentences that consisted of a limited of range of grammar structures. But the
computing power it required was prodigious; it took 50 contemporary computers to process a
recognition channel. [source http://www.nexus.carleton.ca/~kekoura/history.html]

© M. Rauterberg, TU/e 109

TX-0 (1959)

MIT Speech
group with
TX-0, c1959.
Speech, handwriting
recognition, neuro data
analysis, etc.
Interactive editors,
debuggers, etc.

The TX-0 computer was the world's

first high speed transistorized
computer. The TX-0 was the most
powerful system of its day (1957),
representing a quantum leap in
computer technology at the time.
© MIT Lincoln Lab

© M. Rauterberg, TU/e 110

History of Speech Recognition Technology

Spontaneous natural
speech 2-way conversation
dialogue
network transcription
Fluent word system driven agent &
speech spotting dialogue intelligent
Speaking style

messaging
digit
strings name
Read dialing
speech
form fill office
by voice dictation
Connected
speech
directory
voice assistance
Isolated commands
words

2 20 200 2000 20000 Unrestricted

Vocabulary size (number of words)
© M. Rauterberg, TU/e 111
History of DARPA speech recognition benchmark

100%
Switchboard
Conversational
Speech foreign
Read
Speech
WSJ
Broadcast
WORD ERROR RATE

Spontaneous Varied Speech

 20k
Speech  Microphone foreign

ATIS NAB
10%  5k
1k  Noisy

 
 


Resource
Management Courtesy NIST 1999 DARPA
1% HUB-4 Report, Pallett et al.

1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003
© M. Rauterberg, TU/e 112
Speech controlled application

A voice activated dental record system (ORATEL, ORQUEST)

(200 different words, relatively simple grammar)

1. Dragon
2. L&H
3. IBM Pure Speech
4. Kurzweil

© M. Rauterberg, TU/e 113

International Standards

Based on the path-breaking work of Dzida, Herda and Itzfeld (1978) twenty years ago, the
German national standard DIN 66234, part 8 was developed and published in 1988.
Dzida, W., Herda,S. and Itzfeld, W.D. 1978, User-perceived quality of interactive systems. IEEE Transactions on Software
Engineering, SE-4, 270 – 276.

Its definitions of usability principles for software user interfaces for office work became the
basis for the international and European standard ISO EN 9241-10 (ISO 1996). This standard
serves as the reference for the European Community directive 90/270/EEC for minimum safety
and health requirements to be guaranteed by an employer for his staff working at computer
workstations.

1981-1988: DIN 66 234: Grundsätze ergonomischer Dialoggestaltung.

1990: European Community Directive 90/270/EEC for minimum safety and health requirements.

1995: ISO 9241: Ergonomic requirements for office work with display terminals (VDTs).

© M. Rauterberg, TU/e 114