MPEG-4 Demystified

Apple Worldwide Developers Conference 24 June 2003 Rob Koenen

President, MPEG-4 Industry Forum Chairman, MPEG Requirements Group Vice President, InterTrust Technologies
24 June 03 0

Overview
What is MPEG-4? How does MPEG-4 work? Recent Developments in MPEG-4 Why use MPEG-4 Deployment of MPEG-4 About M4IF

24 June 03

Overview
What is MPEG-4? How does MPEG-4 work? Recent Developments in MPEG-4 Why use MPEG-4? Deployment of MPEG-4 About M4IF

24 June 03

What is MPEG-4?
The Media Standard
Set by ISO/IEC

An interoperable cross-platform ecosystem An architecture and coding methods for representing rich multimedia content
beyond video and audio, supporting synthetic content, raster and vector graphics

Designed for all digital multimedia platforms

24 June 03

Where does it come from?

From Moving Picture Experts Group, who also did: MPEG-1: Standard for Video CD, CD-I, web, digital photo cameras
MPEG-1 Layer III audio codec (MP3)

MPEG-2: the DVD and digital TV standard And also: MPEG-7: standard for description of content MPEG-21: framework for interoperable use and exchange of digital media
24 June 03 4

MPEG-4 Vision
No convergence, but proliferation of multimedia over different networks, terminals. Common multimedia technology necessary that supports:
Broadcast Communication Retrieval
online packaged media

24 June 03

One Single Technology for

24 June 03

Write once - Play everywhere

24 June 03

Applications of MPEG-4
Mobile devices Broadcast
Low datarates, errors, scalability New, on-demand services HD broadcast MPEG-4 over MPEG-2 Scalability, low to medium datarates, interactivity

Streaming services

Packaged Media (DVD)

Interactivity, HD through higher compression rates
24 June 03 8

Overview
What is MPEG-4? How does MPEG-4 work? Recent Developments in MPEG-4 Why use MPEG-4? Deployment of MPEG-4 About M4IF

24 June 03

The Nature of MPEG-4

Object-based multimedia content representation standard Revolutionary Systems Layer State-of-the art coders, responsibly upgraded Profiles and Levels restrict complexity and guarantee interoperability

24 June 03

10

The MPEG-4 Ecosystem

Creation
24 June 03

Distribution Consumption
11

An MPEG-4 Scene
Animated 2D photo / mesh Graphic objects

Background with music

Natural audiovisual object

24 June 03 12

The Scene Tree

root

background voiceover sprite music

baby

animated fish

graphics

video bubbles sound mesh picture soil

fish

seaweed red

blue

24 June 03

13

Recapping: MPEG-4 is about the Coding of Audiovisual Objects

Audiovisual Scene is with objects These Objects can be of different nature Compositor puts objects in scene Efficient, Real-time Binary scene description language (BIFS) Coding scheme is optimal for object type Principle is fully independent of bitrate

24 June 03

MPEG-4 Supports a Variety of Objects

Visual
Video Animated face & body; 2D and 3D animated meshes Text and Graphics

Audio
General audio mono - 5.1 channels Speech Synthetic sounds (Structured Audio) Text-to-speech Environmental spatialization
24 June 03 15

The Parts of the MPEG-4 Standard

4. Conformance

1. Systems 2. 10. 3. AVC Visual Audio 1. Systems 6. DMIF Transport layer 7. MPEG-4 on IP
24 June 03

presentation decoding

demux & buffer transport interface

In principle not in standard

16

5. Reference SW

Overview
What is MPEG-4? How does MPEG-4 work? Recent Developments in MPEG-4 Why use MPEG-4? Deployment of MPEG-4 About M4IF

24 June 03

17

Visual Media Object Types in MPEG-4

Video from 10 1,000,000 Kbit/s
Multiple rectangular or arbitrary shape objects in the scene Scalable Interlaced and Progressive

Sprites (e.g. backgrounds): send once, then warp Computer-generated visual information
Synchronized graphics & animated text Face and body animation, animated meshes with moving texture,

24 June 03

Audio Media Objects in MPEG-4

Audio from 4 kbit/s to 64 kbit/s/channel
Arbitrary number of objects in the scene MPEG-4 AAC, TWINVQ > 4kbit/s: Harmonic and Individual Lines plus Noise (HILN)

Voice from 2 kbit/s to 24 kbit/s

2-4 kbit/s: HVXC (Harmonic Vector eXcitation Coding) 4-24 kbit/s: CELP:

Large step + Fine Grain Scalability

1 kbit/s steps through Bit-Sliced Arithmetic Coding

24 June 03

19

MPEG-4s Synthetic Audio Objects

Structured Audio
SAOL (SA Orchestra Language) SASL (SA Score Language) Great music at very low bitrates

Coded form of MIDI Wavetable synthesis for simple decoders

Including effects

Text-To-Speech (interface!)
To complement face/body animation

24 June 03

20

MPEG-4 Systems
Binary Scene Description
VRML concepts + Streaming + Real Time + Efficiency Support for content in scene from different sources Allows interaction (local/remote) 2D and 3D Dynamic scene updates and scene animation

XMT: eXtensible MPEG-4 Textual Format

Textual format for BIFS Includes some SMIL harmonization

24 June 03

21

MPEG-4 Systems
Predictable behavior of a decoder and decoded content:
Tight synchronization of A, V, synthetic, graphic elements Buffer management

Flextime
Spring-like timing model (a-la SMIL)

IP management and protection (IPMP) Interface

Standard interface to proprietary DRM Systems MPEG-21 to bring more interoperability in DRM

File format MP4 (based on QuickTime)

24 June 03 22

MPEG-4 Systems (cntd.)

MPEG-J(ava)
API for complicated content behavior, APIs to network / terminal / UI resources Application Engine

Audio Rendering
Specify downmix from arbitrary number of channels Environmental spatialization
modeling of environment for spatial sound reproduction
Physical and perceptual model

24 June 03

23

Profiles & Levels

Conformance points are Profiles@Levels
A bit like in MPEG-2

Profiles determine tool set

E.g. B frames, pel Motion Compensation

Levels limit complexity

E.g. MacroBlocks/sec, Complexity Units in Audio

Profiles are convergence point for Industry Standards built on MPEG-4

vehicle for market decisions and uptake e.g. Internet Streaming Media Alliance

24 June 03

24

Profile Dimensions:
Visual (natural, synthetic, natural + synthetic) Audio (natural, synthetic, natural + synthetic) Graphics Scene Description (Scene Graph)
Tools to describe and manipulate scene

Media Profiles

MPEG-J (Main and Personal) Object Descriptor (Synch and Buffers) MPEG does not prescribe how to combine these
Thats what other industry fora do

24 June 03

25

Overview
What is MPEG-4? How does MPEG-4 work? Recent Developments in MPEG-4 Why use MPEG-4? Deployment of MPEG-4 About M4IF

24 June 03

26

MPEG-4 Advanced Video Coding

Also standardized as ITU-T H.264
Part 10 of MPEG-4 standard Also known as the JVT codec

Built by worlds top video coding experts from ITU and MPEG in Joint Video Team (JVT) Best quality/complexity trade-off
Better than WM9, according to industry leader LSI Logic Improvements will continue

Significantly further reduces video bitrates Fits into the MPEG-4 framework
Will coexist with MPEG-4 part 2 in market
24 June 03 27

High Efficiency Advanced Audio Coding

CD quality @ 48kbps; High quality @ 32kbps
Achieved through Spectral Bandwidth Replication predict upper half of spectrum from lower half Forward and backward compatible with normal AAC

Amendment (addition) to MPEG-4 Audio Already operational in XM radio, chosen for Digital Radio Mondiale (DRM) Tested as best codec by EBU (European Broadcasting Union) over Windows, Real

24 June 03

28

EBU listening test results for 48Kbps stereo

100 80 60 40 20 0
LP F p3 PR O rig in al R ea l8 lu s 2 ea lG 3. 5k H R A A aa cP W M z M P 7k H O m z LP F C A 8 3

LPF = Low Pass Filter. aacPlus is brand name for High Efficiency AAC. mp3PRO is brand name for MPEG-1 Layer III with Spectral Band Replication EBU = European Broadcasting Union
24 June 03 29

Other developments
Animation Framework eXtension (AFX)
Unified framework for interactive animation and graphics

Work on 3D Video coding is in Requirements phase Work on lossless audio coding is entering development phase

24 June 03

30

Overview
What is MPEG-4? How does MPEG-4 work? Recent Developments in MPEG-4 Why use MPEG-4? Deployment of MPEG-4 About M4IF

24 June 03

31

Standards Make Sense. They

fuel innovation (GSM, WLAN/802.11, ) survive the test of time (PAL, NTSC, MP3, ) prevent expensive format churn (VHS, CD, ) help markets mature (CD, DVD, MPEG-2, ) provide an interoperable ecosystem of tools and equipment: different providers make what they excel in No vendor lock-in - avoid channel conflicts Market-based price control mechanisms Multi-vendor sourcing of equipment No monopolies - no single vendor controls the format

24 June 03

32

MPEG-4 has Clear Benefits

Code once, use across platforms and players Users pick their favorite players, devices Content producers pick their favorite tools Providers only need to stream in one format Competition drives quality up

24 June 03

33

(R)evolution
A revolution in functionality
Designed from the start for interactivity State of the art compression; responsible upgrades Object-based = intrinsic flexibility Profiles, Levels limit complexity, guarantee interoperability Offers synthetic content Local rendering of 2D & 3D graphics and audio

An evolution in infrastructure
Allows all types of casting: unicast, multicast, broadcast on all network types, no need for new networks or transports Leverage existing MPEG-2 and other infrastructures

24 June 03

34

Business Benefits
MPEG-4 saves money
More efficient use of bandwidth Repurpose existing content No need to duplicate work when adding value to assets Integrate into existing MPEG delivery environments Used on all types of network, including IP

MPEG-4 makes money

Use existing content and infrastructure in new ways Add new dimensions to content, including interactivity Low-risk / high-return path for new entrants
24 June 03 35

Risks of Proprietary Technology

Third party business and pricing models Proprietary, confidential 3rd party technology roadmaps
May change without warning

Potential channel conflicts

Supplier also competitor License includes more bundled features" than required

Single sourcing problems

24 June 03

36

Worldwide MPEG-4 Forecast

Shipments in millions of units*

*Includes standalone MPEG-4 chips and cores embedded in processors

Source: ISUPPLI CORP.
24 June 03 37

Improvement Through Competition

Proprietary vendors underrate MPEG-4 quality:
Comparisons with early, non-optimized codecs Industry-standard video quality does not exist!

MPEG standardizes the minimum: only decoders

Competition on encoding and post-processing

MPEG-2 bit-rates have reduced by over 50%

after the standard was frozen in 1996 without upgrading decoders

Competition between the experts will:

Drive MPEG-4 quality up after deployment of decoders Drive functionality of tools Make AVC beat all proprietary codecs, including WM9
24 June 03 38

MPEG-2 Coding Efficiency Harmonic, Inc.s view

24 June 03

39

MPEG-2 Coding Efficiency TandbergTVs view

24 June 03

40

Overview
What is MPEG-4? How does MPEG-4 work? Recent Developments in MPEG-4 Why use MPEG-4? Deployment of MPEG-4 About M4IF

24 June 03

41

Widely adopted and supported

Industry wide PC media player support
QuickTime 6 RealOne Player / Helix (Envivio plug-in) DivX: Millions of downloads weekly Windows Media Player: several MPEG-4 plug-ins

MPEG-4 is part of the standard for 3G and 2.5G mobile phones

3GPP, 3GPP2 adopted MPEG-4 Profiles for video to mobile AAC is the optional audio codec in 3GPP File format is adaptation of MP4 QuickTime 6.3 supports 3GPP
24 June 03 42

Widely adopted and supported

Recommended by Internet Streaming Media Alliance (ISMA)
Vertical spec using horizontal MPEG-4 standard Founded by Sun, Kasenna, IBM, Apple, Cisco, Philips

MPEG-4 AVC to be adopted by DVB for DVB over IP Becoming de-facto standard for security, surveillance Supported in emerging Home Media Centers

24 June 03

43

Recent announcements and releases

MPEG-4 HD-based video camera (Samsung) Solid state video cameras (Panasonic, Divio) DVD players that read MP4 (Toshiba, Kiss) HDTV Chips (Sigma Designs) PDAs (Sonys Clie, Sharps Zaurus) Video jukeboxes ([Link], Archos) Home entertainment gateways (e. Digital, Equator, Sigma, ) Mobile phones that decode and stream MP4 (Nokia, Motorola)
24 June 03 44

Overview
What is MPEG-4? How does MPEG-4 work? Recent Developments in MPEG-4 Why use MPEG-4? Deployment of MPEG-4 About M4IF

24 June 03

45

About the MPEG-4 Industry Forum

24 June 03

46

History
First Meetings 1999
What do we need to do to get MPEG-4 adopted?

Officially Established June 2000

Not-for-Profit Organization in Switzerland

Now approximately 100 members

Worldwide Across Industries IT, Telecoms, CE, Academia, content and service providers

24 June 03

47

Membership includes
AMD, Agfa Monotype, AOL Time Warner, Apple Computer, Canon, Cisco, DivXNetworks, Dolby, Envivio, Fraunhofer, Fujitsu, Fuji-Xerox PAL, Harmonic, Hitachi, IBM, Intel, InterTrust, iVAST, Matsushita, Microsoft, Mitsubishi, Motorola/GI, MPEG LA, NEC, NTT, Nokia, Oki, PacketVideo, Philips, RealNetworks, Scientific-Atlanta, Siemens, Sony, STMicroelectronics, Sun, TandbergTV, TDK, Tektronix, Thomson, Toshiba, Xilinx and many others (~100 members)

24 June 03

48

Goal
Getting MPEG-4 widely adopted Doing the things that MPEG does not and can not address
Bootstrapping licensing Product interoperability Compliance Program Education, Information Clearinghouse Marketing, Exhibitions

24 June 03

49

Membership
3,000 US$ per year for full membership
Access to meetings, documents, Interop Program Speaking opportunities and co-exhibiting with M4IF

300 US$ for associate membership

Same rights, except voting rights

24 June 03

50

Licensing the Responsibilities

MPEG Standardizes
And collects Patent Statements But as a rule does not discuss licensing terms

M4IF initiates, discusses, encourages, suggests, catalyzes, bootstraps

But never licenses, determines, decides, And does not recommend specific licensing terms (although it may suggest alternative approaches)

Licensors determine, decide, sell licenses

In discussions with the market Usually in patent pools (joint licensing schemes)
24 June 03 51

Thank you
Further information: [Link] (website M4IF) [Link] (website MPEG) [Link]/mpeg4 (Apples MPEG-4 Pages)

24 June 03

52