Prof. Dr. Mario Döller & Prof. Dr. Harald Kosch & Prof. Dr.

Michael Granitzer
Chair of Distributed Information Systems / Chair of Data Science
 Organisation
◦ Stud.IP
◦ Office hours (Sprechstunde): Mo, 16-18, Upon request per Email
to [email protected]
◦ All Material available on Stud.IP

 Credits
◦ Master Computer Science in the Focus Group “Information and
Communication Systems”
◦ Master Computational Mathematics in the Focus Group “Data
Analysis and Data Management and Programming”
◦ Master AI Engineering in the Module Group “AI Applications“
◦ Double-Master Informatique-Informatik Lyon-Passau

 Exams
◦ Written exam, 90min

Multimedia Databases 2
 Lecturers:
◦ Prof. Mario Döller
◦ Prof. Harald Kosch

 The first three lectures take place 16-18 each Tuesday (28.4, 25.4, 2.5) in HS 13.

 Further lectures, times and locations see Stud.IP

 Lecture slides and support material is available before.

 Exercises are done by Kanishka Ghosh Dastidar Kanishka.GhoshDastidar@uni-

passau.de and Alaa Alhamzeh [email protected]

 Information on the schedule is given in Stud.IP by Kanishka Ghosh Dastidar and Alaa
Alhamzeh

Multimedia Databases 3
 Books
◦ Meinel & Sack, Digital Communication, Springer, 2014
◦ Steinmetz R & Nahrstedt K, “Multimedia Technologie: Grundlagen,
Komponenten und Systeme”, 3. Auflage , Springer, 2000.
◦ Steinmetz R & Nahrstedt K, “Multimedia Systems”, Springer-Verlag, Berlin,
Heidelberg, New York, 2004
◦ Steinmetz R & Nahrstedt K, “Multimedia Applications”, Springer-Verlag, Berlin,
Heidelberg, New York, 2004
◦ Ze-Nian Li & Mark S. Drew, "Fundamentals of Multimedia", Pearson Education,
2004.
◦ Harald Kosch: “Distributed Multimedia Database Technologies supported by
MPEG-7 and MPEG-21”, CRC Press, November 2003, ISBN 0-8493-1854-8.
◦ Kenny A. Hunt: "The Art of Image Processing with Java", CRC Press, 2010.

 Conferences
◦ ACM Multimedia
◦ IEEE International Conference on Multimedia & Expo (ICME)
◦ …

 Journals
◦ IEEE MultiMedia
◦ Multimedia Tools and Applications
◦ …

Multimedia Databases 4
 Topics Contents
Definition, Multimedia according to Steinmetz,
01 Introduction
Media Types, Development of MM-DB
02 Colors Color and Color Perception
Raster graphics/Vector graphics, Image
03 Image medium
formats, Image manipulation (Filter, etc.)
Basics of media (formats, etc.), technology for
04 Video/Text/Audio
video segmentation etc.
05 Compression Encoding and compression of media data
06 Modeling Modeling of media data (MPEG-7)
Information- and content based retrieval of
07 CBIR
media data
Characteristics and types of MM query
08 Query languages
languages and MM query processing

Multimedia Databases 5
 Block Contents
High dimensional feature vectors indexing
09 Index Structures
techniques
Presentation of current systems
10 Systems (commercial/research) for media storage and
retrieval

Multimedia Databases 6
 Main literature:
◦ Steinmetz R & Nahrstedt K, “Multimedia Systems”, Springer-
Verlag, Berlin, Heidelberg, New York, 2004
◦ Steinmetz R & Nahrstedt K, “Multimedia Applications”,
Springer-Verlag, Berlin, Heidelberg, New York, 2004
 Supplemental Article:
◦ Zixia Huang, Klara Nahrstedt, and Ralf Steinmetz. 2013.
Evolution of temporal multimedia synchronization
principles: A historical viewpoint. ACM Trans. Multimedia
Comput. Commun. Appl. 9, 1s, Article 34 (October 2013),
◦ https://dl.acm.org/doi/10.1145/2490821

Multimedia Databases 7
1 Multimedia Definition
2 Multimedia Systems
3 Multimedia – Main Domains
4 Data Streams
5 Types of Media
6 Multimedia Databases

Parts of the lecture adapted from the set of slides provided by

Prof. Dr. Andreas Henrich, University of Bamberg

Multimedia Databases 8
1 Multimedia Definition
2 Multimedia Systems
3 Multimedia – Main Domains
4 Data Streams
5 Types of Media
6 Multimedia Databases

Parts of the lecture adapted from the set of slides provided by

Prof. Dr. Andreas Henrich, University of Bamberg

Multimedia Databases 9
  Multi - many, various, …
 Medium - a substance regarded as the means of
transmission of a force or effect; a channel or system of
communication, information, or entertainment
(Merriam-Webster Dictionary )

- An intervening substance through which something else is

transmitted or carried on (FreeDictionary)

 So, Multimedia???
◦ The term Multi & Medium does not make a lot of sense!
◦ The notion of “medium” must be further investigated

Multimedia Databases 10
 Medium as defined by the
Multimedia and Hypermedia
Experts Group (MHEG) of ISO
◦ Medium
◦ Means of distributing and
presenting information
◦ Classification based on
perception, presentation,
representation, transmission
etc.

Is adequate to the
definition of multimedia

Multimedia Databases 11
 The medium of perception (Perception = Sensory perception)
◦ Primary differentiation by and
 The medium of representation
◦ Reference to the internal computer presentation of the information
◦ Formats for text, graphics, single image, animated pictures, …
 The medium of presentation
◦ Reference to in- and output
◦ Output media: Paper, screen, loudspeakers, …
◦ Input media: Keyboard, camera, mouse, microphone, …
 The medium of storage
◦ Distinction based on the data carrier: magnetic discs, DVD, paper, …
 The medium of transmission
◦ By what means is the information transferred? Coaxial cable, glass fiber, …
 The medium of information exchange
◦ What information carriers are used for the exchange between various spots?
(storage media + transfer media)

Multimedia Databases 12
 Space and Time dimensions for representing media
 Time is an important dimension in the representation of the
media.
 Wrt. time, a medium may be qualified as:
◦ Time independent (discrete)
 Ex.: Text, Graphics
◦ Time dependent (continuous)
 Continuous sequences of time-dependent values
 Ex.: Audio, Video

 The characteristics discrete/continuous do not constrain the

internal representation of the medium – they only refer to the
impression of the user at presentation time)

Multimedia Databases 13
 In the most general sense, Multimedia is any combination of
digitally manipulable types of media (text, sound, image,
animation, video).

 A stricter version of this definition subsequently restricts the

possible combinations of media:
◦ It requires:
 Mixing of both continuous and discrete media
 A significant degree of independence between these media

 The less strict definition is very common in practice.

 Independence between mixed media is a usual feature of
multimedia databases

Multimedia Databases 14
 Multimedia elements can be produced by means of authoring
systems (Flash/SMIL/HTML 5 etc.).

 Multimedia authoring systems are programs that provide

means to create complete multimedia presentations by
interlinking objects (text, audio, illustrations, etc.) and taking
interactive, user-related processes into account

 Adobe Director, Dreamweaver, MS Powerpoint,…

Multimedia Databases 15
 Multimedia is interactive when the user is able to control
which elements are delivered and when

 Interactive multimedia is hypermedia when the users are

presented with a structure of linked elements in which they
can navigate.

 Example: Simple Interactive Video Authoring Suite (SIVA)

producer, the interactive video authoring tool of the Mirkul
project
http://www.mirkul.uni-passau.de/

Multimedia Databases 16
 Multimedia is linear when it can only be presented in a single
continuous flow over time.
◦ Examples:
 Film
 Internet Radio
 (in general, linear multimedia is non-interactive)

 Multimedia is non-linear when it consists of a set of elements

that may be presented according to different flows
◦ Examples:
 Games
 Interactive CD
 SIVA interactive video

Multimedia Databases 17
1 Multimedia Definition
2 Multimedia System
3 Multimedia – Main Domains
4 Data Streams
5 Types of Media
6 Multimedia Databases

Parts of the lecture adapted from the set of slides provided by Prof. Dr.
Andreas Henrich, University of Bamberg

Multimedia Databases 18
 Definition:
◦ A Multimedia System is characterized by the computer-controlled
generation, manipulation, presentation, storage, and
communication of a set of independent media, which include at
least one continuous (time-dependent) and one discrete
(independent of time) medium.[1]

[1] Adapted from R. G. Herrtwich, R. Steinmetz, Towards Integrated Multimedia Systems:

Why and How, Telekommunikation und multimediale Anwendungen der Informatik,
Informatik-Fachberichte Vol 293, 1991, pp. 327-342

Multimedia Databases 19
 Combination of media
◦ An MM application should be able to process at least one discrete
and one continuous medium!

 Independence
◦ No rigid connection between the combined media

 Computer-aided integration: integrated MM-Systems

◦ Not only recording and presentation but also other functions
◦ Example: temporal, spatial, content-related synchronization

 Communicating systems
◦ Realization of MM-systems in distributed environments

Multimedia Databases 20
(c) Shiguo Lian, Dimitris Kanellopoulos, Giancarlo Ruffo; Recent Advances in
Multimedia Information System Security; Informtica; 33(1):3-24; 2009

Multimedia Databases 21
1 Multimedia Definition
2 Multimedia System
3 Multimedia – Main Domains
4 Data Streams
5 Types of Media
6 Multimedia Databases

Parts of the lecture have been adopted and applied from the set of slides provided by
Prof. Dr. Andreas Henrich, University of Bamberg

Multimedia Databases 22
Usage
Services

.....
System
Basics

(According to Ralf Steinmetz: Multimedia-Technologie, 2.

edition, Springer, 1999)
Multimedia Databases 23
 Basics
 Basic principles for the processing of digital
audio/video data:
◦ Nyquist-Shannon sampling theorem and Pulse Code
Modulation
◦ Different techniques, specialized in the respective
medium
PCM
◦ Audio technology include music and speech processing
◦ Video technology based on (digital) television
technology
f(t) 0 4 5 4 3 4 6 7 . . .
 Further considered: t 1 2 3 4 5 6 7 8 ...
◦ Single images, graphics, animations, text
Digital Signal
 With current quality requirements:
◦ efficient, quality-preserving compression necessary
(mp3, jpeg etc.)
 Aim:
◦ Fully digital systems

Multimedia Databases 24

System
Central aspect: Quality of service
◦ Represents a defined, controllable system behavior according to
measurable parameters
 Basic Functionalities:
◦ Processing
 Task of operating systems and programming environment:
 abstracted interface to the computer hardware
 Abstraction can be provided by the MM-operational system, a computer language
or a OO-class hierarchy
◦ Storage
 Usage of specific storage devices; the stored multimedia is managed by e.g. a
media server or a DBMS
◦ Communication
 Multimedia imposes a number of requirements on communication networks:
 high bandwidth
 high level of reliability in order to meet the time constraints (video streaming)

Multimedia Databases 25
 Services
.....

Services provide ready-to-use integrated functions to user

applications
 Communication
◦ e.g. Email, conferencing applications, joint editing
◦ These services need to take into account the constraints of multimedia
 Synchronization
◦ Establish a temporal relation between multimedia data items (e.g. Games)
 Security
◦ Measures to prevent attacks
 Documents
◦ Structuring of different media to form a “whole“ (= a MM document)
 Content analysis
◦ Considers the semantics of the contents
◦ Enables more effective access and new types of applications

Multimedia Databases 26
 Usage
Applications and user interfaces represent the user-
perceivable aspects of multimedia data
 Important: the question of MM-specific design
 Examples of applications:
◦ E-teaching and e-learning
 Areas of increasing importance:
◦ Specific tools for MM applications development
◦ Project management for MM applications development
 But:
◦ There is more to MM-Systems than just MM-applications
development!

Multimedia Databases 27
1 Multimedia Definition
2 Multimedia System
3 Multimedia – Main Domains
4 Data Streams
5 Types of Media
6 Multimedia Databases

Multimedia Databases 28
 A data stream is a sequence of individual packets that are
transmitted under time-dependent constraints
◦ Packets can include any information (continuous or discrete)

 Modes of transmission
◦ Asynchronous
 Packets should reach the recipient as soon as possible –
but no guarantee is provided
 Perfectly suitable for discrete media
 For continuous media, additional temporal restrictions
may have to be considered (when streaming)

Multimedia Databases 29
◦ Synchronous
 Defines a maximal end-to-end delay
 Packets may also arrive earlier
 Sufficient memory is required on the receiver side! Ex:
processing uncompressed video data with a data rate of 150
Mbps and a maximal end-to-end delay of 1 second to the
recipient, 18.75 Mbytes of memory are required.

◦ Isochronous
 Defines a maximal & minimal end-to-end delay
 Reduces storage requirements on the receiver

Multimedia Databases 30
 Periodicity of the transmission
◦ Strongly periodical
 Ex: PCM -Pulse-Code Modulation to encode speech for VoIP

T T T T
…
t
◦ Weakly periodical: periodic properties every n packets
T1 T2 T3 T1 T2 T3
…
T T
t
◦ Aperiodic
 Ex: Cooperative applications with a shared window
T1 T2 T3 T4
…
t
Multimedia Databases 31
 Variations of contiguous packet sizes
T

t t t
FBR VBR VBR
(fixed bitrate) (variable bitrate)

◦ Strictly even (Strictly regular)

 Uncompressed digital data transmission
◦ Weakly even/regular
 MPEG Standard → fixed recurring pattern of 3 types of frames I,
B, P with fixed size ratio
◦ Uneven/Irregular

Multimedia Databases 32
1 Multimedia Definition
2 Multimedia Systems
3 Multimedia – Main Domains
4 Data Streams
5 Types of Media
6 Multimedia Databases

Multimedia Databases 33
 discrete = static = independent of time

 The information is displayed in a time independent way, the

duration of use is determined by the receiver

 Examples
◦ Text (font)
◦ Graphics (drawings, diagrams, etc.)
◦ Real pictures (Photography, etc.)

Multimedia Databases 34
 Audio  Video
◦ Speech ◦ Real film
◦ Music ◦ Animation
◦ Tones, sounds, noises

Predetermined temporal sequence

Multimedia Databases 35
1 Multimedia Definition
2 Multimedia Systems
3 Multimedia – main domains
4 Data streams
5 Types of media
6 Multimedia Databases

Multimedia Databases 36
A MMDBMS must provide the following
functionalities:

◦ Storage, Indexing and searching of multimedia objects

◦ Transparence of all physical aspects
◦ Search functionality – “Content-based Retrieval and Search”
◦ Access structures for multimedia- and their descriptive data
(metadata)
◦ “Multi-user access” by “concurrency control”
◦ “Data consistency”
◦ “Reliability” by “recovery mechanisms”
◦ “ Cross-Media“ and “composite media“
◦ “ Real-time capacities“ and “ streaming“

Multimedia Databases 37
Multimedia Databases 38
 Multimedia-Data (no streaming)
Client

Descriptive information

Storage

Multimedia Databases
Multimedia-Query

39

Multimedia Database System

 Example:
An insurance company stores the file corresponding to an
accident in the form of a multimedia object. The object
contains:

◦ Pictures of the accident

◦ Descriptive forms containing structured text
◦ Interviews of the involved parties (audio recordings)
◦ Written report of a representative of the
insurance company

 Multimedia retrieval systems must retrieve structured and

unstructured data

Multimedia Databases 40
 Retrieval of structured data:
◦ is typically managed by a Database Management System (DBMS)
◦ The DBMS provides a query language (e.g. SQL)
◦ A deterministic matching of query and stored data is conducted

 Retrieval of unstructured data:

◦ is typically managed by an Information Retrieval (IR) System
◦ Comparison of similarity between query data and stored
documents representations; fuzzy matching
◦ Result: Ranked list of relevant documents ordered by similarity

Multimedia Databases 41
 A Multimedia Database Management System should combine
both DBMS and IR technologies
◦ Best of both worlds: Data modelling capabilities of DBMS +
extended and similarity-based capabilities of an IR-system

 Challenge = Finding of a data model that enables:

◦ Storage, matching and distribution of structured and
unstructured data

 Good candidate = Object-Relational Model

◦ Possibility of adding the required functions to a relational
database through user defined types and other object-
oriented extensions to classical SQL

Multimedia Databases 42
 Matching of queries and document representations:
◦ Consider the whole set of attributes and their relation in queries
and document representations (make use of structured and
unstructured attributes and objects)
◦ Combine exact matching of structured data with fuzzy matching
of unstructured data

 Distribution of data:
◦ Browsing and retrieval
◦ Time-related restrictions of video and audio presentations must
be considered

Multimedia Databases 43
1) As in many retrieval-systems the user must be able to
browse and navigate the dataset through hyperlinks, using
for example:
◦ Topic maps (ISO/IEC 13250:2000 standard)
◦ Summarization of multimedia objects

2) Queries specify the conditions of the search

◦ Multimedia query language:
 Predicates for expressing conditions on attributes, structure,
content and semantics of multimedia objects

Multimedia Databases 44
◦ Attribute-predicates:
 concern the attributes of multimedia objects with precise values
(cf. traditional DB attributes):
 e.g. date of an image, name of a show

◦ Structure-predicates:
 temporal predicates to specify info related to time
synchronization:
 for continuous media, such as sound and video
 for the expression of temporal relations
 e.g. “Find all the objects in which a jingle is running along
for the duration of an image”

Multimedia Databases 45
◦ Spatial-predicates to specify spatial layout properties of
multimedia objects:
 Example predicates: contains, is contained in, cuts, adjoins
 e.g. “Find all images in which the car is parked next to a tree”

◦ Temporal and spatial predicates can be combined:

 e.g. “Find all video segments in which the ball is seen within a
goal-box followed by crowd cheering lasting more than 30s”

◦ Temporal and spatial predicates can:

 Make reference to whole objects
 Make reference to subcomponents of objects: on condition that
the data model supports complex object representations

Multimedia Databases 46
 Semantic predicates:
◦ Target the semantic content of the data
◦ Are represented by features that are extracted and stored for
each multimedia object
◦ Uncertainty, proximity and significance can be expressed in
the query
◦ Ex. "Find all videos in which two brothers shake hands"

 Multimedia query language:

◦ Structured language
◦ Users do not express queries in this language (generally too
complex) but describe their information needs through a user-
friendly interface
◦ Natural language capabilities?
◦ The interface translates the user input into a correctly formed
query expressed in the MM query language

Multimedia Databases 47
Query by example:
 The user submits an example data item. Its features are extracted
and compared to those of the objects of the DB. A ranked list of
matching objects is returned.
 e.g. in a graphic user interface (GUI): Users provide an image of a
house and select desired features to express the query: “Find all
houses with similar shape but different color”

Question-answering
◦ e.g. Pose questions related to the content of MM objects: “How
many birds sit on the tree? ”

Multimedia Databases 48