A short introduction to Multimedia

Forensics: the science

discovering the history of
multimedia contents
Sebastiano Battiato
Dipartimento di Matematica e Informatica,
Università di Catania

Image Processing LAB – http://iplab.dmi.unict.it

[email protected]
 Forensic science (often shortened to forensics) is the practical
application of science to matters of the law. Use of scientific
methods for gaining probative facts (from physical/analog or
digital evidences )

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 History of Digital Forensics
• Earliest notion of digital forensics came when the Federal
Rules (US) of evidence first started to discuss digital
evidence in the 1970s

• Real digital forensics investigations started in the mid-to late

1980s when federal agents had to start figuring out ways to
search computers for digital evidence

• This “home-grown”, bottom-up approach continued until the

late 1990s when security researchers at universities and
labs started to figure out that this problem was big enough to
warrant investigation.

• First research groups started around 2000 or 2001.

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 The first Digital Forensics Research
Workshop (DFRWS) was held in Utica, NY in
August 2001.

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Multimedia
Forensics

- Source identification
- Integrity verification/tampering detection

Techniques from multimedia forensics merely provide a way to

test for the authenticity and source of digital sensor data. In this
sense is not about analyzing the semantics of digital or
digitized media objects.
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Forensics (in academic
literature)
• The terms appeared in early 2000
• K. J. Ray Liu, on “Multimedia Forensics: Where
Sherlock Holmes Meets Signal Processing” Invited
talk @ ICME 2006
• Multimedia Forensics is not Computer Forensics
(2009, Bohem et al.)
• Recent Surveys on the field (Stamm 2013, Piva 2013)
• WIFS, TIFS, Dedicated workshop at major conference
(ACM MM, ICME, ICIAP), Special issues on Int. Journal

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Multimedia Forensics (in practice)

• Source Identification
• Integrity/Authenticity
• Enhancement/Restoration
• Interpretation and Content Analysis
– Plate Recognition
– Dynamic Reconstruction (car crashes, etc.)
– Antropomethric issues
– …

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 “Forensics Image (Video) analysis is
the application of IMAGE SCIENCE
and DOMAIN EXPERTISE to interpret
the content of an image or the image
itself in legal matters” (SWGIT –
www.fbi.gov)

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 https://www.swgde.org/

Recent documents:

• 2015-09-29 SWGDE Best Practices for Photographic Comparison for All

Disciplines

• 2015-09-29 SWGDE Image Processing Guidelines

• 2015-09-29 SWGDE Proposed Techniques for Advanced Data Recovery

from Security Digital Video Recorders v1-1

• 2015-09-29 SWGDE Training Guidelines for Video Analysis, Image

Analysis and Photography V1-1

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Fantasy/Fiction

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 CSI

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Reality

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
I Need That Plate! No Way...
 Fantasy
• We cannot introduce new information
(there is no data..)

• But we can enhance and extract only

something already present

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Understand When It’s Possible To Get
Something
• What is the minimum quality for video? “Minimum quality”
doesn’t exist.
• The success of the enhancement depends on several
factors:
Main goal (video captured with an HD camera but the
license plate we need to extract is too far away)
Technical related details: Resolution of the area of
interest, Level of compression, Presence of blur /
focus, Number of available frames, Noise / brightness
and contrast
• It’s important to understand which defects are
present in order to apply the proper tools.
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Perspective Correction

Geometrical transformation

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Forensics is based on the idea
that inherent traces (like digital fingerprints)
are left behind in a digital media during both
the creation phase and any other
successively process.

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Camera Ballistics
Which Device Has Created This Picture?

• Example:
• Forensic analysis of a smartphone: which pictures have been generated
on the device and which ones have been generated by other devices
and sent by messaging application or saved from the internet

• We can identify:
• Type of device
• Maker and model
• Specific exemplar

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Source Identification Noise Based
Sensor output carries not only pure signal
but also various noise components. Sensor
noise model could be used as a
representative feature for cameras.

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Sensor Identification Using
Pattern Noise

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Sensor Identification Using
Pattern Noise
This method provide good results, and is
quite reliable also using:
–images with different level of JPEG
compression (low, medium and high)
–images processed using point-wise operator
such as brightness/contrast adjustment or
gamma correction.
–images acquired by two cameras of the same
brand and model.

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Integrity: What is a Forgery?
• “Forgery” is a
subjective word.

• An image can
become a forgery
based upon the
context in which
it is used.

• An image altered for fun or someone who has taken an bad

photo, but has been altered to improve its appearance
cannot be considered a forgery even though it has been
altered from its original capture.

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
What is a Forgery?
• The other side of forgery are those who perpetuate a
forgery for gain and prestige

• They create an image in which to dupe the recipient into

believing the image is real and from this be able to gain
payment and fame

• Three type of forgery can be identified:

•An image that is created using graphical software
•An image where the content has been altered
•An image where the context has been altered

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
The content has been altered
Duping the recipient into believing that the objects in an
image are something else from what they really are!

November 1997: After 58 tourists were killed in a terrorist attack at the

temple of Hatshepsut in Luxor Egypt, the Swiss tabloid Blick digitally altered
a puddle of water to appear as blood flowing from the temple.

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
The context has been altered
• Objects are be removed or added,
for example, a person can be
added or removed
• The easiest way is to cut an
object from one image and insert
it into another image – image
editing software makes this a
simple task

• An example is this altered image

which could be used to influence
events in foreign countries which
are not aware of manipulation.

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Altering Images

The concepts have moved

into the digital world by
virtue of digital cameras
and the availability of
digital image editing
software

circa 1860: This nearly iconic portrait of U.S. President

Abraham Lincoln is a composite of Lincoln's head and the
Southern politician John Calhoun's body.

The ease of use of digital image editing software, which does

not require any special skills, makes image manipulation easy
to achieve.
27
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Off-side (February 2011)

Barcelona – Athletico Bilbao

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Sydney conman
who photoshopped
pictures of himself
with famous people
gets 12 years’jail
http://www.smh.com.au

31
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 More (and more) examples
Photo Tampering through History
http://www.fourandsix.com/photo-tampering-history/

Photoshopdisaster
http://www.photoshopdisasters.com/

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 /
http://petapixel.com/2013/05/08/how-photographers-photoshopped-their-pictures-back-in-1946
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Image Editing (2)

Piva 2013

 Malicious image editing alters the image semantic

content, mainly:
 Adding information
 Removing information

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Image Editing (2)
• Splicing (two images)
– Also called cut and paste, compositing
– Used to add information
• Cloning (single image)
– Also called copy and paste, copy move, region duplication
– Used to add or remove information
– Can be exact, or the clone can be resized, rotated…)
• Inpainting (kind of intelligent clone)
– Seam carving, content aware resize, content aware fill, content dependent crop
– Used to remove information
• Retouch (local editing)
– Dodge and burn, healing tool…
• Image enhancement/filtering
– Histogram equalization, contrast enhancement, median filtering, denoise, smooth…
• Image editing (geometric transformation)
– Resize, crop, zoom, shear

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Analog Vs. Digital
• Analogue evidence (audio, photo, video)
– Has always an original (negative or positive)
– Copies are never an exact replica of the original
– Copies of copies will be further deteriorated
• Digital evidence
– Only a finite sequence of number
– Exact copies can be made
– No loss of information between generations
– Every copy of a digital data can be considered the original,
unless is tagged in order to be distinguished by copies
Best evidence clause (Federal Rules of Evidence 1003): if data is stored
in a computer or similar device, any printout readable by sight, shown
36
to
reflect the data accurately, is an “original”.

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Original File: Special Cases
• Recapture: create a fake and then take a
picture with the camera we want to
pretend the picture was taken with

• Staging: the image file is authentic, but

the content has been staged

In these cases an authentic file does not

37

imply an authentic content.

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 How To Authenticate An Image?
• Visual Inspection
• File Analysis
 File Format and Structures
 Metadata (EXIF)
 Compression Parameters (Quantization
Tables)

• Global Analysis
 Pixel and compressed data statistics

• Local Analysis
 Finding inconsistencies of pixel statistics
across the image

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Image Forensics Methods
Active Methods: Hiding a mark or a
message in a picture when it is created:

WATERMARKING
Drawbacks:
• limitate to specially equipped
(=expensive) digital cameras;
• not so robust.
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Image Forensics Methods
Passive Methods: Using the alterations of
the underlying statistics produced by digital
forgeries on an image:

PHYSICS BASED
CAMERA BASED
PIXEL BASED
GEOMETRIC BASED
FORMAT BASED
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Acquisition

INTERPOLATION

ORIGINAL LENS CFA SENSOR

IMAGE
Processing and
Storage

POST
STORAGE DIGITAL IMAGE PROCESSING

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Types Of Analysis: Scene Level
• Use features of the scene, not of the image pixels and could work well also
on scanned photos
• Difficult to spoof
– Most forgeries contains slight errors not visible by human eye but detectable by
proper analysis
• Difficult to automatize the analysis
– Requires a lot of experience
• Better results on splicing, less successful on forgeries which are hiding
something
• Physics based:
– Lighting
• Geometry and perspective inconsistencies
– Principal point analysis
– Shadows
– Photogrammetry

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Types Of Analysis: Signal Level
Based on statistical features of pixel values; need good quality image

• Clone detection
– Cloned image blocks
– Similar couples of key points
• Resampling detection
– For resize, rotate, but also when splicing or cloning
• Enhancement Detection
– Specific for algorithms (median, histogram equalization, color
adjustment)
• Seam carving detection
• General intrinsic footprints
• Inconsistencies from acquisition and coding fingerprints
– CFA, PRNU, DCT, ELA…

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Format-Based
JPEG compression engine
(for both luminance and chrominance channels):
the input image is
partitioned into 8x8
non-overlapping blocks

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Format-Based

A JPEG image (luminance component only)

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Format-Based

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Format-Based
JPEG compression engine
(for both luminance and chrominance channels):
a DCT transform
the input image
is applied to each
is partitioned
block
into 8x8 non-
overlapping
blocks
Format-Based
JPEG forgery engine
 THE TYPICAL PIPELINE
FOR A COPY-PASTE
original image
QF(1) = q1
OPERATION
duplicating

+ 2nd image
QF(2) = q2

resizing

=
resulting image
QF(3) = q3
 Order of quantization

Top row: single quantized with steps 2 (left) and 3 (right).

Bottom row: double quantized with steps 3 followed by 2 (left), and 2
followed by 3 (right)
F. Galvan, G. Puglisi, A. R. Bruna, S. Battiato, First Quantization Matrix Estimation from Double Compressed JPEG
Images, IEEE Transactions on Information Forensics and Security, 2014.

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Current Trends And
Challenges

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Current Trends: Point&Shoot
and Share…
 Social MF on Facebook
Preliminar studies involving:
- Different devices
- Dataset of images:
- Different Scene (outdoor artificial,
outdoor natural, indoor)
- Different Quality (resolution and
compression)
- Different upload setting

has proven that some invariance could be

guaranteed.
Some specific editing could be then traced
and used to retrieve useful info about
image before uploading.

Moltisanti, Paratore, Battiato, Saravo - Image Manipulation on Facebook for Forensics

Evidence – ICIAP 2015, LNCS 2015;
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Social (Multimedia) Forensics
• Image and Video Phylogeny

ReVeal project

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Video Fingerprint
• Photo DNA
(http://www.microsoftphotodna.com/ )
• VideoGenome
(http://v-nome.org/ )
• Videntifier
(http://www.eff2.net/ )

73
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 PRNU on Video (online)
Each video is re-encoded (compressed) after it is uploaded. This
compression is needed to reduce the bandwidth usage by reducing the file
size and thus increasing streaming speeds.

By re-encoding the video, the video is altered which makes it harder to

identify the source.

Related performances depends on the brand of the camera, the resolution

and the amount of compression.

The reliability of the PRNU pattern decreases exponentially if the

compression becomes higher.

Challenges:
- Video obtained by some post processing (e.g., transition effects, fusion of
more than one source, …)
- No reference available
- Various Encoders
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Who Cares?

media
Who Cares?

geopolitics
…

…and political propaganda

 Advertisement

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
WorldPhotoPress 2012: faked?

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
When is an image fake, and when
is it merely enhanced?

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 WPP Report:The integrity of the
Image (Nov. 2014)
Current practices and accepted
standards relating to the
manipulation of still images in
photojournalism and documentary
photography.

Media organizations
prohibit the alteration of
images beyond
traditional darkroom
techniques.

It means – as first – that the

alteration of images – where
alteration means the digital
addition or subtraction of
elements isTruthfulness
Multimedia forbidden Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Data Manipulation on Science: Images

Source: Office For Research Integrity (ORI)

Gilbert, N. “Science Journals Crack down on Image Manipulation.” Nature (2009):

doi:10.1038/news.2009.991.
 Forgery on Biomedical Images

Corriere della Sera – Ottobre 2013

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Forgery on Science
Original Paper
Oriented Assembly of
Fe3O4 Nanoparticles into
Monodisperse Hollow
Single-Crystal Microspheres
Yu et al, J. Phys. Chem. B
2006, 110, 21667-21671
(Figure 3)

Plagiarized paper:
Fabrication of
Monodisperse Magnetic
Fe3O4-SiO2
Nanocomposites with Core-
Shell Structures Hua Fang,*
Chun-yang Ma, Tai-li Wan,
Mei Zhang, and Wei-hai Shi
J. Phys. Chem C 2007, 111,
1065-1070

M. Rossner and K. M. Yamada, “What’s in a picture? The temptation of image manipulation.,”

J. Cell Biol., vol. 166, no. 1, pp. 11–5, Jul. 2004.
 Forgery on Science

“What’s in a picture? The temptation of image manipulation.,” J. Cell Biol., vol. 166, no. 1, pp.
11–5, Jul. 2004.
Body-Worn camera
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Image Manipulation: Case “Mozzarella Blu”

Multimedia Truthfulness
Verification
Multimedia Truthfulness Verification in Legal in and
Environment Legal
Social Media - WIFS 2015
 Multimedia Truthfulness
Multimedia Truthfulness Verification in Verification in Legal
Legal Environment and Social Media - WIFS 2015
 Multimedia Truthfulness
Verification
Multimedia Truthfulness Verification in Legal Environment in Legal
and Social Media - WIFS 2015
 Evidence on the web

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 http://www.rewindproject.eu/
https://s-five.eu/
The first public draft of the FIVE Best
Practice Manual is publically available
from Oct. 9, 2015 ("October/DIWG2015
version"): DRAFT_BPM_FIVE_20151009
http://revealproject.eu/

http://maven-project.eu/#_=_

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Future of Imaging

Nikon
Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Sharing

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Conclusions
• Multimedia Forensics is now a
consolidated field but new intriguing
challenges emerge every day.
• The various involved actors are expected:
– To disseminate «basic knowledge»
– To share «best practices» and results
– To identify new challenges
– ..

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Survey

• Matthew C. Stamm, Min Wu and K. J. Ray Liu,

Information Forensics: An Overview of the First
Decade (2013), in: IEEE Access, 1(167-200)
• Alessandro Piva, An Overview on Image Forensics
(2013), in: ISRN Signal Processing, 2013 (Article ID
496701, 22 pages)

- C. Baron - Adobe Photoshop Forensics – Sleuths,

Thruts, and Fauxtography – Thomson Course
Tehcnology - 2009

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 On line Resources
Tutorial by Prof. Hany Farid - Digital Image Forensics:
lecture notes, exercises, and matlab code for a survey
course in digital image and video
forensics. http://www.cs.dartmouth.edu/farid/downloads/tutor
ials/digitalimageforensics.pdf

Special Issue on Multimedia in Forensics, Security and

Intelligence - IEEE Multimedia Magazine Vol. 19, Issue 1,
pp. 17-19, 2012

SOFTWARE: Amped5, Authenticate, Adroit, Four&Six, Izitru,

Ghiro, …

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Credits
• Amped (www.amped.it)

• Undergraduate course: Computer

Forensics – University of Catania
(http://www.dmi.unict.it/~battiato/CF.html)

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
 Prof. Sebastiano Battiato
Dipartimento di Matematica e Informatica
University of Catania, Italy

Image Processing LAB – http://iplab.dmi.unict.it

[email protected]
 Main Contacts
Further Info
Image Processing Lab
Università di Catania
www.dmi.unict.it/~iplab

Email
[email protected]

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015