Data Mining:
Concepts and Techniques
— Chapter 2 —
Descriptive Data Summarization
1
Types of Data Sets
Record
Relational records
Data matrix, e.g., numerical matrix,
timeout
season
coach
game
score
team
ball
lost
pla
crosstabs
wi
n
y
Document data: text documents: term-
frequency vector
Document 1 3 0 5 0 2 6 0 2 0 2
Transaction data
Graph and network Document 2 0 7 0 2 1 0 0 3 0 0
World Wide Web
Document 3 0 1 0 0 1 2 2 0 3 0
Social or information networks
Molecular Structures
Ordered TID Items
Video data: sequence of images
1 Bread, Coke, Milk
Temporal data: time-series
Sequential Data: transaction sequences 2 Beer, Bread
Genetic sequence data 3 Beer, Coke, Diaper, Milk
Spatial, image and multimedia: 4 Beer, Bread, Diaper, Milk
Spatial data: maps 5 Coke, Diaper, Milk
Image data:
Video data:
2
Data Objects
Data sets are made up of data objects.
A data object represents an entity.
Examples:
sales database: customers, store items, sales
medical database: patients, treatments
university database: students, professors, courses
Also called samples , examples, instances, data points,
objects, tuples.
Data objects are described by attributes.
Database rows -> data objects; columns ->attributes.
3
Attributes
Attribute (or dimensions, features, variables):
a data field, representing a characteristic or feature
of a data object.
E.g., customer _ID, name, address
Types:
Nominal
Binary
Ordinal
Numeric: quantitative
4
Attribute Types
Nominal: categories, states, or “names of things”
Hair_color = {auburn, black, blond, brown, grey, red, white }
marital status, occupation, ID numbers, zip codes
Binary
Nominal attribute with only 2 states (0 and 1)
Symmetric binary: both outcomes equally important
e.g., gender
Asymmetric binary: outcomes not equally important.
e.g., medical test (positive vs. negative)
Convention: assign 1 to most important outcome (e.g., HIV
positive)
Ordinal
Values have a meaningful order (ranking) but magnitude between
successive values is not known.
Size = {small, medium, large}, grades, army rankings
Numeric Attributes - is quantitative; that is, it is a measurable quantity,
represented in integer or real values.
5
Data Quality: Why Preprocess the Data?
Measures for data quality: A multidimensional view
Accuracy: correct or wrong, accurate or not
Completeness: not recorded, unavailable, …
Consistency: some modified but some not, dangling, …
Timeliness: timely update?
Believability: how trustable the data are correct?
Interpretability: how easily the data can be
understood?
6
DESCRIPTIVE DATA
SUMMARISATION
7
Descriptive data summarization
Essential to have overall picture of your data
Data summarization tech used to identify typical
properties of data
8
Basic Statistical Descriptions of Data
Motivation
To better understand the data: central tendency, variation and spread
Measure of central tendencies- Mean, Median, Mode etc
Measure of Data dispersion- Quartiles, Inter Quartile Range, Percentile
Distributive measure
A measure that can be computed by partitioning the data into smaller
subsets, computing the measure for each subset and then merging
the results. Eg sum, count
Algebraic measure
A measure that can be computed by applying algebraic fn to one or
more distributive measure. Eg Average= Sum/Count
Holistic Measure
A measure that is to be computed on entire set of data and cannot be
computed by partitioning data. Eq Median
9
Measuring the Central Tendency
1 n
Mean (algebraic measure)
x xi
Weighted arithmetic mean: n i 1
n
Trimmed mean: chopping extreme values
w x i i
x i 1
L1- lower bdy of median interval n
Median: N- total values in data set
i 1
wi
Middle value(ΣFreq)l-
if odd number of values, or average of median interval
summation of all freq lower than
Freg median- freq of the median interval
the middle two values otherwise
Width- width of median interval
Estimated by interpolation (for grouped data):
n / 2 ( freq )l
median L1 ( ) width
Mode freq median
Value that occurs most frequently in the data
Unimodal, bimodal, trimodal
10
Symmetric vs. Skewed Data
Median, mean and mode of symmetric, symmetric
positively and negatively skewed data
positively skewed negatively skewed
January 16, 2021 Data Mining: Concepts and Techniques 11
Measuring the Dispersion of Data
Quartiles, outliers and boxplots
Quartiles: Q1 (25th percentile), Q3 (75th percentile)
Inter-quartile range: IQR = Q3 – Q1
Five number summary: min, Q1, median, Q3, max
Boxplot: ends of the box are the quartiles; median is marked; add
whiskers, and plot outliers individually
Outlier: usually, a value higher/lower than 1.5 x IQR
Variance and standard deviation (sample: s, population: σ)
Variance: (algebraic, scalable computation)
1 n 1 n 2 1 n 1 n
1 n
s
2
n 1 i 1
( xi x )
2
[ xi ( xi ) 2 ]
n 1 i 1 n i 1
2
N
i 1
( xi
2
)
N
xi 2
i 1
2
Standard deviation s (or σ) is the square root of variance s2 (or σ2)
12
13
Properties of Normal Distribution Curve
The normal (distribution) curve
From μ–σ to μ+σ: contains about 68% of the
measurements (μ: mean, σ: standard deviation)
From μ–2σ to μ+2σ: contains about 95% of it
From μ–3σ to μ+3σ: contains about 99.7% of it
14
Boxplot Analysis
Five-number summary of a distribution
Minimum, Q1, Median, Q3, Maximum
Boxplot
Data is represented with a box
The ends of the box are at the first and third
quartiles, i.e., the height of the box is IQR
The median is marked by a line within the
box
Whiskers: two lines outside the box extended
to Minimum and Maximum
Outliers: points beyond a specified outlier
threshold, plotted individually
15
Visualization of Data Dispersion: 3-D Boxplots
January 16, 2021 Data Mining: Concepts and Techniques 16
