LDA GDA Comparison References Matlab Examples

○○ ○○○○○○○○○ ○ ○ ○○○

Generalized Discriminant Analysis

Jinyong Kang

Generalized Discriminant Analysis

LDA GDA Comparison References Matlab Examples
●
○○ ○○○○○○○○○ ○ ○ ○○○

Linear Discriminant Analysis

Fisher’s Linear Discriminant

Projection from d-dimensional space to (c-1) dimensional space

: vector form

: matrix equation

Generalized Discriminant Analysis 2/17

LDA GDA Comparison References Matlab Examples
○○
● ○○○○○○○○○ ○ ○ ○○○

Find matrix W that maximize J(W)

Solve

Generalized Discriminant Analysis 3/17

LDA GDA Comparison References Matlab Examples
○○ ○○○○○○○○○
● ○ ○ ○○○

Generalized Discriminant Analysis

Linear Discriminant Analysis

– Powerful method for classification
– But, the LDA method fails for a nonlinear problem

Generalized Discriminant Analysis

– Method for obtaining a nonlinear extension of the LDA
– Kernel-based method (Kernel LDA)
– Map the data into a higher dimensional feature space F
– And then perform the LDA algorithm in F instead of the original input space

The LDA algorithm can be reformulated into dot product form in F

– Easy to compute scalar products in some feature space using kernel functions

Generalized Discriminant Analysis 4/17

LDA GDA Comparison References Matlab Examples
○○ ○○○○○○○○○
● ○ ○ ○○○

Generalized Discriminant Analysis

Kernel-based algorithm

– d-dimensional data

– c different classes

Nonlinear mapping into a high dimensional feature space

LDA can be performed in on the set

Generalized Discriminant Analysis 5/17

LDA GDA Comparison References Matlab Examples
○○ ○○○○○○○○○
● ○ ○ ○○○

Between-class scatter matrix

: the number of data

: the number of data of the class
Within-class scatter matrix

Total class scatter matrix

Generalized Discriminant Analysis 6/17

LDA GDA Comparison References Matlab Examples
○○ ○○○○○○○○○
● ○ ○ ○○○

GDA Method In Feature Space

Projection from d-dimensional space to (c-1) dimensional space

Find matrix W that maximize J(W)

K is a matrix composed of dot

product in the feature space F
(Kernel Matrix, Gram Matrix)

: any vector w must lie in the span of all training samples in F

The largest eigenvalue gives the maximum of the following quotient of the inertia
: diagonal matrix of non-zero
eigenvalues
: matrix of eigenvectors
: nxn block diagonal matrix
Appendix B in [1]
: nlxnl matrix with all terms equal to 1/nl

Generalized Discriminant Analysis 7/17

LDA GDA Comparison References Matlab Examples
○○ ○○○○○○○○○
● ○ ○ ○○○

Kernel Matrix K

: total number of data

Generalized Discriminant Analysis 8/17

LDA GDA Comparison References Matlab Examples
○○ ○○○○○○○○○
● ○ ○ ○○○

Kernel Examples

Kernel

Calculate the dot product in

Do not need to map to

Generalized Discriminant Analysis 9/17

LDA GDA Comparison References Matlab Examples
○○ ●
○○○○○○○○○ ○ ○ ○○○

Kernel Examples

Polynomial kernels

dimension of a
d=2
feature space = 6

Radial basis functions (Gaussian)

The corresponding dimension of feature space is infinite

Neural network type kernels

Generalized Discriminant Analysis 10/17

LDA GDA Comparison References Matlab Examples
○○ ○○○○○○○○○
● ○ ○ ○○○

Eigenvalue Resolution

Normalization

Generalized Discriminant Analysis 11/17

LDA GDA Comparison References Matlab Examples
○○ ○○○○○○○○○
● ○ ○ ○○○

Summary - GDA Procedure

Compute the matrices K and L

Decompose K using eigenvectors decomposition

Compute eigenvectors and eigenvalues

Normalize

Compute projections

Generalized Discriminant Analysis 12/17

LDA GDA Comparison References Matlab Examples
○○ ○○○○○○○○○ ●
○ ○ ○○○

Comparison of GDA

Generalized Discriminant Analysis 13/17

LDA GDA Comparison References Matlab Examples
○○ ○○○○○○○○○ ○ ○
● ○○○

References

[1] G. Baudat, F. Anouar. “Generalized discriminant analysis using a kernel

approach,” in Neural Computation, vol. 12, no. 10,pp 2385-2404, 2000.
[2] F. Abdallah, C. Richard, R. Lengelle, “A Sequential Approach For Multi-Class
Discriminant Analysis With Kernels,” ICASSP ’04.

Generalized Discriminant Analysis 14/17

LDA GDA Comparison References Matlab Examples
○○ ○○○○○○○○○ ○ ○ ○○○
●

Matlab Examples

Matlab Toolbox for Dimensionality Reduction (v0.7.2 - November 2010)

– http://homepage.tudelft.nl/19j49/Matlab_Toolbox_for_Dimensionality_Reduction.html

Generalized Discriminant Analysis 15/17

LDA GDA Comparison References Matlab Examples
○○ ○○○○○○○○○ ○ ○ ○○○
●

SVM and Kernel Methods Matlab Toolbox

– http://asi.insa-rouen.fr/enseignants/~arakotom/toolbox/index.html

Generalized Discriminant Analysis 16/17

LDA GDA Comparison References Matlab Examples
○○ ○○○○○○○○○ ○ ○ ○○○
●

Example 1 : Example 2 :

Input Data
Input Data
20
7
15
6
10
5

5 4

0 3

2
-5
1
-10
0
-15
-1

-20 -2
-25 -20 -15 -10 -5 0 5 10 15 20 25 -4 -2 0 2 4 6

Results of PCA, LDA, Kernel PCA, and GDA ?

Generalized Discriminant Analysis 17/17