![Algebraic Derivation of PCs
Main steps for computing PCs
Form the covariance matrix S.
Compute its eigenvectors: {ai }i =1
d
The first p eigenvectors {a }
p
i i =1 form the p
PCs. G ← [a , a , L , a ]
1 2 p
The transformation G consists of the p PCs.
A test point x ∈ ℜ → G x ∈ ℜ .
d T p
64](https://image.slidesharecdn.com/dimensionalityreduction-110215050112-phpapp02/75/Dimensionality-Reduction-64-2048.jpg)
![Algebraic Derivation of CCA
In general, the k-th basis vectors are given
by the k–th eigenvector of
The two transformations are given by
[
WX = wx1 , wx 2 , L wxp ]
WY = [w y1 , wy 2 , L wyp ]
75](https://image.slidesharecdn.com/dimensionalityreduction-110215050112-phpapp02/75/Dimensionality-Reduction-75-2048.jpg)
![Feature Interaction
:
A set of features are interacting with each, if they
become more relevant when considered together than
considered individually.
A feature could lose its relevance due to the absence of
any other feature interacting with it, or irreducibility
[Jakulin05].
120](https://image.slidesharecdn.com/dimensionalityreduction-110215050112-phpapp02/75/Dimensionality-Reduction-120-2048.jpg)
![Handle Feature Interactions (INTERACT)
• Designing a feature
scoring metric based on
the consistency
hypothesis: c-contribution.
• Designing a data
structure to facilitate the
fast update of c-
contribution
• Selecting a simple and
fast search schema
• INTERACT is a backward
elimination algorithm
[Zhao-Liu07I]
124](https://image.slidesharecdn.com/dimensionalityreduction-110215050112-phpapp02/75/Dimensionality-Reduction-124-2048.jpg)
![Measure Feature Relevance
Transformation Function:
Relevance Measurement:
Construct cluster indicator from features.
Measure the fitness of the cluster indicator using both
labeled and unlabeled data.
sSelect algorithm uses spectral analysis [Zhao-Liu07S].
126](https://image.slidesharecdn.com/dimensionalityreduction-110215050112-phpapp02/75/Dimensionality-Reduction-126-2048.jpg)
More Related Content
![Dimension reduction techniques[Feature Selection]](https://cdn.slidesharecdn.com/ss_thumbnails/dimensionreductiontechnibyaakankshajain-210625102243-thumbnail.jpg?width=640&height=640&fit=bounds)
Dimensionality Reduction
- 1. Dimensionality Reduction forData Mining - Techniques, Applications and Trends Lei Yu Binghamton University Jieping Ye, Huan Liu Arizona State University
- 2. Outline Introduction todimensionality reduction Feature selection (part I) Basics Representative algorithms Recent advances Applications Feature extraction (part II) Recent trends in dimensionality reduction 2
- 3. Why Dimensionality Reduction? It is so easy and convenient to collect data An experiment Data is not collected only for data mining Data accumulates in an unprecedented speed Data preprocessing is an important part for effective machine learning and data mining Dimensionality reduction is an effective approach to downsizing data 3
- 4. Why Dimensionality Reduction? Most machine learning and data mining techniques may not be effective for high- dimensional data Curse of Dimensionality Query accuracy and efficiency degrade rapidly as the dimension increases. The intrinsic dimension may be small. For example, the number of genes responsible for a certain type of disease may be small. 4
- 5. Why Dimensionality Reduction? Visualization: projection of high-dimensional data onto 2D or 3D. Data compression: efficient storage and retrieval. Noise removal: positive effect on query accuracy. 5
- 6. Application of DimensionalityReduction Customer relationship management Text mining Image retrieval Microarray data analysis Protein classification Face recognition Handwritten digit recognition Intrusion detection 6
- 7. Document Classification Terms Web Pages Emails T1 T2 ….…… TN C D1 12 0 ….…… 6 Sports D2 3 10 ….…… 28 Travel Documents … … … … DM 0 11 ….…… 16 Jobs Internet Task: To classify unlabeled documents into categories ACM Portal IEEE Xplore PubMed Challenge: thousands of terms Solution: to apply Digital Libraries dimensionality reduction 7
- 8. Gene Expression MicroarrayAnalysis Expression Microarray Image Courtesy of Affymetrix Task: To classify novel samples into known disease types (disease diagnosis) Challenge: thousands of genes, few samples Solution: to apply dimensionality reduction Expression Microarray Data Set 8
- 9. Other Types ofHigh-Dimensional Data Face images Handwritten digits 9
- 10. Major Techniques ofDimensionality Reduction Feature selection Definition Objectives Feature Extraction (reduction) Definition Objectives Differences between the two techniques 10
- 11. Feature Selection Definition A process that chooses an optimal subset of features according to a objective function Objectives To reduce dimensionality and remove noise To improve mining performance Speed of learning Predictive accuracy Simplicity and comprehensibility of mined results 11
- 12. Feature Extraction Feature reduction refers to the mapping of the original high-dimensional data onto a lower- dimensional space Given a set of data points of p variables {x1 , x2 , L, xn } Compute their low-dimensional representation: xi ∈ ℜ d → yi ∈ ℜ p ( p << d ) Criterion for feature reduction can be different based on different problem settings. Unsupervised setting: minimize the information loss Supervised setting: maximize the class discrimination 12
- 13. Feature Reduction vs.Feature Selection Feature reduction All original features are used The transformed features are linear combinations of the original features Feature selection Only a subset of the original features are selected Continuous versus discrete 13
- 14. Outline Introduction todimensionality reduction Feature selection (part I) Basics Representative algorithms Recent advances Applications Feature extraction (part II) Recent trends in dimensionality reduction 14
- 15. Basics Definitionsof subset optimality Perspectives of feature selection Subset search and feature ranking Feature/subset evaluation measures Models: filter vs. wrapper Results validation and evaluation 15
- 16. Subset Optimality forClassification A minimum subset that is sufficient to construct a hypothesis consistent with the training examples (Almuallim and Dietterich, AAAI, 1991) Optimality is based on training set The optimal set may overfit the training data A minimum subset G such that P(C|G) is equal or as close as possible to P(C|F) (Koller and Sahami, ICML, 1996) Optimality is based on the entire population Only training part of the data is available 16
- 17. An Example forOptimal Subset Data set (whole set) F1 F2 F3 F4 F5 C Five Boolean features 0 0 1 0 1 0 C = F1∨F2 0 1 0 0 1 1 1 0 1 0 1 1 F3 = ┐F2 , F5 = ┐F4 1 1 0 0 1 1 Optimal subset: 0 0 1 1 0 0 {F1, F2} or {F1, F3} 0 1 0 1 0 1 Combinatorial nature 1 0 1 1 0 1 of searching for an 1 1 0 1 0 1 optimal subset 17
- 18. A Subset SearchProblem An example of search space (Kohavi & John 1997) Forward Backward 18
- 19. Different Aspects ofSearch Search starting points Empty set Full set Random point Search directions Sequential forward selection Sequential backward elimination Bidirectional generation Random generation 19
- 20. Different Aspects ofSearch (Cont’d) Search Strategies Exhaustive/complete search Heuristic search Nondeterministic search Combining search directions and strategies 20
- 21. Illustrations of SearchStrategies Depth-first search Breadth-first search 21
- 22. Feature Ranking Weightingand ranking individual features Selecting top-ranked ones for feature selection Advantages Efficient: O(N) in terms of dimensionality N Easy to implement Disadvantages Hard to determine the threshold Unable to consider correlation between features 22
- 23. Evaluation Measures forRanking and Selecting Features The goodness of a feature/feature subset is dependent on measures Various measures Information measures (Yu & Liu 2004, Jebara & Jaakkola 2000) Distance measures (Robnik & Kononenko 03, Pudil & Novovicov 98) Dependence measures (Hall 2000, Modrzejewski 1993) Consistency measures (Almuallim & Dietterich 94, Dash & Liu 03) Accuracy measures (Dash & Liu 2000, Kohavi&John 1997) 23
- 24. Illustrative Data Set Sunburn data Priors and class conditional probabilities 24
- 25. Information Measures Entropyof variable X Entropy of X after observing Y Information Gain 25
- 26. Consistency Measures Consistencymeasures Trying to find a minimum number of features that separate classes as consistently as the full set can An inconsistency is defined as two instances having the same feature values but different classes E.g., one inconsistency is found between instances i4 and i8 if we just look at the first two columns of the data table (Slide 24) 26
- 27. Accuracy Measures Usingclassification accuracy of a classifier as an evaluation measure Factors constraining the choice of measures Classifier being used The speed of building the classifier Compared with previous measures Directly aimed to improve accuracy Biased toward the classifier being used More time consuming 27
- 28. Models of FeatureSelection Filter model Separating feature selection from classifier learning Relying on general characteristics of data (information, distance, dependence, consistency) No bias toward any learning algorithm, fast Wrapper model Relying on a predetermined classification algorithm Using predictive accuracy as goodness measure High accuracy, computationally expensive 28
- 29. Filter Model 29
- 30.
- 31. How to ValidateSelection Results Direct evaluation (if we know a priori …) Often suitable for artificial data sets Based on prior knowledge about data Indirect evaluation (if we don’t know …) Often suitable for real-world data sets Based on a) number of features selected, b) performance on selected features (e.g., predictive accuracy, goodness of resulting clusters), and c) speed (Liu & Motoda 1998) 31
- 32. Methods for ResultEvaluation Accuracy For one ranked list Learning curves Number of Features For results in the form of a ranked list of features Before-and-after comparison For results in the form of a minimum subset Comparison using different classifiers To avoid learning bias of a particular classifier Repeating experimental results For non-deterministic results 32
- 33. Representative Algorithms forClassification Filter algorithms Feature ranking algorithms Example: Relief (Kira & Rendell 1992) Subset search algorithms Example: consistency-based algorithms Focus (Almuallim & Dietterich, 1994) Wrapper algorithms Feature ranking algorithms Example: SVM Subset search algorithms Example: RFE 33
- 34.
- 35.
- 36. Representative Algorithms forClustering Filter algorithms Example: a filter algorithm based on entropy measure (Dash et al., ICDM, 2002) Wrapper algorithms Example: FSSEM – a wrapper algorithm based on EM (expectation maximization) clustering algorithm (Dy and Brodley, ICML, 2000) 36
- 37. Effect of Featureson Clustering Example from (Dash et al., ICDM, 2002) Synthetic data in (3,2,1)-dimensional spaces 75 points in three dimensions Three clusters in F1-F2 dimensions Each cluster having 25 points 37
- 38. Two Different DistanceHistograms of Data Example from (Dash et al., ICDM, 2002) Synthetic data in 2-dimensional space Histograms record point-point distances For data with 20 clusters (left), the majority of the intra-cluster distances are smaller than the majority of the inter-cluster distances 38
- 39. An Entropy basedFilter Algorithm Basic ideas When clusters are very distinct, intra-cluster and inter-cluster distances are quite distinguishable Entropy is low if data has distinct clusters and high otherwise Entropy measure Substituting probability with distance Dij Entropy is 0.0 for minimum distance 0.0 or maximum 1.0 and is 1.0 for the mean distance 0.5 39
- 40. FSSEM Algorithm EMClustering To estimate the maximum likelihood mixture model parameters and the cluster probabilities of each data point Each data point belongs to every cluster with some probability Feature selection for EM Searching through feature subsets Applying EM on each candidate subset Evaluating goodness of each candidate subset based on the goodness of resulting clusters 40
- 41. Guideline for SelectingAlgorithms A unifying platform (Liu and Yu 2005) 41
- 42. Handling High-dimensional Data High-dimensional data As in gene expression microarray analysis, text categorization, … With hundreds to tens of thousands of features With many irrelevant and redundant features Recent research results Redundancy based feature selection Yu and Liu, ICML-2003, JMLR-2004 42
- 43. Limitations of ExistingMethods Individual feature evaluation Focusing on identifying relevant features without handling feature redundancy Time complexity: O(N) Feature subset evaluation Relying on minimum feature subset heuristics to implicitly handling redundancy while pursuing relevant features 2 Time complexity: at least O(N ) 43
- 44. Goals High effectiveness Able to handle both irrelevant and redundant features Not pure individual feature evaluation High efficiency Less costly than existing subset evaluation methods Not traditional heuristic search methods 44
- 45. Our Solution –A New Framework of Feature Selection A view of feature relevance and redundancy A traditional framework of feature selection A new framework of feature selection 45
- 46. Approximation Reasons forapproximation Searching for an optimal subset is combinatorial Over-searching on training data can cause over-fitting Two steps of approximation To approximately find the set of relevant features To approximately determine feature redundancy among relevant features Correlation-based measure C-correlation (feature Fi and class C) F-correlation (feature Fi and Fj ) Fi Fj C 46
- 47. Determining Redundancy F1 Hard to decide redundancy F2 F5 Redundancy criterion Which one to keep F3 F4 Approximate redundancy criterion Fj is redundant to Fi iff Fi Fj C SU(Fi , C) ≥ SU(Fj , C) and SU(Fi , Fj ) ≥ SU(F j , C) Predominant feature: not redundant to any feature in the current set F1 F2 F3 F4 F5 47
- 48. FCBF (Fast Correlation-BasedFilter) Step 1: Calculate SU value for each feature, order them, select relevant features based on a threshold Step 2: Start with the first feature to eliminate all features that are redundant to it Repeat Step 2 with the next remaining feature until the end of list F1 F2 F3 F4 F5 Step 1: O(N) Step 2: average case O(NlogN) 48
- 49. Real-World Applications Customer relationship management Ng and Liu, 2000 (NUS) Text categorization Yang and Pederson, 1997 (CMU) Forman, 2003 (HP Labs) Image retrieval Swets and Weng, 1995 (MSU) Dy et al., 2003 (Purdue University) Gene expression microarrray data analysis Golub et al., 1999 (MIT) Xing et al., 2001 (UC Berkeley) Intrusion detection Lee et al., 2000 (Columbia University) 49
- 50. Text Categorization Textcategorization Automatically assigning predefined categories to new text documents Of great importance given massive on-line text from WWW, Emails, digital libraries… Difficulty from high-dimensionality Each unique term (word or phrase) representing a feature in the original feature space Hundreds or thousands of unique terms for even a moderate-sized text collection Desirable to reduce the feature space without sacrificing categorization accuracy 50
- 51. Feature Selection inText Categorization A comparative study in (Yang and Pederson, ICML, 1997) 5 metrics evaluated and compared Document Frequency (DF), Information Gain (IG), Mutual Information (MU), X2 statistics (CHI), Term Strength (TS) IG and CHI performed the best Improved classification accuracy of k-NN achieved after removal of up to 98% unique terms by IG Another study in (Forman, JMLR, 2003) 12 metrics evaluated on 229 categorization problems A new metric, Bi-Normal Separation, outperformed others and improved accuracy of SVMs 51
- 52. Content-Based Image Retrieval(CBIR) Image retrieval An explosion of image collections from scientific, civil, military equipments Necessary to index the images for efficient retrieval Content-based image retrieval (CBIR) Instead of indexing images based on textual descriptions (e.g., keywords, captions) Indexing images based on visual contents (e.g., color, texture, shape) Traditional methods for CBIR Using all indexes (features) to compare images Hard to scale to large size image collections 52
- 53. Feature Selection inCBIR An application in (Swets and Weng, ISCV, 1995) A large database of widely varying real-world objects in natural settings Selecting relevant features to index images for efficient retrieval Another application in (Dy et al., Trans. PRMI, 2003) A database of high resolution computed tomography lung images FSSEM algorithm applied to select critical characterizing features Retrieval precision improved based on selected features 53
- 54. Gene Expression MicroarrayAnalysis Microarray technology Enabling simultaneously measuring the expression levels for thousands of genes in a single experiment Providing new opportunities and challenges for data mining Microarray data 54
- 55. Motivation for Gene(Feature) Selection Data mining tasks Data characteristics in sample classification High dimensionality (thousands of genes) Small sample size (often less than 100 samples) Problems Curse of dimensionality Overfitting the training data 55
- 56. Feature Selection inSample Classification An application in (Golub, Science, 1999) On leukemia data (7129 genes, 72 samples) Feature ranking method based on linear correlation Classification accuracy improved by 50 top genes Another application in (Xing et al., ICML, 2001) A hybrid of filter and wrapper method Selecting best subset of each cardinality based on information gain ranking and Markov blanket filtering Comparing between subsets of the same cardinality using cross-validation Accuracy improvements observed on the same leukemia data 56
- 57. Intrusion Detection viaData Mining Network-based computer systems Playing increasingly vital roles in modern society Targets of attacks from enemies and criminals Intrusion detection is one way to protect computer systems A data mining framework for intrusion detection in (Lee et al., AI Review, 2000) Audit data analyzed using data mining algorithms to obtain frequent activity patterns Classifiers based on selected features used to classify an observed system activity as “legitimate” or “intrusive” 57
- 58. Dimensionality Reduction forData Mining - Techniques, Applications and Trends (Part II) Lei Yu Binghamton University Jieping Ye, Huan Liu Arizona State University
- 59. Outline Introduction todimensionality reduction Feature selection (part I) Feature extraction (part II) Basics Representative algorithms Recent advances Applications Recent trends in dimensionality reduction 59
- 60. Feature Reduction Algorithms Unsupervised Latent Semantic Indexing (LSI): truncated SVD Independent Component Analysis (ICA) Principal Component Analysis (PCA) Manifold learning algorithms Supervised Linear Discriminant Analysis (LDA) Canonical Correlation Analysis (CCA) Partial Least Squares (PLS) Semi-supervised 60
- 61. Feature Reduction Algorithms Linear Latent Semantic Indexing (LSI): truncated SVD Principal Component Analysis (PCA) Linear Discriminant Analysis (LDA) Canonical Correlation Analysis (CCA) Partial Least Squares (PLS) Nonlinear Nonlinear feature reduction using kernels Manifold learning 61
- 62. Principal Component Analysis Principalcomponent analysis (PCA) Reduce the dimensionality of a data set by finding a new set of variables, smaller than the original set of variables Retains most of the sample's information. By information we mean the variation present in the sample, given by the correlations between the original variables. The new variables, called principal components (PCs), are uncorrelated, and are ordered by the fraction of the total information each retains. 62
- 63. Geometric Picture ofPrincipal Components (PCs) z1 • the 1st PC z1 is a minimum distance fit to a line in X space • the 2nd PC z 2 is a minimum distance fit to a line in the plane perpendicular to the 1st PC PCs are a series of linear least squares fits to a sample, each orthogonal to all the previous. 63
- 64. Algebraic Derivation ofPCs Main steps for computing PCs Form the covariance matrix S. Compute its eigenvectors: {ai }i =1 d The first p eigenvectors {a } p i i =1 form the p PCs. G ← [a , a , L , a ] 1 2 p The transformation G consists of the p PCs. A test point x ∈ ℜ → G x ∈ ℜ . d T p 64
- 65. Optimality Property ofPCA Main theoretical result: The matrix G consisting of the first p eigenvectors of the covariance matrix S solves the following min problem: 2 min G∈ℜ d× p X − G (G X ) T subject to G T G = I p F 2 X −X reconstruction error F PCA projection minimizes the reconstruction error among all linear projections of size p. 65
- 66. Applications of PCA Eigenfaces for recognition. Turk and Pentland. 1991. Principal Component Analysis for clustering gene expression data. Yeung and Ruzzo. 2001. Probabilistic Disease Classification of Expression-Dependent Proteomic Data from Mass Spectrometry of Human Serum. Lilien. 2003. 66
- 67. Motivation for Non-linearPCA using Kernels Linear projections will not detect the pattern. 67
- 68. Nonlinear PCA usingKernels Traditional PCA applies linear transformation May not be effective for nonlinear data Solution: apply nonlinear transformation to potentially very high-dimensional space. φ : x → φ ( x) Computational efficiency: apply the kernel trick. Require PCA can be rewritten in terms of dot product. K ( xi , x j ) = φ ( xi ) • φ ( x j ) 68
- 69. Canonical Correlation Analysis(CCA) CCA was developed first by H. Hotelling. H. Hotelling. Relations between two sets of variates. Biometrika, 28:321-377, 1936. CCA measures the linear relationship between two multidimensional variables. CCA finds two bases, one for each variable, that are optimal with respect to correlations. Applications in economics, medical studies, bioinformatics and other areas. 69
- 70. Canonical Correlation Analysis(CCA) Two multidimensional variables Two different measurement on the same set of objects Web images and associated text Protein (or gene) sequences and related literature (text) Protein sequence and corresponding gene expression In classification: feature vector and class label Two measurements on the same object are likely to be correlated. May not be obvious on the original measurements. Find the maximum correlation on transformed space. 70
- 71. Canonical Correlation Analysis(CCA) Correlation XT WX Transformed data measurement transformation YT WY 71
- 72. Problem Definition Find two sets of basis vectors, one for x and the other for y, such that the correlations between the projections of the variables onto these basis vectors are maximized. Given Compute two basis vectors wx and wy : y → < wy , y > 72
- 73. Problem Definition Compute the two basis vectors so that the correlations of the projections onto these vectors are maximized. 73
- 74. Algebraic Derivation ofCCA The optimization problem is equivalent to C xy = XY T , C xx = XX T where C yx = YX T , C yy = YY T 74
- 75. Algebraic Derivation ofCCA In general, the k-th basis vectors are given by the k–th eigenvector of The two transformations are given by [ WX = wx1 , wx 2 , L wxp ] WY = [w y1 , wy 2 , L wyp ] 75
- 76. Nonlinear CCA usingKernels Key: rewrite the CCA formulation in terms of inner products. C xx = XX T C xy = XY T α X XY Yβ T T T ρ = max α ,β α T X T XX T Xα β TY T YYT Yβ wx = Xα wy = Yβ Only inner products Appear 76
- 77. Applications in Bioinformatics CCA can be extended to multiple views of the data Multiple (larger than 2) data sources Two different ways to combine different data sources Multiple CCA Consider all pairwise correlations Integrated CCA Divide into two disjoint sources 77
- 78. Applications in Bioinformatics Source:Extraction of Correlated Gene Clusters from Multiple Genomic Data by Generalized Kernel Canonical Correlation Analysis. ISMB’03 http://cg.ensmp.fr/~vert/publi/ismb03/ismb03.pdf 78
- 79. Multidimensional scaling (MDS) •MDS: Multidimensional scaling • Borg and Groenen, 1997 • MDS takes a matrix of pair-wise distances and gives a mapping to Rd. It finds an embedding that preserves the interpoint distances, equivalent to PCA when those distance are Euclidean. • Low dimensional data for visualization 79
- 80. Classical MDS ( D = xi − x j 2 ) : distancematrix ij Centering matrix : ⇒ P DP = −2(( xi − μ ) • ( x j − μ ))ij 1 T e e P = I − ee e n 80
- 81. Classical MDS (Geometric Methods for Feature Extraction and Dimensional Reduction – Burges, 2005) ( D = xi − x j 2 ) : distance matrix ⇒ P DP = −2(( x − μ ) • ( x − μ )) ij e e i j ij Problem : Given D, how to find xi ? − = D = U d Σ dU = (U d Σ )(U d Σ ) P e DP e T 0.5 0.5 T 2 d d d ⇒ Choose xi , for i = 1,L, n, from the rows of U d Σ0.5 d 81
- 82. Classical MDS If Euclidean distance is used in constructing D, MDS is equivalent to PCA. The dimension in the embedded space is d, if the rank equals to d. If only the first p eigenvalues are important (in terms of magnitude), we can truncate the eigen-decomposition and keep the first p eigenvalues only. Approximation error 82
- 83. Classical MDS So far, we focus on classical MDS, assuming D is the squared distance matrix. Metric scaling How to deal with more general dissimilarity measures Non-metric scaling Metric scaling : − P e DPe = 2(( xi − μ ) • ( x j − μ ) )ij Nonmetricscaling : − P e DPe may not be positibe semi - definite Solutions: (1) Add a large constant to its diagonal. (2) Find its nearest positive semi-definite matrix by setting all negative eigenvalues to zero. 83
- 84. Manifold Learning Discoverlow dimensional representations (smooth manifold) for data in high dimension. A manifold is a topological space which is locally Euclidean An example of nonlinear manifold: 84
- 85. Deficiencies of LinearMethods Data may not be best summarized by linear combination of features Example: PCA cannot discover 1D structure of a helix 20 15 10 5 0 1 0.5 1 0 0.5 0 -0.5 -0.5 -1 -1 85
- 86. Intuition: how doesyour brain store these pictures? 86
- 87.
- 88. Brain Representation Every pixel? Or perceptually meaningful structure? Up-down pose Left-right pose Lighting direction So, your brain successfully reduced the high- dimensional inputs to an intrinsically 3- dimensional manifold! 88
- 89. Nonlinear Approaches- Isomap Josh. Tenenbaum, Vin de Silva, John langford 2000 Constructing neighbourhood graph G For each pair of points in G, Computing shortest path distances ---- geodesic distances. Use Classical MDS with geodesic distances. Euclidean distance Geodesic distance 89
- 90. Sample Points withSwiss Roll Altogether there are 20,000 points in the “Swiss roll” data set. We sample 1000 out of 20,000. 90
- 91. Construct Neighborhood GraphG K- nearest neighborhood (K=7) DG is 1000 by 1000 (Euclidean) distance matrix of two neighbors (figure A) 91
- 92. Compute All-Points ShortestPath in G Now DG is 1000 by 1000 geodesic distance matrix of two arbitrary points along the manifold (figure B) 92
- 93. Use MDS toEmbed Graph in Rd Find a d-dimensional Euclidean space Y (Figure c) to preserve the pariwise diatances. 93
- 94.
- 95. Isomap: Advantages • Nonlinear •Globally optimal • Still produces globally optimal low-dimensional Euclidean representation even though input space is highly folded, twisted, or curved. • Guarantee asymptotically to recover the true dimensionality. 95
- 96. Isomap: Disadvantages • Maynot be stable, dependent on topology of data • Guaranteed asymptotically to recover geometric structure of nonlinear manifolds – As N increases, pairwise distances provide better approximations to geodesics, but cost more computation – If N is small, geodesic distances will be very inaccurate. 96
- 97. Characterictics of aManifold Rn M z Locally it is a linear patch Key: how to combine all local patches together? x: coordinate for z R2 x2 x x1 97
- 98. LLE: Intuition Assumption: manifold is approximately “linear” when viewed locally, that is, in a small neighborhood Approximation error, e(W), can be made small Local neighborhood is effected by the constraint Wij=0 if zi is not a neighbor of zj A good projection should preserve this local geometric property as much as possible 98
- 99. LLE: Intuition We expect each data point and its neighbors to lie on or close to a locally linear patch of the manifold. Each point can be written as a linear combination of its neighbors. The weights chosen to minimize the reconstruction Error. 99
- 100. LLE: Intuition The weightsthat minimize the reconstruction errors are invariant to rotation, rescaling and translation of the data points. Invariance to translation is enforced by adding the constraint that the weights sum to one. The weights characterize the intrinsic geometric properties of each neighborhood. The same weights that reconstruct the data points in D dimensions should reconstruct it in the manifold in d dimensions. Local geometry is preserved 100
- 101. LLE: Intuition Low-dimensional embedding the i-th row of W Use the same weights from the original space 101
- 102. Local Linear Embedding(LLE) Assumption: manifold is approximately “linear” when viewed locally, that is, in a small neighborhood Approximation error, ε(W), can be made small Meaning of W: a linear representation of every data point by its neighbors This is an intrinsic geometrical property of the manifold A good projection should preserve this geometric property as much as possible 102
- 103. Constrained Least SquareProblem Compute the optimal weight for each point individually: Neightbors of x Zero for all non-neighbors of x 103
- 104. Finding a Mapto a Lower Dimensional Space Yi in Rk: projected vector for Xi The geometrical property is best preserved if the error below is small Use the same weights computed above Y is given by the eigenvectors of the lowest d non-zero eigenvalues of the matrix 104
- 105.
- 106. Examples Images of facesmapped into the embedding space described by the first two coordinates of LLE. Representative faces are shown next to circled points. The bottom images correspond to points along the top-right path (linked by solid line) illustrating one particular mode of variability in pose and expression. 106
- 107.
- 108. Laplacian Eigenmaps Laplacian Eigenmaps for Dimensionality Reduction and Data Representation M. Belkin, P. Niyogi Key steps Build the adjacency graph Choose the weights for edges in the graph (similarity) Eigen-decomposition of the graph laplacian Form the low-dimensional embedding 108
- 109. Step 1: AdjacencyGraph Construction 109
- 110. Step 2: Choosingthe Weight 110
- 111.
- 112.
- 113. Justification Consider theproblem of mapping the graph to a line so that pairs of points with large similarity (weight) stay as close as possible. xi → yi A reasonable criterion for choosing the mapping is to minimize 113
- 114. Justification 114
- 115. An Example 115
- 116. A Unified frameworkfor ML Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering. Bengio et al., 2004 116
- 117. Flowchart of theUnified Framework Construct neighborhood Graph (K NN) Construct the embedding Form similarity matrix M based on the eigenvectors Normalize M to Compute the eigenvectors of optional 117
- 118. Outline Introduction todimensionality reduction Feature selection (part I) Feature extraction (part II) Basics Representative algorithms Recent advances Applications Recent trends in dimensionality reduction 118
- 119. Trends in DimensionalityReduction Dimensionality reduction for complex data Biological data Streaming data Incorporating prior knowledge Semi-supervised dimensionality reduction Combining feature selection with extraction Develop new methods which achieve feature “selection” while efficiently considering feature interaction among all original features 119
- 120. Feature Interaction : A set of features are interacting with each, if they become more relevant when considered together than considered individually. A feature could lose its relevance due to the absence of any other feature interacting with it, or irreducibility [Jakulin05]. 120
- 121. Feature Interaction Two examples of feature interaction: MONK1 & Corral data. SU(C,A1)=0 SU(C,A2)=0 MONK1: Y :(A1=A2)V(A5==1) SU(C,A1&A2) Feature =0.22 Interaction Corral: Y :(A0^A1)V(B0^B1) Existing efficient feature selection algorithms can not handle feature interaction very well 121
- 122. Illustration using syntheticdata MONKs data, for class C = 1 (1) MONK1:(A1 = A2) or (A5 = 1); (2) MONK2: Exactly two Ai = 1; (all features are relevant) (3) MONK3: (A5 = 3 and A4 = 1) or (A5 ≠4 and A2 ≠ 3) Experiment with FCBF, ReliefF, CFS, FOCUS 122
- 123. Existing Solutions forFeature Interaction Existing efficient feature selection algorithms usually assume feature independence. Others attempt to explicitly address Feature Interactions by finding them. Find out all Feature Interaction is impractical. Some existing efficient algorithm can only (partially) address low order Feature Interaction, 2 or 3-way Feature Interaction. 123
- 124. Handle Feature Interactions(INTERACT) • Designing a feature scoring metric based on the consistency hypothesis: c-contribution. • Designing a data structure to facilitate the fast update of c- contribution • Selecting a simple and fast search schema • INTERACT is a backward elimination algorithm [Zhao-Liu07I] 124
- 125. Semi-supervised Feature Selection : For handling small labeled-sample problem Labeled data is few, but unlabeled data is abundant Neither supervised nor unsupervised works well Using both labeled and unlabeled data 125
- 126. Measure Feature Relevance Transformation Function: Relevance Measurement: Construct cluster indicator from features. Measure the fitness of the cluster indicator using both labeled and unlabeled data. sSelect algorithm uses spectral analysis [Zhao-Liu07S]. 126
- 127. References 127
- 128. References 128
- 129. References 129
- 130. References 130
- 131. References 131
- 132. References 132
- 133. References 133
- 134. References 134
- 135. Reference Z. Zhao,H. Liu, Searching for Interacting Features, IJCAI 2007 A. Jakulin, Machine learning based on attribute interactions, Ph.D. thesis, University of Ljubljana 2005. Z. Zhao, H. Liu, Semi-supervised Feature Selection via Spectral Analysis, SDM 2007 135