arXiv is now an independent nonprofit! Learn more
License: arXiv.org perpetual non-exclusive license
arXiv:1809.04185v2 [cs.CV] 25 Dec 2018

Deep Micro-Dictionary Learning and Coding Network

Hao Tang Affiliation: Department of Information Engineering and Computer Science, University of Trento, Trento, Italy    Heng Wei Affiliation: Department of Electrical Engineering, Hong Kong Polytechnic University, Hong Kong, China    Wei Xiao Affiliation: Lingxi Artificial Intelligence Co., Ltd, Shen Zhen, China    Wei Wang Affiliation: Computer Vision Laboratory, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland    Dan Xu Affiliation: Department of Engineering Science, University of Oxford, Oxford, UK    Yan Yan Affiliation: Department of Computer Science, Texas State University, San Marcos, USA{hao.tang, niculae.sebe}@unitn.it, 15102924d@connect.polyu.hk, xiaoweithu@163.comwei.wang@epfl.ch, danxu@robots.ox.ac.uk, y_y34@txstate.edu    Nicu Sebe Affiliation: Department of Information Engineering and Computer Science, University of Trento, Trento, Italy
Abstract

In this paper, we propose a novel Deep Micro-Dictionary Learning and Coding Network (DDLCN). DDLCN has most of the standard deep learning layers (pooling, fully, connected, input/output, etc.) but the main difference is that the fundamental convolutional layers are replaced by novel compound dictionary learning and coding layers. The dictionary learning layer learns an over-complete dictionary for the input training data. At the deep coding layer, a locality constraint is added to guarantee that the activated dictionary bases are close to each other. Next, the activated dictionary atoms are assembled together and passed to the next compound dictionary learning and coding layers. In this way, the activated atoms in the first layer can be represented by the deeper atoms in the second dictionary. Intuitively, the second dictionary is designed to learn the fine-grained components which are shared among the input dictionary atoms. In this way, a more informative and discriminative low-level representation of the dictionary atoms can be obtained. We empirically compare the proposed DDLCN with several dictionary learning methods and deep learning architectures. The experimental results on four popular benchmark datasets demonstrate that the proposed DDLCN achieves competitive results compared with state-of-the-art approaches.

Refer to caption
Figure 1: Multiple local coordinates and “fake” anchor points. Suppose that we want to encode the input signal 𝒚i∈ℝm\bm{y}_{i}{\in}\mathbb{R}^{m} in order to find the nonlinear function defined on it, for instance with three local coordinates in the same space ℝm\mathbb{R}^{m}. In fact, 𝒚i\bm{y}_{i} relies on the manifold that can be described by the 3r​d3^{rd} local coordinate, but unfortunately, we do not have enough atoms (anchor points) on the 3r​d3^{rd} manifold close to it. Therefore, some nearby atoms (“fake” anchor points) from the 1s​t1^{st} or 2n​d2^{nd} local coordinate will “kidnap” 𝒚i\bm{y}_{i}, overlooking in this way the true coordinate (3r​d3^{rd}), where 𝒚i\bm{y}_{i} really resides. This happens due to the small numbers of training samples and/or a large variety of signals, arriving at overfitting even though the learned model fits the training samples well.
Refer to caption
Figure 2: The pipeline of the Deep Micro-Dictionary Learning and Coding Network (DDLCN). The idea of DDLCN comes from the architectures of CNNs, while the difference from CNNs is that the convolutional layers in CNNs are replaced by our compound dictionary learning and coding layers.

1 Introduction

In the past few years, the most popular representation learning frameworks are dictionary learning and deep learning. Dictionary learning aims at learning a set of atoms such that a given feature can be well approximated by a sparse linear combination of these atoms, while deep learning methods focus on extracting semantic features via a deep network. So far most studies in dictionary learning employ a shallow (single layer) architecture, e.g., currently popular dictionary learning techniques are K-SVD [1], Discriminative K-SVD (D-KSVD) [57] and Label Consistent K-SVD (LC-KSVD) [20] which decompose the training data into a dense basis and sparse coefficients. In addition, both Local Coordinate Coding (LCC) [55, 54] and its fast implementation algorithm [43] are traditional dictionary learning methods. LCC and Locality Constrained Coding (LLC) [43] are based on the empirical observation that the sparse representations tend to be “local”. In other words, nonzero coefficients are often assigned to the atoms nearby to the encoded signal 𝒚\bm{y}. However, LLC has a major disadvantage: to achieve higher approximation, one has to use a large number of so-called “anchor points” to make a better linear approximation of the signal. Since LLC is a local linear approximation of a complex signal 𝒚i\bm{y}_{i}, for a nonlinear function on 𝒚i\bm{y}_{i} the local linear approximation may not necessarily be optimal. It means that the anchor points need to provide higher approximation power, allowing some of them to not necessary be “real” local anchors on the manifold where 𝒚i\bm{y}_{i} resides. In this context, our goal is to equip anchors with more descriptive power for better approximating 𝒚i\bm{y}_{i} in order to finally make more accurate inferences from it. An illustrative example is shown in Figure 1.

Recent work [40] has shown that deeper architectures can be built from dictionary learning. Chun et al. [8] present a Block Proximal Gradient method using a Majorizer for convolutional dictionary learning. Hu et al. [18] propose a nonlinear dictionary learning method and apply it to image classification task. Xiao et al. [46] propose a two-layer local coordinate coding framework for object recognition task. Zhang et al. [58] introduce an analysis discriminative dictionary learning framework for image classification task. Nguyen et al. [30] propose a domain adaptation framework using a sparse and hierarchical network, which shares the ideas with our work. However, our DDLCN is different from [30] in two ways: (i) Our dictionary is learned from features and then the learned dictionary acts as a candidate pool for the next layer dictionary. Our dictionaries from different layers have connections while in [30] which used a fixed dictionary in different layers, i.e., there is no message passing between the dictionaries of different layers; (ii) To represent an atom in the previous layer, we pick out a few atoms in the next layer and linearly combine them. These atoms have a linear contribution in constructing the atom in the previous layer. This is vital for the diversity, and in this way could incorporate more information into the next layer’s codes and alleviate the influence of incorrect atoms. However, there is no such mechanism in [30].

Inspired by both dictionary and deep learning, the goal of this paper is to improve the deep representation ability of dictionary learning. To address this problem, we present a novel network, named Deep Micro-Dictionary Learning and Coding Network (DDLCN), which is composed of several layers: input, feature extraction, dictionary learning, feature coding, pooling, fully connected and output layer as shown in Figure 2. The idea of the DDLCN comes from the standard architecture of Convolutional Neural Networks (CNNs), the biggest difference being that the convolutional layers in CNNs are replaced by our compound dictionary learning and coding layers. In this way, edges, lines and corners can be learned from the shallow layers which correspond to the shallow dictionaries. The more complicated “hierarchical” patterns/features can be obtained from deeper dictionaries.

DDLCN takes advantage of the manifold geometric structure of the underlying data to locally embed points from the underlying data manifold into a lower dimensional deep structural space. The benefit of DDLCN is that the learned feature representation after the feature learning and coding layers has a better approximation capability of the original data, in other words, the deep dictionary learning structure can fully exploit the space where the data reside. Meanwhile, the deep dictionary structure is very flexible, making it possible to use a micro dictionary, e.g., we can learn only one dictionary item per category.

Our contributions are summarized as follows:

  • •

    We propose a novel compound dictionary learning and coding layer, which has the similar function as the convolutional layer in the standard deep learning architecture.

  • •

    We present a new deep dictionary learning framework named Deep Micro-Dictionary Learning and Coding Network (DDLCN), which combines the advantages of dictionary and deep learning methods.

  • •

    Exhaustive experiments on a broader range of datasets have been conducted, demonstrating that the proposed layer and framework outperform the existing dictionary learning methods and achieve competitive results compared with deep learning approaches.

2 The Proposed DDLCN

In this section, we sequentially introduce each layer of DDLCN. For simplicity, we provide details on two layers of dictionary learning and coding of the DDLCN framework. Extension of DDLCN to multiple layers is straight forward.

Feature Extraction Layer. Let 𝒀\bm{Y} denote a set of mm-dimensional local descriptors, which is extracted from the data sampled from some uni-modal sensors, i.e., 𝒀=[𝒚1,⋯,𝒚N]∈ℝm×N\bm{Y}{=}\left[{{\bm{y}_{1}},\cdots,{\bm{y}_{N}}}\right]{\in}{\mathbb{R}^{m\times N}}, where NN is the total number of local descriptors. In order to emphasize the viability of the proposed deep dictionary learning and coding method, we only use a single descriptor, the Scale-Invariant Feature Transform (SIFT) [26] throughout our experiment. SIFT features used in dictionary learning are pretty common in the computer vision field [36, 47, 48, 21, 4, 59]. In our experiments, we achieve better performance using SIFT than using raw pixels. For image II in the dataset, we extract the SIFT feature 𝒚𝒊\bm{y_{i}} as:

𝒚i=F⁡(I),i∈[1,…,N],\bm{y}_{i}=F(I),i\in[1,...,N],\vskip-5.69046pt (1)

where FF denotes the feature extractor.

First Dictionary Learning Layer. We assume the number of classes in the dataset is rr (r=3r{=}3 in Figure 2). For each class, we select pp images to train the corresponding dictionary of the class, the size of this class denotes as qq, as shown in Figure 2, p=3p{=}3 and q=4q{=}4. We adopted the following dictionary learning algorithm:

min𝑽⁡[12​‖𝒚i−𝑽​αi‖22]s.t.‖αi‖1<=λ\begin{array}[]{c}\min\limits_{\bm{V}}\left[\frac{1}{2}||\bm{y}_{i}-\bm{V}\alpha_{i}||_{2}^{2}\right]\quad s.t.\quad||\alpha_{i}||_{1}<=\lambda\end{array}\vskip-5.69046pt (2)

where 𝒚i\bm{y}_{i} is the SIFT feature learned from FF and we set λ=0.35\lambda{=}0.35 in the following experiments. After learning the dictionary of each class, we group all dictionaries of each class to form the first dictionary 𝑽=[𝒗1,𝒗2,⋯,𝒗s1]∈ℝm×s1\bm{V}{=}\left[{{\bm{v}_{1}},{\bm{v}_{2}},\cdots,{\bm{v}_{{s_{1}}}}}\right]\in{\mathbb{R}^{m\times{s_{1}}}}. Then the first dictionary learning layer contains s1s_{1} entries and we have s1=r×qs_{1}{=}r{\times}q (s1=3×4=12s_{1}{=}3{\times}4{=}12 in Figure 2).

Second Dictionary Learning Layer. The second layer dictionary 𝑼=[𝒖1,𝒖2,⋯,𝒖s2]\bm{U}{=}\left[{{\bm{u}_{1}},{\bm{u}_{2}},\cdots,{\bm{u}_{{s_{2}}}}}\right] is obtained by learning from the first layer codebook 𝑽\bm{V}.

min𝑼⁡[12​‖𝒗i−𝑼​αi‖22]s.t.‖αi‖1<=λ\begin{array}[]{c}\min\limits_{\bm{U}}\left[\frac{1}{2}||\bm{v}_{i}-\bm{U}\alpha_{i}||_{2}^{2}\right]\quad s.t.\quad||\alpha_{i}||_{1}<=\lambda\end{array}\vskip-5.69046pt (3)

where 𝒗i∈𝑽{\bm{v}_{i}}\in\bm{V} is one of the basis vectors in the first dictionary (λ=0.35\lambda{=}0.35), as shown in Figure 2, s2=2s_{2}{=}2.

The nt​h\text{n}^{th} Dictionary Learning Layer. We learn the nt​h\text{n}^{th} dictionary 𝑫𝒏=[𝒅1n,𝒅2n,⋯,𝒅snn]\bm{D^{n}}{=}\left[{{\bm{d}_{1}^{n}},{\bm{d}_{2}^{n}},\cdots,{\bm{d}_{{s_{n}}}^{n}}}\right] from the previous layer dictionary 𝑫𝒏−𝟏\bm{D^{n-1}}.

min𝑫𝒏⁡[12​‖𝒅in−1−𝑫𝒏​αi‖22]s.t.‖αi‖1<=λ,\begin{array}[]{c}\min\limits_{\bm{D^{n}}}\left[\frac{1}{2}||\bm{d}_{i}^{n-1}-\bm{D^{n}}\alpha_{i}||_{2}^{2}\right]\quad s.t.\quad||\alpha_{i}||_{1}<=\lambda,\end{array}\vskip-5.69046pt (4)

where 𝒅in−1∈𝑫𝒏−𝟏{\bm{d}_{i}^{n-1}}{\in}\bm{D^{n-1}} is one of the basis vectors in the (n-1)t​h\text{(n-1)}^{th} dictionary layer.

First Feature Coding Layer. After obtaining the dictionary 𝑽\bm{V}, each feature is then encoded by 𝑽\bm{V} through several nearest items to produce the first coding. The number of the nearest items of the first coding layer is set to a small value (e.g., 15). The first feature coding scheme converts each local descriptor 𝒚i\bm{y}_{i} into a s1s_{1} dimensional code 𝜸i1=[𝜸i1​(𝒗1),𝜸i1​(𝒗2),⋯,𝜸i1​(𝒗s1)]𝖳∈ℝs1\bm{\gamma}_{i}^{1}{=}{\left[{\bm{\gamma}_{i}^{1}({\bm{v}_{1}}),\bm{\gamma}_{i}^{1}({\bm{v}_{2}}),\cdots,\bm{\gamma}_{i}^{1}({\bm{v}_{{s_{1}}}})}\right]^{\mathsf{T}}}\in{\mathbb{R}^{s_{1}}}. We arrange each code corresponding to each descriptor into a matrix: 𝜸1=[𝜸11,𝜸21,⋯,𝜸N1]∈ℝs1×N{\bm{\gamma}^{1}}{=}\left[{\bm{\gamma}_{1}^{1},\bm{\gamma}_{2}^{1},\cdots,\bm{\gamma}_{N}^{1}}\right]\in{\mathbb{R}^{{s_{1}}\times N}}. Specifically, each code can be obtained using the following optimization:

min𝜸i1[12​‖𝒚i−𝑽​𝜸i1‖22+β​‖𝜸i1⊙𝜻i1‖1]s.t.𝟏𝖳​𝜸i1=1,\begin{array}[]{c}\mathop{\min}\limits_{\bm{\gamma}_{i}^{1}}\left[{\frac{1}{2}\left\|{{\bm{y}_{i}}-\bm{V}\bm{\gamma}_{i}^{1}}\right\|_{2}^{2}+\beta{{\left\|{\bm{\gamma}_{i}^{1}\odot{\bm{\zeta}_{i}^{1}}}\right\|}_{1}}}\right]\\ \quad s.t.\quad{\bm{1}^{\mathsf{T}}}\bm{\gamma}_{i}^{1}=1,\end{array}\vskip-5.69046pt (5)

where 𝜻i1∈ℝs1\bm{\zeta}_{i}^{1}{\in}\mathbb{R}^{s_{1}} is a distance vector, measuring the distance between 𝒚i\bm{y}_{i} and 𝒗i\bm{v}_{i}, and ⊙\odot denotes the element-wise multiplication or Hadamard product, which enables the corresponding items of both vectors (𝜸i1\bm{\gamma}_{i}^{1} and 𝜻i1\bm{\zeta}_{i}^{1}) to multiply. Typically, 𝜻i1\bm{\zeta}_{i}^{1} can be obtained using ℓ2\ell_{2} norm, that is 𝜻i1=[‖𝒚i−𝒗1‖2,‖𝒚i−𝒗2‖2,⋯,‖𝒚i−𝒗s1‖2]𝖳{\bm{\zeta}_{i}^{1}}{=}\left[{{{\left\|{{\bm{y}_{i}}-{\bm{v}_{1}}}\right\|}_{2}},{{\left\|{{\bm{y}_{i}}-{\bm{v}_{2}}}\right\|}_{2}},\cdots,{{\left\|{{\bm{y}_{i}}-{\bm{v}_{{s_{1}}}}}\right\|}_{2}}}\right]^{\mathsf{T}}.

Second Feature Coding Layer. Similarly, at the second layer, we have,

min𝜸i2⁡[12​‖𝒗^i−𝑼​𝜸i2‖22+β​‖𝜸i2⊙𝜻i2‖1]s.t.1𝖳​𝜸i2=1,\begin{array}[]{c}\min\limits_{\bm{\gamma}_{i}^{2}}\left[{\frac{1}{2}\left\|{{\hat{\bm{v}}_{i}}-\bm{U}\bm{\gamma}_{i}^{2}}\right\|_{2}^{2}+\beta{{\left\|{\bm{\gamma}_{i}^{2}\odot\bm{\zeta}_{i}^{2}}\right\|}_{1}}}\right]\\ \quad s.t.\quad{1^{\mathsf{T}}}\bm{\gamma}_{i}^{2}=1,\end{array}\vskip-5.69046pt (6)

where 𝜸i2=[𝜸i2​(𝒖1),𝜸i2​(𝒖2),⋯,𝜸i2​(𝒖s2)]𝖳∈ℝs2\bm{\gamma}_{i}^{2}{=}{\left[{\bm{\gamma}_{i}^{2}({\bm{u}_{1}}),\bm{\gamma}_{i}^{2}({\bm{u}_{2}}),\cdots,\bm{\gamma}_{i}^{2}({\bm{u}_{{s_{2}}}})}\right]^{\mathsf{T}}}\linebreak{\in}{\mathbb{R}^{{s_{2}}}} is the second coding and 𝜻i2∈ℝs2\bm{\zeta}_{i}^{2}{\in}{\mathbb{R}^{{s_{2}}}} is used to measure the distance between 𝒗^i\hat{\bm{v}}_{i} and each atom in the dictionary matrix 𝑼\bm{U}. 𝒗^i∈𝑽\hat{\bm{v}}_{i}{\in}\bm{V} is one of the basis vectors adopted in the representation of 𝒚i\bm{y}_{i} at the first layer. We decompose these nearest atoms 𝒗^i\hat{\bm{v}}_{i} in the first layer to acquire the second layer coding using the second layer dictionary. For better understanding, there is an illustrative explanation in Figure 2.

[Uncaptioned image]
Figure 3: Multi layers coding strategy. y\bm{y} is linearly combined with v𝟏\bm{v_{1}}, v𝟐\bm{v_{2}}, v𝟑\bm{v_{3}} and v|C𝟏|,t​h​i​s​i​s​w​h​a​t​w​e​c​a​l​l​e​d​s​i​n​g​l​e​l​a​y​e​r​c​o​d​i​n​g.F​u​r​t​h​e​r,t​h​e​a​t​o​m​s​i​n​t​h​e​f​i​r​s​t​l​a​y​e​r​c​a​n​b​e​r​e​p​r​e​s​e​n​t​e​d​b​y​t​h​e​a​t​o​m​s​i​n​t​h​e​s​e​c​o​n​d​l​a​y​e​r.I​n​o​t​h​e​r​w​o​r​d​s,\bm{v_{\left|{{\bm{C}^{1}}}\right|}$},thisiswhatwecalledsinglelayercoding.Further,theatomsinthefirstlayercanberepresentedbytheatomsinthesecondlayer.Inotherwords,v_1,v_2,v_3a​n​dandv_— C^1 —arelinearlyrepresentedbytheatomsinthesecondlayer,respectively.Forexample,oneoftheseatomsv_2,iscombinedby,iscombinedbyu_1,u_2,u_3a​n​dandu_— C^2,v —linearly.Thisiswhatwecalledtwolayerscoding.Bythatanalogy,wecanobtainn^thl​a​y​e​r′​c​o​d​e​s.{{layer^{\prime}codes.}}

The nt​h\text{n}^{th} Feature Coding Layer.Wegeneralizeourtwolayersframeworktoadeeperone,

min𝜸in⁡[12​‖𝒅^in−1−𝑫𝒏​𝜸in‖22+β​‖𝜸in⊙𝜻in‖1]s.t.1𝖳​𝜸in=1,\begin{array}[]{c}\min\limits_{\bm{\gamma}_{i}^{n}}\left[{\frac{1}{2}\left\|{\hat{\bm{d}}_{i}^{n-1}-\bm{D^{n}}\bm{\gamma}_{i}^{n}}\right\|_{2}^{2}+\beta{{\left\|{\bm{\gamma}_{i}^{n}\odot\bm{\zeta}_{i}^{n}}\right\|}_{1}}}\right]\\ \hskip 7.3612pts.t.\hskip 7.3612pt{1^{\mathsf{T}}}\bm{\gamma}_{i}^{n}=1,\end{array}\vskip-5.69046pt (7)

whereγ _i^ni​s​t​h​eisthen^thl​a​y​e​r​c​o​d​i​n​g​a​n​dlayercodingandζ_i^ni​s​u​s​e​d​t​o​m​e​a​s​u​r​e​t​h​e​d​i​s​t​a​n​c​e​b​e​t​w​e​e​nisusedtomeasurethedistancebetween ^ d _i^n-1a​n​d​e​a​c​h​a​t​o​m​i​n​t​h​eandeachatominthen^thd​i​c​t​i​o​n​a​r​ydictionaryD^n. ^ d _i^n-1 ∈ D^n-1i​s​o​n​e​o​f​t​h​e​b​a​s​i​s​v​e​c​t​o​r​s​a​d​o​p​t​e​d​i​n​t​h​e​r​e​p​r​e​s​e​n​t​a​t​i​o​n​o​fisoneofthebasisvectorsadoptedintherepresentationofy_ia​t​t​h​eatthe(n-1)^thc​o​d​i​n​g​l​a​y​e​r.Pooling Layer.​A​f​t​e​r​t​h​e​f​e​a​t​u​r​e​c​o​d​i​n​g​l​a​y​e​r,t​h​e​p​o​o​l​i​n​g​l​a​y​e​r​t​h​e​n​t​a​k​e​s​o​v​e​r.F​o​r​e​a​c​h​i​m​a​g​e,w​e​e​m​p​l​o​ycodinglayer.\par\noindent\textbf{Pooling Layer.}Afterthefeaturecodinglayer,thepoolinglayerthentakesover.Foreachimage,weemploy1×1,2×2a​n​dand4×4s​p​a​t​i​a​l​p​y​r​a​m​i​d​s​m​a​t​c​h​i​n​g​w​i​t​h​m​a​x−p​o​o​l​i​n​g.Fully Connected Layer.​T​h​e​f​i​n​a​l​o​u​t​p​u​t​o​fspatialpyramidsmatchingwithmax-pooling.\par\noindent\textbf{Fully Connected Layer.}Thefinaloutputofn^thl​a​y​e​r′​s​c​o​d​e​s​f​o​rlayer^{\prime}scodesfory_ic​a​n​b​e​o​b​t​a​i​n​e​d​b​y​i​n​t​e​g​r​a​t​i​n​g​e​a​c​h​l​a​y​e​r​s′​c​o​d​e​s.M​o​r​e​s​p​e​c​i​f​i​c​a​l​l​y,e​a​c​h​i​t​e​m​i​n​t​h​e​f​i​r​s​t​l​a​y​e​r′​s​c​o​d​e​scanbeobtainedbyintegratingeachlayers^{\prime}codes.Morespecifically,eachiteminthefirstlayer^{\prime}scodesγ _i^1c​a​n​b​e​a​u​g​m​e​n​t​e​d​i​n​t​o​a​v​e​c​t​o​r,f​o​r​i​n​s​t​a​n​c​e,t​h​ecanbeaugmentedintoavector,forinstance,thej^thi​t​e​m​c​a​n​b​e​a​u​g​m​e​n​t​e​d​i​n​t​o​t​h​e​f​o​r​m​o​fitemcanbeaugmentedintotheformof[ γ _i^1(v_j), γ _i^1(v_j)[γ _j^2(u_1),γ _j^2(u_2), ⋯,γ _j^2(u_s_2)] ]^T(pleaserefertoAlgorithm).Output Layer.TheSupportVectorMachine(SVM)isadoptedastheclassifier.TheimplementationofmulticlassSVMisprovidedbyLIBSVM[7]. Algorithm 1 1 1 Algorithm 1 1The two-layer model of the DDLCN. Algorithm 1 The two-layer model of the DDLCN. 0: 0 0   : ∈ Y R × m N 0: 0 0   : γ i 1: 1 1   First dictionary learning: ← V V ⁢ D i c t i o n a r y 2: 2 2   First locality constraint calculating: = ζ i 1 [ ‖ - y i v 1 ‖ 2 , ‖ - y i v 2 ‖ 2 , ⋯ , ‖ - y i v s 1 ‖ 2 ] T 3: 3 3   First feature coding: = ⁢ for i ⁢ 1 t o N ← γ i 1 min γ i 1 [ + ⁢ 1 2 ‖ - y i ⁢ V γ i 1 ‖ 2 2 ⁢ β ‖ ⊙ γ i 1 ζ i 1 ‖ 1 ] s . t . = ⁢ 1 T γ i 1 1    end 4: 4 4   Second dictionary learning: ← U U ⁢ D i c t i o n a r y 5: 5 5   Second locality constraint calculating: = ζ i 2 [ ‖ - v i u 1 ‖ 2 , ‖ - v i u 2 ‖ 2 , ⋯ , ‖ - v i u s 2 ‖ 2 ] T 6: 6 6   Second feature coding: = ⁢ for i ⁢ 1 t o N ← γ i 2 min γ i 2 [ + ⁢ 1 2 ‖ - v i ⁢ U γ i 2 ‖ 2 2 ⁢ β ‖ ⊙ γ i 2 ζ i 2 ‖ 1 ] s . t . = ⁢ 1 T γ i 2 1    end 7: 7 7   Coding augmentation: = ⁢ for i ⁢ 1 t o N = ⁢ for j ⁢ 1 t o s 1 ← ⁢ γ i 1 ( v j ) [ ⁢ γ i 1 ( v j ) , ⁢ γ i 1 ( v j ) [ ⁢ γ j 2 ( u 1 ) , ⋯ , ⁢ γ j 2 ( u s 2 ) ] ] T       end    end (pleaserefertoAlgorithm~\ref{alg:SA}).\par\noindent\textbf{Output Layer.}TheSupportVectorMachine(SVM)isadoptedastheclassifier.TheimplementationofmulticlassSVMisprovidedbyLIBSVM\cite[cite]{[\@@bibref{}{chang2011libsvm}{}{}]}.\par\par\begin{algorithm}\small\@@toccaption{{\lx@tag[ ]{{1}}{The two-layer model of the DDLCN.}}}\@@caption{{\lx@tag[ ]{{Algorithm 1}} {The two-layer model of the DDLCN.}}} \lx@algorithmic@beginlist{{\footnotesize 0:}}{ }\lx@algorithmic@item@@[\algorithmicrequire]\hskip 5.32494pt: $\bm{Y}\in{\mathbb{R}^{m\times N}}$ \lx@algorithmic@item@@[\algorithmicensure]\hskip 5.32494pt: $\bm{\gamma}_{i}$\par\addvspace\ignorespaces\lx@algorithmic@item@@\hskip 5.32494ptFirst dictionary learning: $\hskip 9.24994pt\bm{V}\leftarrow\bm{V}_{Dictionary}$ \lx@algorithmic@item@@\hskip 5.32494ptFirst locality constraint calculating:\par\addvspace\ignorespaces${\hskip 9.24994pt\bm{\zeta}_{i}^{1}}=\left[{{{\left\|{{\bm{y}_{i}}-{\bm{v}_{1}}}\right\|}_{2}},{{\left\|{{\bm{y}_{i}}-{\bm{v}_{2}}}\right\|}_{2}},\cdots,{{\left\|{{\bm{y}_{i}}-{\bm{v}_{{s_{1}}}}}\right\|}_{2}}}\right]^{\mathsf{T}}$ \lx@algorithmic@item@@\hskip 5.32494ptFirst feature coding:\par\addvspace\ignorespaces$\hskip 9.24994pt\textbf{for}\hskip 9.24994pti=1\;to\;N$\par\addvspace\ignorespaces$\hskip 9.24994pt\begin{array}[]{l}\bm{\gamma}_{i}^{1}\leftarrow\mathop{\min}\limits_{\bm{\gamma}_{i}^{1}}\left[{\frac{1}{2}\left\|{{\bm{y}_{i}}-\bm{V}\bm{\gamma}_{i}^{1}}\right\|_{2}^{2}+\beta{{\left\|{\bm{\gamma}_{i}^{1}\odot{\bm{\zeta}_{i}^{1}}}\right\|}_{1}}}\right]\\ \hskip 9.24994pt\hskip 9.24994pts.t.\hskip 9.24994pt{1^{\mathsf{T}}}\bm{\gamma}_{i}^{1}=1\end{array}$\par\addvspace\ignorespaces$\hskip 9.24994pt\textbf{end}$ \par\lx@algorithmic@item@@\hskip 5.32494ptSecond dictionary learning: $\hskip 9.24994pt\bm{U}\leftarrow\bm{U}_{Dictionary}$ \lx@algorithmic@item@@\hskip 5.32494ptSecond locality constraint calculating:\par\addvspace\ignorespaces$\hskip 9.24994pt{\bm{\zeta}_{i}^{2}}=\left[{{{\left\|{{\bm{v}_{i}}-{\bm{u}_{1}}}\right\|}_{2}},{{\left\|{{\bm{v}_{i}}-{\bm{u}_{2}}}\right\|}_{2}},\cdots,{{\left\|{{\bm{v}_{i}}-{\bm{u}_{{s_{2}}}}}\right\|}_{2}}}\right]^{\mathsf{T}}$ \lx@algorithmic@item@@\hskip 5.32494ptSecond feature coding:\par\addvspace\ignorespaces$\hskip 9.24994pt\textbf{for}\hskip 9.24994pti=1\;to\;N$\par\addvspace\ignorespaces$\hskip 9.24994pt\begin{array}[]{l}\bm{\gamma}_{i}^{2}\leftarrow\mathop{\min}\limits_{\bm{\gamma}_{i}^{2}}\left[{\frac{1}{2}\left\|{{\bm{v}_{i}}-\bm{U}\bm{\gamma}_{i}^{2}}\right\|_{2}^{2}+\beta{{\left\|{\bm{\gamma}_{i}^{2}\odot\bm{\zeta}_{i}^{2}}\right\|}_{1}}}\right]\\ \hskip 9.24994pt\hskip 9.24994pts.t.\hskip 9.24994pt{1^{\mathsf{T}}}\bm{\gamma}_{i}^{2}=1\end{array}$\par\addvspace\ignorespaces$\hskip 9.24994pt\textbf{end}$ \lx@algorithmic@item@@\hskip 5.32494ptCoding augmentation:\par\addvspace\ignorespaces$\hskip 9.24994pt\textbf{for}\hskip 9.24994pti=1\;to\;N$\par\addvspace\ignorespaces$\hskip 9.24994pt\hskip 9.24994pt\textbf{for}\hskip 9.24994ptj=1\;to\;s_{1}$ \par\addvspace\ignorespaces$\hskip 9.24994pt\hskip 9.24994pt\hskip 9.24994pt{{\bm{\gamma}_{i}^{1}({\bm{v}_{j}})}\leftarrow\left[{\bm{\gamma}_{i}^{1}({\bm{v}_{j}}),\;\bm{\gamma}_{i}^{1}({\bm{v}_{j}})[\bm{\gamma}_{j}^{2}({\bm{u}_{1}}),\cdots,\bm{\gamma}_{j}^{2}({\bm{u}_{{s_{2}}}})]}\right]^{\mathsf{T}}}$ \par\addvspace\ignorespaces$\hskip 9.24994pt\hskip 9.24994pt\textbf{end}$ \par\addvspace\ignorespaces$\hskip 9.24994pt\textbf{end}$ \lx@algorithmic@endlist\end{algorithm}\par

3 Experiments

Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 4: Classification rate with different parameter pp on the Extended YaleB (Top) and AR Face (Bottom) datasets.

3.1 Datasets

According to [4, 39, 19, 2, 58, 18, 46], we evaluate the performance of DDLCN on four popular datasets, which are all standard datasets for dictionary learning evaluation.

(i) Extended YaleB dataset [15] contains 2,414 frontal face images of 38 people. There are 59 to 64 images for each person. All the experiments randomly select half of the images per category as training data and the other half for testing; (ii) AR Face dataset [27] comprises of over 4,000 color images of 126 people (70 men and 56 women), including frontal views of faces with different facial expressions, lighting conditions and occlusions. Each person has 26 face images taken during two sessions, in each of which, each person has 13 images. Among them, 3 are obscured by scarves, 6 by sunglasses, and the remaining faces are of different facial expressions or illumination variations which we refer to as unobscured images. Following the standard evaluation procedure, we use a subset of the dataset which consists of 2,600 images from 50 male subjects and 50 female subjects. For each subject, we randomly select 20 samples for training and the other 6 images for testing; (iii) Caltech 256 dataset [16] contains 30,607 images of 257 categories. The number of images per category varies from 80 to 827; (iv) MNIST dataset [22] consists of 60,000 training digits and 10,000 testing digits.

3.2 Parameter Setting

In the proposed DDLCN model there are only three parameters while CNN-based methods have more parameters that need to be tuned. We have conducted exhaustive experiments to emphasize the superiority of the proposed method. Three parameters for the experiments are,

  • •

    pp, the number of training dictionary samples per category;

  • •

    qq, the size of the first layer dictionary per category;

  • •

    tt, the number of training samples per category.

For simplicity, pp-qq denotes pp images are randomly selected per category for training dictionary and qq dictionary bases are learned per category in the first dictionary. For instance,“15-15” means p=15p{=}15 and q=15q{=}15. The size of the first dictionary is D1=q∗rD_{1}{=}q*r, while the second one is fixed to D2=64D_{2}{=}64. The number of the nearest atoms is fixed to 15 and 10 for the first and second feature coding layers, respectively. We repeat all the experiments 10 times with different random splits of the training and testing images to obtain reliable results. The final classification rates are reported as the average of each run.

3.3 Results on Parameter p\bm{p}

The first dictionary is trained on p=[1,2,3,4,5,10,15,20,25,30,35,40,45,50,55]p{=}[1,2,3,4,5,10,15,20,25,30,35,40,45,50,55] and p=[1,2,3,4,5,10,15,20,25,26]p{=}[1,2,3,4,5,10,15,20,25,26] samples per category on the Extended YaleB and AR Face datasets, respectively. We consider that in this exhaustive way, the superiority of our method could be fully reflected. From the results shown in Figure 4, we can clearly see the relationships between q & p, q & t and p & t. In addition, we consistently observe that the classification accuracy achieves a peak with 10 training samples and then tends to be stable.

3.4 Results on Parameter q\bm{q}

We evaluate our approach with different q=[1,2,3,4,5,10,15,20,25,30]q{=}[1,2,3,4,5,10,15,20,25,30] per person on both the Extended YaleB and AR Face datasets. The results are shown in Figure 5. We observer that on the Extended YaleB dataset that with different qq, the gaps among all the classification rates are marginal due to the introduction of deep dictionary learning and coding strategy. This strategy can exploit more information about signal 𝒚′\bm{y^{\prime}} and incorporate more gradient information about 𝒚′\bm{y^{\prime}} into the coding structure. Interestingly, when 20 images per class are randomly selected as training data, the classification rate is close to 100% when only using 1 atom per person on the AR Face dataset. The basic reason for the excellent recognition performance is that the proposed DDLCN fully exploits the intrinsic structure of the manifold where features reside, and incorporates more information about the nonlinear function on each group of basis vectors.

Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 5: Classification rate with varying qq on the Extended YaleB (Top) and AR Face (Bottom) datasets.
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 6: Classification rate with different tt on the Extended YaleB (Top) and AR Face (Bottom) datasets.

3.5 Results on Parameter t\bm{t}

The number of the training images is t=[1,2,3,4,5,10,15,20,25,30,35,40,45,50,55]t{=}[1,2,3,4,5,10,15,20,25,30,35,40,45,50,55] and t=[1,2,3,4,5,10,15,20,25]t{=}[1,2,3,4,5,10,15,20,25] per category on the Extended YaleB and AR Face datasets, we can draw two conclusions from Figure 6, (i) the classification rate first rises to the peak rapidly and then tends to be stable as tt increasing; (ii) there is a small impact to classification accuracy when changing pp.

Table 1: Classification rate (%) on the Extended YaleB dataset.
Method Included (%) Excluded∗ (%) Time (ms)
SRC (15 per person) [45] 80.50 86.70 11.22
LLC (30 local bases) [43] 82.20 92.10 -
DL-COPAR [42] 86.47 ±\pm 0.69 - 31.11
FDDL [50] 90.01 ±\pm 0.69 - 42.48
LLC (70 local bases) [43] 90.70 96.70 -
DBDL [3] 91.09 ±\pm 0.59 - 1.07
JBDC [2] 92.14 ±\pm 0.52 - 1.02
K-SVD (15 per person) [1] 93.10 98.00 -
SupGraphDL-L [53] 93.44 - -
D-KSVD (15 per person) [57] 94.10 98.00 -
LC-KSVD1 (15-15) [20] 94.50 98.30 0.52
LC-KSVD2 (15-15) [20] 95.00 98.80 0.49
Multi-Block Alternating Proximal[4] 95.12 - -
ITDL [34] 95.39 - -
VAE + GAN [28] 96.4 - -
EasyDL [35] 96.22 - -
LC-KSVD2 (A-15) [20] 96.70 99.00 -
SRC (all training samples) [45] 97.20 99.00 20.78
CSDL-SRC(power) [24] 98.28 ±\pm 0.57 - -
RRC_L1 (300) [51] 99.80 - -
PCANet-1 [6] 97.77 - -
PCANet-2 [6] 99.85 - -
DDLCN (1-1) 87.42 ±\pm 1.33 89.54 ±\pm 1.02 0.18
DDLCN (15-15) 97.38 ±\pm 0.54 98.48 ±\pm 0.48 0.71
DDLCN (55-15) 97.68 ±\pm 0.60 98.64 ±\pm 0.52 0.92
DDLCN (A-15) 98.34 ±\pm 0.56 99.18 ±\pm 0.46 0.98

3.6 Comparison Against Baselines

We compare our results with the state-of-the-art, the comparative results of the Extended YaleB, AR Face, Caltech 256 and MNIST datasets.

Extended YaleB. We compare our DDLCN with the traditional dictionary learning methods, e.g., D-KSVD [57], LC-KSVD [20], and other deep learning approaches, e.g., PCANet [6] and VAE + GAN [28]. DDLCN is better than all the baselines except [6] and [51] as shown in the second column of Table 1. A different training strategy is adopted which helps improve the classification performance in [6]. Even under this unequal conditions, our method also outperforms the PCANet-1 (97.77%) and is slightly worse than PACNet-2 (99.85%), which validates the advantages of our method. Followed the evaluation metric of [20], another experiment with the bad images excluded (10 for each person) is performed and the results are listed in the third column of Table 1. It is observed that the proposed DDLCN achieves higher classification rate than other methods when using the A-15 strategy. In addition, we compare with SRC [45], LC-KSVD [20], DBDL [3]and JBDC [2] in terms of the computation time for classifying one test image, as shown in the fourth column of Table 1. The time that our approach took is much less than LC-KSVD, DBDL, JBDC and other methods.

AR Face. We compare the DDLCN with some advanced methods, e.g., LC-KSVD [20], SupGraphDL-L [53], etc. We can observe that our approach outperforms others including [6] and [51] when only using 1-1 strategy in Table 2. It is a surprising result and the reason is that the proposed DDLCN makes the approximation quality from one layer O⁡(ε2)O({\varepsilon^{2}}), ascending to two layers O⁡(ε3)O({\varepsilon^{3}}) according to Lipschitz Smoothness. This ensures the quality of approximation to achieve O⁡(ε3)O({\varepsilon^{3}}) on imperfect atoms. In addition, we also report the computation time (ms) for classification on several methods. As shown in the third column of Table 2, the time that our approach took is marginally more than LC-KSVD, but is much less than SRC, DL-COPAR, JBDC, DBDL and FDDL.

Table 2: Classification rate (%) on the AR Face dataset.
Method Accuracy (%) Time (ms)
SRC (5 per person) [45] 66.50 17.76
LLC (30 local bases) [43] 69.50 -
DL-COPAR [42] 83.29 ±\pm 1.23 36.49
FDDL [50] 85.97 ±\pm 1.23 50.03
DBDL [3] 86.15 ±\pm 1.19 1.20
K-SVD (5 per person) [1] 86.50 -
JBDC [2] 87.17 ±\pm 0.99 1.18
LLC (70 local bases) [43] 88.70 -
D-KSVD (5 per person) [57] 88.80 -
LC-KSVD1 (5-5) [20] 92.50 0.541
LC-KSVD2 (5-5) [20] 93.70 0.479
Multi-Block Alternating Proximal[4] 93.88 -
RRC_L1[51] 96.30 -
ADDL (5 items, 20 labels) [58] 97.00 -
SRC (all training samples) [45] 97.50 83.79
LC-KSVD2 (A-5) [20] 97.80 -
LGII [31] 99.00 -
PCANet-1 [6] 98.00 -
PCANet-2 [6] 99.50 -
DDLCN (1-1) 99.56 ±\pm 0.21 0.73
DDLCN (5-5) 99.84 ±\pm 0.36 1.26
DDLCN (A-5) 99.87 ±\pm 0.19 1.63
Table 3: Classification rate (%) on the Caltech 256 dataset.
Num. of train. samp. 15 train 30 train 45 train 60 train
KC [41] - 27.17 ±\pm 0.46 - -
LLC [43] 25.61 30.43 - -
K-SVD [1] 25.33 30.62 - -
D-KSVD [57] 27.79 32.67 - -
LC-KSVD1 [20] 28.10 32.95 - -
SRC [45] 27.86 33.33 - -
Griffin [17] 28.30 34.10 ±\pm 0.20 - -
LC-KSVD2 [20] 28.90 34.32 - -
Graph-matching [9] - 38.10 ±\pm 0.60 - -
Local NBNN [29] 33.50 ±\pm 0.90 40.10 ±\pm 0.10 - -
Latent Structural[25] 36.34 42.01 - -
ScSPM [49] 27.73 ±\pm 0.51 34.02 ±\pm 0.35 37.46 ±\pm 0.55 40.14 ±\pm 0.91
NDL [18] 29.30 ±\pm 0.29 36.80 ±\pm 0.45 - -
LScSPM [13] 30.00 ±\pm 0.14 35.74 ±\pm 0.10 38.54 ±\pm 0.36 40.43 ±\pm 0.38
SSC [32] 30.60 ±\pm 0.30 37.00 ±\pm 0.30 40.70 ±\pm 0.10 43.50 ±\pm 0.30
SNDL [18] 31.10 ±\pm 0.35 38.25 ±\pm 0.43 - -
MLCW [12] 34.10 39.90 42.40 45.60
CRBM [38] 35.1 42.1 45.7 47.9
LP-β\beta[14] - 45.8 - -
M-HMP [5] 42.7 50.7 54.8 58.0
Convolutional Networks [56] - - - 74.2 ±\pm 0.3
VGG19 [37] - - - 84.10
DDLCN (1-1) 26.30 ±\pm 0.40 31.45 ±\pm 0.21 34.69 ±\pm 0.31 37.76 ±\pm 0.25
DDLCN (15-15) 35.06 ±\pm 0.26 41.26 ±\pm 0.22 44.17 ±\pm 0.35 47.48 ±\pm 0.26
DDLCN (30-30) 45.25 ±\pm 0.31 51.64 ±\pm 0.51 55.11 ±\pm 0.26 59.66 ±\pm 0.45

Caltech 256. We evaluate our approaches on 15, 30, 45 and 60 training images per class and compare with the state of the art, the comparisons are shown in Table 3. The proposed method achieves better results than other traditional dictionary learning methods in all cases except VGG Net [37] and convolutional network [56] when using 60 training samples. In [37], the authors use a very deep convolutional network (up to 19 layers) for the task, which reveals that the network depth is of crucial importance, and the leading results on the challenging dataset all exploit “very deep” models. Beside, note that after extracting features, the feature learner and coder of the DDLCN would be a fixed block that would not be updated during training, and only the linear classifier on top is updated during training. Compared with [37] and [56], both are CNN-based models and they learn features directly from the raw pixels. Both methods need to train the weights for the entire network rather than only the weights of linear classifier on the top layer, being as such computationally expensive. However, the training of the DDLCN is offline which represents a big advantage. The test phase of the DDLCN is pretty fast for our method. Moreover, our approaches outperform the other competing dictionary learning approaches even when using the 15-15 strategy, including K-SVD, D-KSVD, LC-KSVD, LLC, etc. What is surprising is that even when using the 1-1 strategy, our method still achieves better results than K-SVD, KC and CRBM, which validates the advantages of the proposed method. Figure 7 shows some example images from classes with high classification accuracy.

Refer to caption
(a) car side, acc: 100.00%
Refer to caption
(b) faces easy, acc: 99.20%
Refer to caption
(c) leopards, acc: 98.46%
Refer to caption
(d) motorbikes, acc: 98.29%
Refer to caption
(e) airplanes, acc: 98.03%
Refer to caption
(f) tower pisa, acc: 97.33%
Refer to caption
(g) ketch, acc: 97.16%
Refer to caption
(h) brain, acc: 96.30%
Refer to caption
(i) sunflower, acc: 96.00%
Refer to caption
(j) watch, acc: 95.36%
Figure 7: Example images from classes with high classification accuracy from the Caltech 256 dataset.
Table 4: Classification rate (%) on the MNIST dataset.
Method Accuracy (%)
Deep Representation Learning [52] 85.47
RGF [11] 98.09
DCN [23] 98.15
Embed CNN [44] 98.50
Convolutional Clustering [10] 98.60
Deep Convolutional Learning [33] 99.58
DDLCN (1-2) 96.56
DDLCN (100-100) 98.55
DDLCN (500-500) 99.02
Refer to caption
Refer to caption
Figure 8: Confusion matrix on the MNIST dataset.

MNIST. The classification rates of different approaches on the MNIST dataset are provided in Table 4. We observe that the DDLCN consistently outperforms all the baselines except [10] and [33] when using the100-100 strategy. The authors in [33] use a two layers model and plus a one layer features to achieve better result (+1.03%) than us. This is because both the CNN-based methods are jointly optimized between forward and backward propagation, and the proposed method has no end-to-end tuning. Thus the training of DDLCN is more efficieint than CNN-based methods. Such deep networks integrate low/mid/highlevel features and classifiers under supervision. In addition, when p=1p{=}1 and q=2q{=}2, we even achieve 96.56% classification accuracy, which proves again that our method can also achieve good recognition rate when the number of the training samples is limited and the size of the dictionary is small. This phenomenon will be of great benefit in practical applications, especially when the training data are limited. Note that when p=100p{=}100 and q=100q{=}100, the classification rate is boosted to 98.55% (Figure 8 (a)). When p=500p{=}500 and q=500q{=}500, the classification rate is further improved to 99.02%. Figure 8 (b) shows the confusion matrix and we observe that the most confused pairs are (2, 7), (4, 9) and (3, 5).

4 Conclusion

The goal of this paper is to improve the deep representation capability of dictionary learning. To this end, we propose a novel deep dictionary learning network DDLCN to learn multi-layer deep dictionaries, which combines the advantages of both deep learning and dictionary learning and achieves impressive performance. We designed a dictionary learning layer and used it to replace traditional convolutional layers in a CNN. More specifically, for two-layer framework, the first layer learns a dictionary to represent input data, the next layer learns a dictionary to represent the atoms of the first dictionary. For a given input sample, the code consists of its locality constrained sparse code, together with the locality constrained sparse codes for all of the atoms that participate in its code. Experimental results on four popular benchmarks demonstrate that the proposed DDLCN outperforms the existing dictionary learning methods and achieves competitive results compared with the CNN-based models. Our code is available at https://github.com/Ha0Tang/DDLCN.

References

  • [1] M. Aharon, M. Elad, and A. Bruckstein. K-svd: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE TSP, 54(11):4311–4322, 2006.
  • [2] N. Akhtar, A. Mian, and F. Porikli. Joint discriminative bayesian dictionary and classifier learning. in CVPR, 2017.
  • [3] N. Akhtar, F. Shafait, and A. Mian. Discriminative bayesian dictionary learning for classification. IEEE TPAMI, 38(12):2374–2388, 2016.
  • [4] C. Bao, H. Ji, Y. Quan, and Z. Shen. Dictionary learning for sparse coding: Algorithms and convergence analysis. IEEE TPAMI, 38(7):1356–1369, 2016.
  • [5] L. Bo, X. Ren, and D. Fox. Multipath sparse coding using hierarchical matching pursuit. in CVPR, 2013.
  • [6] T.-H. Chan, K. Jia, S. Gao, J. Lu, Z. Zeng, and Y. Ma. Pcanet: A simple deep learning baseline for image classification? IEEE TIP, 24(12):5017–5032, 2015.
  • [7] C.-C. Chang and C.-J. Lin. Libsvm: a library for support vector machines. ACM TIST, 2(3):27, 2011.
  • [8] I. Y. Chun and J. A. Fessler. Convolutional dictionary learning: Acceleration and convergence. IEEE TIP, 27(4):1697–1712, 2018.
  • [9] O. Duchenne, A. Joulin, and J. Ponce. A graph-matching kernel for object categorization. in ICCV, 2011.
  • [10] A. Dundar, J. Jin, and E. Culurciello. Convolutional clustering for unsupervised learning. in ICLR Workshop, 2016.
  • [11] M. Fan, X. Zhang, Z. Lin, Z. Zhang, and H. Bao. A regularized approach for geodesic-based semisupervised multimanifold learning. IEEE TIP, 23(5):2133–2147, 2014.
  • [12] S. R. Fanello, N. Noceti, C. Ciliberto, G. Metta, and F. Odone. Ask the image: supervised pooling to preserve feature locality. in CVPR, 2014.
  • [13] S. Gao, I. W.-H. Tsang, L.-T. Chia, and P. Zhao. Local features are not lonely–laplacian sparse coding for image classification. in CVPR, 2010.
  • [14] P. Gehler and S. Nowozin. On feature combination for multiclass object classification. in ICCV, 2009.
  • [15] A. S. Georghiades, P. N. Belhumeur, and D. J. Kriegman. From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE TPAMI, 23(6):643–660, 2001.
  • [16] G. Griffin, A. Holub, and P. Perona. Caltech-256 object category dataset. CIT Technical Report, 2007.
  • [17] G. Griffin, A. Holub, and P. Perona. Caltech-256 object category dataset. California Institute of Technology, 2007.
  • [18] J. Hu and Y.-P. Tan. Nonlinear dictionary learning with application to image classification. PR, 75:282–291, 2018.
  • [19] M. Jian and C. Jung. Semi-supervised bi-dictionary learning for image classification with smooth representation-based label propagation. IEEE TMM, 18(3):458–473, 2016.
  • [20] Z. Jiang, Z. Lin, and L. S. Davis. Learning a discriminative dictionary for sparse coding via label consistent k-svd. in CVPR, 2011.
  • [21] S. Kim, R. Cai, K. Park, S. Kim, and K. Sohn. Modality-invariant image classification based on modality uniqueness and dictionary learning. IEEE TIP, 26(2):884–899, 2017.
  • [22] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
  • [23] Y. Lin, Z. Tong, S. Zhu, and K. Yu. Deep coding network. in NIPS, 2010.
  • [24] B.-D. Liu, B. Shen, L. Gui, Y.-X. Wang, X. Li, F. Yan, and Y.-J. Wang. Face recognition using class specific dictionary learning for sparse representation and collaborative representation. Neurocomputing, 204:198–210, 2016.
  • [25] L. Liu, X. Bai, H. Zhang, J. Zhou, and W. Tang. Describing and learning of related parts based on latent structural model in big data. Neurocomputing, 173:355–363, 2016.
  • [26] D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 60(2):91–110, 2004.
  • [27] A. M. Martinez. The ar face database. CVC TR, 24, 1998.
  • [28] M. F. Mathieu, J. J. Zhao, J. Zhao, A. Ramesh, P. Sprechmann, and Y. LeCun. Disentangling factors of variation in deep representation using adversarial training. in NIPS, 2016.
  • [29] S. McCann and D. G. Lowe. Local naive bayes nearest neighbor for image classification. in CVPR, 2012.
  • [30] H. V. Nguyen, H. T. Ho, V. M. Patel, and R. Chellappa. Dash-n: Joint hierarchical domain adaptation and feature learning. IEEE TIP, 24(12):5479–5491, 2015.
  • [31] S. Nikan and M. Ahmadi. Local gradient-based illumination invariant face recognition using local phase quantisation and multi-resolution local binary pattern fusion. IET IP, 9(1):12–21, 2015.
  • [32] G. L. Oliveira, E. R. Nascimento, and A. W. Vieira. Sparse spatial coding: a novel approach to visual recognition. IEEE TIP, 23(6):2719–2731, 2014.
  • [33] Y. Pu, X. Yuan, and L. Carin. A generative model for deep convolutional learning. in ICLR Workshop, 2015.
  • [34] Q. Qiu, V. M. Patel, and R. Chellappa. Information-theoretic dictionary learning for image classification. IEEE TPAMI, 36(11):2173–2184, 2014.
  • [35] Y. Quan, Y. Xu, Y. Sun, Y. Huang, and H. Ji. Sparse coding for classification via discrimination ensemble. in CVPR, 2016.
  • [36] L. Shen, G. Sun, Q. Huang, S. Wang, Z. Lin, and E. Wu. Multi-level discriminative dictionary learning with application to large scale image classification. IEEE TIP, 24(10):3109–3123, 2015.
  • [37] M. Simon and E. Rodner. Neural activation constellations: Unsupervised part model discovery with convolutional networks. in ICCV, 2015.
  • [38] K. Sohn, D. Y. Jung, H. Lee, and A. O. Hero. Efficient learning of sparse, distributed, convolutional feature representations for object recognition. in ICCV, 2011.
  • [39] J. Sun and J. Ponce. Learning dictionary of discriminative part detectors for image categorization and cosegmentation. IJCV, 120(2):111–133, 2016.
  • [40] S. Tariyal, A. Majumdar, R. Singh, and M. Vatsa. Greedy deep dictionary learning. arXiv preprint arXiv:1602.00203, 2016.
  • [41] J. C. van Gemert, J.-M. Geusebroek, C. J. Veenman, and A. W. Smeulders. Kernel codebooks for scene categorization. in ECCV, 2008.
  • [42] D. Wang and S. Kong. A classification-oriented dictionary learning model: Explicitly learning the particularity and commonality across categories. PR, 47(2):885–898, 2014.
  • [43] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong. Locality-constrained linear coding for image classification. in CVPR, 2010.
  • [44] J. Weston, F. Ratle, H. Mobahi, and R. Collobert. Deep learning via semi-supervised embedding. Neural Networks: Tricks of the Trade, pages 639–655, 2012.
  • [45] J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma. Robust face recognition via sparse representation. IEEE TPAMI, 31(2):210–227, 2009.
  • [46] W. Xiao, H. Liu, H. Tang, and H. Liu. Two-layers local coordinate coding. in CCCV, 2015.
  • [47] Y. Yan, Y. Yang, H. Shen, D. Meng, G. Liu, A. G. Hauptmann, and N. Sebe. Complex event detection via event oriented dictionary learning. in AAAI, 2015.
  • [48] J. Yang and M.-H. Yang. Top-down visual saliency via joint crf and dictionary learning. IEEE TPAMI, 39(3):576–588, 2017.
  • [49] J. Yang, K. Yu, Y. Gong, and T. Huang. Linear spatial pyramid matching using sparse coding for image classification. in CVPR, 2009.
  • [50] M. Yang, L. Zhang, X. Feng, and D. Zhang. Sparse representation based fisher discrimination dictionary learning for image classification. IJCV, 109(3):209–232, 2014.
  • [51] M. Yang, L. Zhang, J. Yang, and D. Zhang. Regularized robust coding for face recognition. IEEE TIP, 22(5):1753–1766, 2013.
  • [52] S. Yang, P. Luo, C. C. Loy, K. W. Shum, X. Tang, et al. Deep representation learning with target coding. in AAAI, 2015.
  • [53] Y. Yankelevsky and M. Elad. Structure-aware classification using supervised dictionary learning. in ICASSP, 2017.
  • [54] K. Yu and T. Zhang. Improved local coordinate coding using local tangents. in ICML, 2010.
  • [55] K. Yu, T. Zhang, and Y. Gong. Nonlinear learning using local coordinate coding. in NIPS, 2009.
  • [56] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. in ECCV, 2014.
  • [57] Q. Zhang and B. Li. Discriminative k-svd for dictionary learning in face recognition. in CVPR, 2010.
  • [58] Z. Zhang, W. Jiang, J. Qin, L. Zhang, F. Li, M. Zhang, and S. Yan. Jointly learning structured analysis discriminative dictionary and analysis multiclass classifier. IEEE TNNLS, 29(8):3798–3814, 2018.
  • [59] S. Zhu, Z. Shi, C. Sun, and S. Shen. Deep neural network based image annotation. PRL, 65:103–108, 2015.