Information-Theoretic Active Learning for Content-Based Image Retrieval

Let $\mathfrak{U}=\{x_{1},\dotsc,x_{m}\}$ be a set of features of unlabeled samples and $\mathfrak{L}=\{(x_{m+1},y_{m+1}),\dotsc,(x_{m+\ell},y_{m+\ell})\}$ be a set of features of labeled samples $x_{i}\in\mathbb{R}^{d}$ and their labels $y_{i}\in\{-1,1\}$. The label $1$ is assigned to relevant and $-1$ to irrelevant samples. $\mathfrak{X}=\{x_{1},\dotsc,x_{m},x_{m+1},\dotsc,x_{m+\ell}\}$ denotes the set of all $n=m+\ell$ samples. In the scenario of content-based image retrieval, $\mathfrak{L}$ usually consists initially of the features of a single relevant sample: the query image provided by the user. However, queries consisting of multiple and even negative examples are possible as well.

Intuitively, we want to ask the user for relevance feedback for a batch $u\subseteq\mathfrak{U}$ of $k=|u|$ unlabeled samples, whose feedback we expect to be most helpful for classifying the remaining unlabeled instances, i.e., assessing their relevance to the user. Note that these chosen samples are also often referred to as “queries” in the active learning literature. To avoid confusion caused by this conflicting terminology, we will refer to the query image as “query” and to the unlabeled samples chosen for annotation as “candidates”.

Ideally, the most informative batch $u$ of candidates can be found by maximizing the conditional mutual information $\mathfrak{I}(R,F\mid u)$ between the relevance $R$ of both labeled and unlabeled samples, which is a multivariate random variable over the space $\{-1,1\}^{n}$ of relevance labels, and the user feedback $F$, being a multivariate random variable over the space $\{-1,0,1\}^{n}$ of possible feedbacks. A feedback of $0$ represents the case that the user has not given any feedback for a certain candidate. This option is a special feature of our approach, which allows the user to omit candidates that cannot be labeled reliably.

Since the size $n$ of the dataset can be huge, this problem is not solvable in practice. We will show later on how it can be approximated to become tractable. But for now, let us consider the ideal optimization objective:

Writing the mutual information (MI) in terms of entropy reveals the relationship of our approach to uncertainty sampling by maximizing the entropy $H(R\mid u)$ of the candidate batch [23, 33, 32] or minimizing the conditional entropy $H(R\mid F,u)$ [14]:

In contrast to pure uncertainty maximization, we also take into account how the relevance model is expected to change after having obtained the feedback from the user: To select those samples whose annotation would reduce uncertainty the most, we maximize the difference between the uncertainty $H(R\mid u)$ according to the current relevance model and the uncertainty $H(R\mid F,u)$ after an update of the model with the expected user feedback.

In contrast to existing works [23, 33, 32], we do not assume $H(R\mid u)$ to be equal to the sum of individual entropies of the samples, but use their joint distribution to compute the entropy. Thus, maximizing $H(R\mid u)$ is also a possible novel approach, which will be compared to maximization of MI in the experiments (cf. Section 4).

In more detail, the mutual information can be decomposed into the following components (a derivation is provided in Section Appendix J):

We fit a probabilistic regression to the training data $\mathfrak{L}$, i.e., the query images and the images annotated so far, using a Gaussian process [26, chapter 2] with an RBF kernel given by the kernel matrix $K\in\mathbb{R}^{n\times n}$ over the entire dataset $\mathfrak{X}$:

where $\delta_{ij}$ is the Kronecker delta function with $\delta_{ij}=1\leftrightarrow i=j$ and zero otherwise, and $\sigma_{\mathrm{var}}$, $\sigma_{\mathrm{ls}}$, and $\sigma_{\mathrm{noise}}$ are the hyper-parameters of the kernel.

The computation of the kernel matrix can be performed off-line in advance and does hence not contribute to the run-time of our active learning method. However, if time and memory required for computing and storing the kernel are an issue, alternative kernels for efficient large-scale Gaussian process inference [27] can be used.

The prediction of the Gaussian process for any finite set of $k$ samples consists of a multivariate normal distribution $\mathcal{N}(\mu,\Sigma)$ over continuous values $\hat{y}\in\mathbb{R}^{k}$, where $\mu\in\mathbb{R}^{k}$ is a vector of predictive means of the samples and $\Sigma\in\mathbb{R}^{k\times k}$ is their predictive joint covariance matrix. Let $p(\hat{y})=\mathcal{N}(\hat{y}\mid\mu,\Sigma)$ denote the probability density function of such a distribution.

We use this probabilistic label regression for binary classification by considering samples $x_{i}$ with $\hat{y}_{i}>0$ as relevant. The probability of a given relevance configuration $r\in\{-1,1\}^{n}$ for the samples in $\mathfrak{X}$ is hence given by

with

for $i=1,\dotsc,n$. This is a multivariate normal distribution function, which can be efficiently approximated using numerical methods [11].

The posterior probability $P(R\mid F,u)$ can be obtained in the same way after updating the GP with the expected feedback. For such an update, it is not necessary to re-fit the GP to the extended training data from scratch, which would involve an expensive inversion of the kernel matrix of the training data. Instead, efficient updates of the inverse of the kernel matrix [24] can be performed to obtain updated predictions at a low cost. This is an advantage of our GP-based approach compared with other methods relying on logistic regression (e.g., [14, 23]), which requires expensive iterative optimization for updating the model.

We employ a simple, but plausible user model for $P(F\mid R,u)$, which comes along with a slight simplification of the optimization objective in Eq. 1: First of all, we assume that if we already know the true relevance $r=[r_{1},\dotsc,r_{n}]^{\top}$ of all samples, the feedback $f_{i}$ given by the user for an individual sample $x_{i}$ is conditionally independent from the feedback provided for the other samples. More formally:

Clearly, if a sample $x_{i}$ has not been included in the candidate batch $u$, the user cannot give feedback for that sample, i.e., $x_{i}\notin u\rightarrow f_{i}=0$. Furthermore, we assume that the user will, on average, label a fraction $p_{\mathrm{label}}$ of the candidate samples. For each labeled sample, the user is assumed to provide an incorrect label with probability $p_{\mathrm{mistake}}$. In summary, this user model can be formalized as follows:

The fact that $\left(\exists_{i\in\{1,\dotsc,n\}}:x_{i}\notin u\wedge f_{i}\neq 0\right)\rightarrow P(F=f\mid R=r,u)=0$ allows us to adjust the sum in Eq. 3 to run over only $3^{k}$ instead of $3^{n}$ possible feedback vectors, where $k\ll n$ is the batch size and independent from the size $n$ of the dataset. This is not an approximation, but an advantage of our user model, that decreases the complexity of the problem significantly.

Modeling the user behavior can enable the active learning technique to find a trade-off between learning as fast as possible by asking for feedback for very diverse samples and improving confidence regarding existing knowledge by selecting not extremely diverse, but slightly redundant samples. The latter can be useful for difficult datasets or tasks, where the user is likely to make mistakes or to refuse to give feedback for a significant number of candidates. Nevertheless, the assumption of a perfect user, who labels all samples in the batch and never fails, is an interesting special case since it results in a simplification of the MI term from Eq. 3 and can reduce computation time drastically:

Even with the perfect user assumption, evaluating Eq. 9 still involves a summation over $2^{n}$ possible relevance configurations, which does not scale to large datasets. To overcome this issue, we employ an approximation based on the assumption, that the probability of observing a certain relevance configuration depends only on the samples in the current candidate batch:

This means that we indeed condition $P(R\mid u)$ on the current batch $u$, though actually $P(R\mid u)=P(R)$ holds for the original problem formulation.

This assumption allows us to restrict the sum in Eq. 3 to $2^{k}$ instead of $2^{n}$ possible relevance configurations, leading to the approximate mutual information

with $u=\{x_{i_{1}},\dotsc,x_{i_{k}}\}$ and, by an abuse of notation, $R_{u}=[R_{i_{1}},\dotsc,R_{i_{k}}]^{\top}$ (analogously for $F_{u}$). The number of involved summands now does not depend on the size of the dataset anymore, but only on the number $k$ of candidates chosen at each round for annotation.

On the other hand, this assumption also restricts the estimation of the expected model output change, expressed by the term $P(R_{u}\mid F_{u})/P(R_{u})$, to the current batch, which is probably the most severe drawback of this approximation. However, estimating the model output change for the entire dataset would be too expensive and the experiments in Section 4.4 show that our approach can still benefit from the expected model output change. Future work might explore the option of taking a tractable subset of context into account additionally.

Although the computational effort required to calculate the approximate MI given in Eq. 11 is independent from the size of the dataset, finding the exact solution to the optimization problem from Eq. 1 would still require computing the MI for all possible $2^{m}$ candidate batches $u\subseteq\mathfrak{U}$ of unlabeled samples. Since we do not want to confront the user with an unlimited number of candidates anyway, we set the batch size to a fixed number $k$, which leaves us with a number of $\binom{m}{k}$ possible candidate batches.

Assessing them all would involve a polynomial number of subsets and is, thus, still too time-consuming in general. On first sight, one might think that this problem can be solved more efficiently using dynamic programming, but this is unfortunately not an option since the MI is not adequately separable.

Thus, we follow a linear-time greedy approach to approximate the optimal batch by successively adding samples to the batch [13], taking their relationship to already selected samples into account: We first select the sample $x_{i_{1}}$ with maximum $\tilde{\mathfrak{I}}(R,F\mid u=\{x_{i_{1}}\})$. The second sample $x_{i_{2}}$ is chosen to maximize MI together with $x_{i_{1}}$. This continues until the batch contains $k$ samples.

At each iteration, the unlabeled samples eligible for being added to the current batch can be treated completely independently from each other, allowing for straightforward parallelization.

All datasets used in our experiments consist of multiple classes and are divided into a training and a test set. We define image retrieval tasks for each dataset as follows: Pick a single random instance from the training set of a certain class as query image and consider all other images belonging to that class as relevant, while instances from other classes are irrelevant. Batch-mode active learning is performed for 10 successive rounds with a batch-size of $k=4$ candidates per round and retrieval performance is evaluated after each round by means of average precision on the test set. This process is repeated multiple times with different random queries for each class and we report the mean average precision (mAP) over all repetitions.

Note that our goal is not to achieve state-of-the-art performance in terms of classification accuracy, but with respect to the active learning objective, i.e., obtaining better performance after fewer feedback rounds.

The smallest dataset used is the Butterflies dataset [18], comprising 1,500 images of 5 different species of butterflies captured over a period of 100 years. We use the CNN features provided by the authors and, following their advice, reduce them to 50 dimensions using PCA. A random stratified subset of 20% of the dataset is used as test set.

Second, we use the USPS dataset [10] consisting of 9,300 gray-scale images of handwritten digits, scanned from envelopes by the U.S. Postal Service. The number of images per class is very unevenly distributed. All images have a size of $16\times 16$ pixels and are used without further feature extraction as 256-dimensional feature vectors. We use the canonical training-test split provided with the dataset.

As a more real-word use-case, we perform evaluation on the 13 Natural Scenes dataset [7] and the MIRFLICKR-25K dataset [16]. The former consists of more than 3,400 images from 13 categories of natural scenes such as forests, streets, mountains, coasts, offices, or kitchens. The latter comprises 25,000 images, each assigned to a subset of 14 very general topics such as “clouds”, “tree”, “people”, “portrait” etc. Thus, query images can belong to multiple categories and will be ambiguous. Asking the user the right questions is hence of great importance. There are also “wide-sense annotations” assigning images to categories if they could be related to a small degree. If a candidate image is annotated in this way, we consider it as unnameable during our simulation. For both datasets, we extracted image features from the first fully-connected layer of the VGG-16 convolutional neural network [29], pre-trained for classification on ImageNet, and reduce their dimensionality to 512 using PCA. Experiments in Section Appendix H show that the relative performance of the different methods is not very sensitive w.r.t. the dimensionality of the features. 25% of the natural scenes and 20% of the MIRFLICKR dataset are used as test set.

Finally, we derive further challenging image retrieval tasks from the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) [28], which comprises more than 1,2 million images from 1,000 classes. Following Freytag et al. [9], we obtain binary classification tasks by randomly choosing a single positive and 19 negative classes. This is repeated 25 times and 10 random queries are chosen for each task, leading to a total of 250 image retrieval scenarios. We use the bag-of-words (BoW) features provided with ImageNet instead of CNN features, mainly for two reasons: First, the BoW features are public and hence facilitate reproduction. Second, most neural networks are pre-trained on ImageNet, which could bias the evaluation.

The number of random repetitions per class for the natural scenes and the MIRFLICKR dataset has been set to 10 as well, while we use 25 queries per class for USPS and 50 for the butterflies dataset.

The features of all datasets were scaled to be in $[0,1]$.

We compare ITAL with a variety of baselines and competing methods, including SUD [33], TCAL [5], RBMAL [3], and the method of Brinker [2] referred to as “border_div” in the following. All these native BMAL methods have been described in Section 2.

In addition, we evaluate the following successful one-by-one active learning techniques in the BMAL scenario by selecting the $k$ samples with the highest selection scores: Uncertainty sampling for SVM active learning by choosing samples close to the decision boundary (border) [31], uncertainty sampling for Gaussian processes (unc) [21], where uncertainty is defined as the ratio between absolute predictive mean and predictive standard deviation, and sample selection by maximizing the expected model output change (EMOC) [9].

All methods have to compete against the baselines of random selection, selecting the topscoring samples with maximum predictive mean, resembling the standard retrieval scenario [1], and variance sampling (var) by maximizing the difference of the sum of variances and the sum of covariances in the batch.

Finally, we also investigate maximizing the joint entropy $H(R_{i_{1}},\dotsc,R_{i_{k}})=-\sum_{r\in\{-1,1\}^{k}}P(R_{u}=r)\cdot\log(P(R_{u}=r))$ of candidate batch $u=\{x_{i_{1}},\dotsc,x_{i_{k}}\}$. Being a component of our ITAL method, this is also a novel approach, but lacks the model output change term and the user model (cf. Eq. 2).

We also tried applying USDM [32], MCMI[min] [14] and AdaptAL [23], especially since the latter two also maximize a mutual information criterion. However, all these methods scale so badly to datasets of realistic size, that they could not be applied in practice. AdaptAL, for example, would require 14 hours for composing a single batch on MIRFLICKR, which is clearly intractable. Our ITAL method, in contrast, can handle this dataset with less than a minute per batch. These three competitors could, thus, only applied to USPS, MIRFLICKR, and ImageNet by randomly sub-sampling 1000 candidates to choose from, as suggested by Li et al. [23]. This usually leads to a degradation of performance, as can be seen from the results reported in Section Appendix F.

The hyper-parameters of the RBF kernel, i.e., $\sigma_{\mathrm{ls}}$, $\sigma_{\mathrm{var}}$, and $\sigma_{\mathrm{noise}}$ (cf. Section 3.2), potentially have a large impact on the performance of the active learning methods. However, the overall goal is to eventually obtain a classifier that performs as well as possible. Therefore, we determine the optimal kernel hyper-parameters for each dataset using tenfold cross-validation on the training set and alternating optimization to maximize mean average precision. This optimization aims only for good classification performance independent of the active learning method being used and the same hyper-parameters are used for all methods.

With regard to the hyper-parameters of the user model used by ITAL, we employ the perfect user assumption for being comparable to competing methods that do not model the user. An experiment evaluating the effect of different user model parameters is presented in Section 4.5.

In case that other methods have further hyper-parameters, we use the default values provided by their authors.

Figure 2 depicts the average precision obtained on average after 10 feedback rounds using the different BMAL methods. On the Butterflies dataset, ITAL obtains perfect performance after the least number of feedback rounds. TCAL and border_div perform similar to ITAL on USPS, but ITAL learns faster at the beginning, which is important in interactive image retrieval scenarios. While sampling candidates based on batch entropy behaves almost identical to ITAL on Butterflies, USPS, and MIRFLICKR, it is slightly superior on the Natural Scenes dataset, but fails to improve after more than 4 rounds of feedback on the ImageNet benchmark, where ITAL is clearly superior to all competitor methods. This indicates that taking the effect of the expected user feedback on the model output change into account is of great benefit for datasets as diverse as ImageNet.

Since it is often desirable to compare different methods by means of a single value, we report the area under the learning curves (AULC) in Table 1, divided by the number of feedback rounds so that the best possible value is always 1.0. In all cases, our method is among the top performers, achieving the best of all results in 3 out of 5 cases. The improvement over the second-best method on 13 Natural Scenes and ImageNet is significant on a level of <1% and on a level of 7% on USPS, according to Student’s paired t-test.

The performance of the competing methods, on the other hand, varies significantly across datasets. ITAL, in contrast, is not affected by this issue and provides state-of-the-art performance independent from the characteristics of the data. To make this more visible, we construct a ranking of the tested methods for each dataset and report the average rank in Table 1 as well. ITAL achieves the best average rank and can thus be considered most universally applicable.

This is of high importance because, in an active learning scenario, labeled data is usually not available before performing the active learning. Thus, adaptation of the AL method or selection of a suitable one depending on the dataset is difficult. A widely applicable method such as ITAL is hence very desirable.

To evaluate the effect of the user model integrated into ITAL, we simulated several types of users behaviors on the 13 Natural Scenes dataset: a) an aggressive user annotating all images but assigning a wrong label in 50% of the cases, b) a conservative user who always provides correct labels but only annotates 25% of the images on average, and c) a blend of both, labeling 50% of the candidate images on average and having a 25% chance of making an incorrect annotation.

The same active learning methods as in the previous sections are applied and the parameters $p_{\mathrm{label}}$ and $p_{\mathrm{mistake}}$ of ITAL are set accordingly.

The results presented in Section Appendix G show that the user model helps ITAL to make faster improvements than with the perfect user model. A possible reason for this effect is that the parameters of the user model control an implicit trade-off between diversity and redundancy used by ITAL: For an imperfect user, selecting samples for annotation more redundantly can help to reduce the impact of wrong annotations. However, ITAL performs reasonably well even with the perfect user assumption. This could hence be used to speed-up ITAL noticeably with only a minor loss of performance.