PFDet: 2nd Place Solution to Open Images Challenge 2018
Object Detection Track

We use a two-stage Faster R-CNN style object detection framework ren2015faster and leverage an SE-ResNeXt or SENet hu2017squeeze model as the backbone feature extractor. To increase the global context information in the extracted features, we add an FPN and a pyramid spatial pooling (PSP) zhao2017pyramid module to the backbone. Additionally, we increase the context information in the head network by concatenating features from twice the area around each RoI to the head before the fully-connected layers zhu2017couplenet. We increase the number of scales of features extracted by the feature extractor to five from four, which is used in the original work of FPN lin2017feature. This modification allows the network to gather even greater global context information.

Batch normalization (BN) is used ubiquitously to speed up convergence of training ioffe2015batch. We use multi-node batch normalization megdet to share batch statistics of images across computing nodes so that the number of images used for collecting statistics is sufficiently large to stably compute the statistics. To maximize the effectiveness of BN, we add BN layers to the top-down path of FPN in addition to the BN layers included in the base feature extractor, and the head network.

We train the CNN by first expanding ground truth labels to include all ancestor classes using the semantic hierarchy prepared in OID. We formulate the learning problem as in a multi-label setting, and we use a sigmoid cross entropy loss for each class separately uijlings2018revisiting. In the case when the ground truth class is not a leaf of the semantic hierarchy, we do not compute sigmoid cross entropy loss for descendants of the ground truth classes.

We use non-maximum weighted (NMW) zhou2017cad suppression during test time to reduce duplicate detections. It was found that this works better than standard non-maximum suppression (NMS). NMS was used in the RPN while training.

In the OID dataset, for each image, only the instances of classes that are verified are annotated. In Figure 1, images from OID with different coverage of verified labels are shown. In Figure 1a, relatively more human part classes are labeled compared to Figure 1b, which has no annotation of human part classes such as Human face. With sparse annotations, conventional losses for training CNN-based object detectors lin2017feature would falsely label regions around instances as negatives.

To reduce false training signals, we introduce co-occurrence loss. The main idea behind this loss is that for some classes, the relationship with other classes is informative enough that we can safely ignore false training samples. For instance, given a ground truth bounding box of a person, it is highly likely that a human face exists inside the box even if human faces are not verified to exist in the image by the human annotator. To implement the loss, we gather pairs of classes that satisfy the relationship “if a proposal is inside a ground truth box of class $X$, it is safe to ignore treating the proposal as a negative sample of class $Y$”. We use a pair when class $Y$ is a part of class $X$ or instances of class $Y$ is usually a possession of an instance of class $X$. For instance, tires are parts of cars and jeans are usually possessed by a person.

In OID, there is an extreme class imbalance, which makes it difficult for a model to learn rare classes. For instance, there are $238$ classes that are annotated in less than $1000$ images, but the most common class Person is annotated in $807$k images. We use expert models fine-tuned from a model trained with the entire dataset. Each expert model is fine-tuned on a very small subset of the full category space, which we find to perform well for rare classes.

For the final submission, we use an ensemble of models trained on all $500$ classes and the expert models. We do not apply duplicate suppression for individual models, but instead apply suppression once on the concatenation of the outputs of all models.

Since the distribution of the performance over classes is different among models, we prioritize outputs of models that are expected to perform better based on validation scores. For each model $m$, we compute a weight $w_{c}^{m}$ for class $c$, which is multiplied to the confidence scores of the outputs of the model for this class. Suppose that the mean of the validation scores of all models for class $c$ to be $\mu_{c}$, we set the weight $w_{c}^{m}$ to $\alpha$ if model $m$ performs lower than the average $\mu_{c}$. Otherwise, we compute the weight by simply interpolating between $\alpha$ and $1$ linearly. The computation is done as $w_{c}^{m}=\frac{s_{c}^{m}-\mu_{c}}{t_{c}-\mu_{c}}+\alpha\frac{t_{c}-s_{c}^{m}}{t_{c}-\mu_{c}}$, where $s_{c}^{m}$ is the validation score of the model $m$ for class $c$ and $t_{c}$ is the highest validation score for class $c$.

We use Chainer tokui2015chainer as our deep learning framework, ChainerMN akiba2017chainermn to scale up training and ChainerCV niitani2017chainercv for quick prototyping. For training, we used MN-1b, an in-house cluster owned by Preferred Networks, Inc. It consists of 64 nodes, where each node has two Intel Xeon Gold 6154 CPUs (3.0 GHz, 18 cores), 384 GB memory and eight NVIDIA Tesla V100 (32 GB memory). The nodes are interconnected by Mellanox Infiniband EDR.

We first study the effectiveness of different techniques on the validation set. Our baseline is FPN lin2017feature with SE-ResNeXt-101 hu2017squeeze as the backbone trained with sigmoid loss for $12$ epochs. The baseline model learning rate is multiplied by $\tfrac{1}{10}$ at epoch $8$ and $11$. Mean average precision (mAP) of the validation split over $500$ classes is shown in Table 1. For the final model, we increased the variation of anchors by using very tall and very wide ones. The ratios of the width and height of the anchors are $[\tfrac{1}{3},\tfrac{1}{2},1,2,3]$. Using 512 GPUs, the computing time of training the final model was 33 hours for $16$ epochs. The scaling efficiency when using 512 GPUs was $83\%$ in comparison to the single-node (i.e. 8 GPUs) baseline.

We also show the results after ensembling models in Table 2. Our final model outperformed the competition winner in the public leaderboard by $1.17$ mAP and falls behind them by only $0.03$ mAP on the private leader board.

In Table 3, we show a comparison of a model trained with co-occurrence model and a model with identical setup except for the co-occurrence loss. For $47$ classes affected by co-occurrence loss, we see $9.2$ AP improvement on average.

In Table 4, the ablative study of expert models is shown. For the rarest 11-250 classes, we see improvement with expert models. However, for more common classes (251-350), the fine-tuning has a negative effect. Also, we consistently see better results if we reduce the size of the classes to which expert models are fine-tuned.