Towards Large Scale Training Of Autoencoders For Collaborative FilteringConference: Late-Breaking Results track part of the Twelfth ACM Conference on Recommender Systems; October 2-7, 2018; Vancouver, BC, Canada

Abstract.

In this paper, we apply a mini-batch based negative sampling method to efficiently train a latent factor autoencoder model on large scale and sparse data for implicit feedback collaborative filtering. We compare our work against a state-of-the-art baseline model on different experimental datasets and show that this method can lead to a good and fast approximation of the baseline model performance. The source code is available here .

1. Introduction

Linear latent factor models (Hu et al. 2008) are the most popular collaborative filtering methods in the industry due to their simplicity and efficiency. Recent advances have shown that making these models learn non-linear representations by generalizing them within Autoencoder framework can achieve better performance (Liang et al. 2018). However, one problem with training these models is that it involves the reconstruction of highly sparse data with large vector size.

In this work, we present a negative sampling method to efficiently train a latent factor autoencoder model. This method is based on the simple idea of sampling, for each user, only the negative items the other users in the mini-batch have interacted with. This is not the first paper to use such method, Hidasi et. al (Hidasi et al. 2015) used a similar one but applied for generating recommendations based on short session data with recurrent neural networks.

2. Method

The user-item interactions matrix is represented as $\textbf{X}\in\{0,1\}^{\left|{U}\right|\times\left|{I}\right|}$ where $U$ and $I$ are the sets of users and items in the dataset, respectively, and $\textbf{X}_{u,i}=1$ if at least one interaction was observed between user $u$ and item $i$, otherwise $\textbf{X}_{u,i}=0$. Given an item $i$ and a user $u$, $U_{i}$ represents the set of users who interacted with $i$, and $I_{u}$ represents the set of items $u$ has interacted with.

2.1. Model

We learn a model $p(\textbf{x}_{u}|\textbf{z}_{u},\theta)=h(g_{\theta}(\textbf{z}_{u}))$, where $\textbf{x}_{u}={\textbf{X}_{u,*}}^{T}$ is the user $u$ vector of interactions, $\textbf{z}_{u}$ is the user latent factor, $g_{\theta}$ is a multi-layer perceptron parameterized by $\theta$, and $h$ is a function that maps the output of $g_{\theta}$ to probabilities based on the likelihood distribution used to model $p(\textbf{x}_{u}|\textbf{z}_{u},\theta)$. $\textbf{z}_{u}$ can be a learned model parameter (Hu et al. 2008), or can be computed as a function of $\textbf{x}_{u}$ such that $\textbf{z}_{u}=f_{\lambda}(\textbf{x}_{u})$, where $f_{\lambda}$ is a multi-layer perceptron parameterized by $\lambda$ (Liang et al. 2018). The advantage of the second approach over the first is that the model number of parameters scales linearly with the number of items only $O(\left|{I}\right|)$, whereas in the first it scales linearly with both the number of items and users $O(\left|{U}\right|+\left|{I}\right|)$. Gaussian likelihood is commonly used to model $p(\textbf{x}_{u}|\textbf{z}_{u},\theta)$ (Hu et al. 2008), and two newly studied likelihoods are logistic (Liang et al. 2018) and multinomial likelihoods (Liang et al. 2018). The order of increasing performance of those likelihoods is as follows: gaussian, logistic and then multinomial (Liang et al. 2018).

In this paper, we compute $\textbf{z}_{u}=f_{\lambda}(\textbf{x}_{u})$, such that $g_{\theta}\circ f_{\lambda}$ forms an autoencoder, and we model $p(\textbf{x}_{u}|\textbf{z}_{u},\theta)$ as a logistic likelihood since it approximates the performance of the multinomial likelihood (Liang et al. 2018) and frees us from having a huge softmax at the output layer. The negative log-likelihood loss function for our model to be minimized is then:

$$-\sum_{i}\log p(\textbf{x}_{u}|\textbf{z}_{u},\theta)_{i}=-\textbf{x}_{u}\mathchoice{\mathbin{\vbox{\hbox{\scalebox{.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{.5}{$\scriptscriptstyle\bullet$}}}}}\log(g_{\theta}(\textbf{z}_{u}))-(\textbf{1}-\textbf{x}_{u})\mathchoice{\mathbin{\vbox{\hbox{\scalebox{.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{.5}{$\scriptscriptstyle\bullet$}}}}}\log(\textbf{1}-g_{\theta}(\textbf{z}_{u}))$$

To regularize the model, we apply dropout at the input layer and the model is then optimized to denoise the corrupted version $\tilde{\textbf{x}}_{u}$ of $\textbf{x}_{u}$ (Vincent et al. 2008), in addition to that we apply L2 weight decay on $\theta$ and $\lambda$.

2.2. Negative Sampling

For each user in the training mini-batch $\textbf{M}\in\{0,1\}^{m\times\left|{I}\right|}$, where $m$ is the number of users in the batch (batch size), we sample only the negative items that the other users in M have interacted with. In other words, we are training the model, at each mini-batch, only on the set of items that have been interacted with by the users in the mini-batch instead of the whole set of items. Such negative sampling procedure approximates the sampling from a distribution that is biased towards popular items, which is a good property to have compared to an unbiased negative sampling. A user is more likely to be having no preference for a popular item that he hasn’t interacted with. However, there are many likely reasons that could make that user not interact with an unpopular item, such as exposure, or the freshness of the item. Theoretically, popular items contribute the most to the variance in X, so they should be given more weight when approximating X’s reconstruction.

Given that at the start of each training epoch the users are uniformly shuffled, and given a user $u$ and an item $i$ such that $\textbf{M}_{u,i}=0$, the probability that $i$ will be sampled for $u$ is equal to

$$P(\textbf{M}_{*,i}>\textbf{0})=min(\frac{\left|{U_{i}}\right|}{N},1)$$

where $N=\left\lceil{\frac{\left|{U}\right|}{m}}\right\rceil$ is the number of training mini-batches. In order to tune the sampling probability, one has to tune the batch size $m$. Having low batch size $m$ can make the sampling highly biased towards popular items, which can lead to overfitting the reconstruction on those popular items. On the other hand, having high batch size will sample almost all items, and make the sampling obsolete.

The motive behind this sampling method is that it’s simple to implement and can speed up both forward and backward propagation through the autoencoder. The idea is, given that M is represented as a sparse matrix in coordinate list format, we can sample efficiently the non-zero columns in M into a new dense matrix $\textbf{M}_{s}$ and reconstruct $\textbf{M}_{s}$. It can be proven that training the model with this negative sampling procedure reduces the time complexity of one training iteration over all users from $O(\left|{U}\right|\times\left|{I}\right|)$ to $O(m\times\sum_{u\in U}\left|{I_{u}}\right|)$, so the time complexity of training the model scales linearly with the number of interactions in the dataset.

For large datasets with large number of items, we need a large number of negative samples, hence a large batch size, which makes the batch, not fit in memory and expensive to train on. In that case, we can simply generate the sparse batch with a large batch size and then slice it into smaller batches, and train on the small batches.

3. Experiments

In our experiments, we follow a similar experimental setup to Liang et. al (Liang et al. 2018), and use their model as a baseline. We experiment with two datasets varying from small-scale to large-scale: ML-20M (Harper and Konstan 2015), a movies rating dataset, and MSD (Bertin-Mahieux et al. 2011), a songs listening count dataset. We create three variations of these datasets based on the below filtering. Datasets statistics are shown in Table 1:

• ML-20M: All ratings above 4 are taken as positive feedback, otherwise they are taken as negative feedback. Only the users who have rated at least 5 movies are kept.

• MSD: The users who have listened to less than 20 songs, and the songs that have been listened to by less than 200 users are filtered out.

• MSD-Large: The users who have listened to less than 20 songs, and the songs that have been listened to by less than 50 users are filtered out.

Table 1. Datasets statistics

	ML-20M	MSD	MSD-Large
# of users	136,677	571,355	629,112
# of items	20,108	41,140	98,485
# of interactions	10.0M	33.6M	39.7M
sparsity %	0.36%	0.14%	0.064%
# of val/test users	10K	50K	50K

3.1. Setup

The datasets are split by user into train/validation/test sets. The number of val/test users are shown in Table 1. For each val/test user, 80% of his interactions are used to predict the other 20%, then the model is evaluated based on those predictions.

On all datasets, we use the same model architecture, we use an autoencoder with a single hidden layer of dimension 200. We set the dropout probability at the input layer to 0.5. The weight decay is set to $2e^{-5}$. We optimize the model using Adam (Kingma and Ba 2014) in batches of 500. For our model, a batch size of 500 was chosen by cross-validation since we noticed that for low batch sizes the model was overfitting and the validation NDCG@50 metric starts decreasing after few epochs. We train on ML-20M for 100 epochs, while we train on both MSD and MSD-Large for 80 epochs.

3.2. Results

To evaluate the recommendation performance of the model, we use the Recall@K and the NDCG@K metrics. In Table 2 we compare the recommendation performance of the baseline model versus our model. In Table 3 we show the time performance of training both models on a CPU and a GPU. The CPU is an 8 cores Intel^® Xeon^® E5-2686 v4 and the GPU is a Nvidia^® Tesla K80. We also show the mean size of the downsampled input vector $\textbf{M}_{s}$ for a batch size of 500, the standard deviation of input vector size was less than 800 on all datasets. The size of the input vector to the baseline model is always equal to the number of items in the dataset irrelevant of the batch size.

It can be seen that our model approximates very well the performance of the baseline model with less than 3.72% decrease in recommendation performance while having more than 2.3x speed-up on CPU and more than 2.0x speed-up on GPU.

In future work, we plan to have a thorough comparison of the performance of our work versus other fast implementations of the matrix factorization (WMF (Hu et al. 2008)).

Table 2. Comparison between the recommendation performance of the baseline model and our model.

	ML-20M		MSD		MSD-Large
	Baseline	Ours	Baseline	Ours	Baseline	Ours
Recall@20	0.3890	0.3884	0.2677	0.2633	0.2552	0.2453
Recall@50	0.5226	0.5215	0.3540	0.3485	0.3345	0.3233
NDCG@100	0.4212	0.4193	0.3199	0.3137	0.3059	0.2945

Table 3. Comparison between the training time performance of the baseline model and our model. The time performance metric is number of batches processed per second.

	ML-20M		MSD		MSD-Large
	Baseline	Ours	Baseline	Ours	Baseline	Ours
Input size	20108	5085	41140	15430	98485	18740
CPU rate	2.1	5.6	1.1	2.5	0.5	1.6
CPU Speed-up	2.6x		2.3x		3.2x
GPU rate	20.6	43.5	10.4	21.0	4.3	14.3
GPU Speed-up	2.1x		2.0x		3.32x