PBODL : Parallel Bayesian Online Deep Learning for Click-Through Rate Prediction in Tencent Advertising System

In Tencent advertising platform, we choose ads from millions of candidates and serve them to hundreds of millions of users every day. In contrast to sponsored search, it is difficult for us to find out the instant needs of users, since we don’t have search keywords by the time serving ads to them. As a result, advertising recommendations are usually made based on user’s historical behaviors and context information.

Click-Through rate in advertising settings can be modeled as:

$ctr=P(click|user,ad,context)$

Features used in the conditional part are usually divided into three categories :

• •

  user features : age, gender, interest, etc;

• •

  Ad features : advertiser id, ad plan id, material id, ad industry, etc;

• •

  Context feature : time, location, connection type, etc.

The click-through rate prediction in tencent advertising platform faces many challenges :

• •

  New Ads : Ad repository within our advertising system is updated frequently. 10% to 20% of the inventory could be replaced with new ads in each day. At the same time, advertisers are allowed to change their targeting rules during the campaign. Apparently, a daily or hourly updated model is not quick enough to support such kind of constant changes, and usually results in suboptimal revenue and advertisers’ complaints according to our experience.

• •

  Features Engineering : As in all machine learning scenarios, features used dominate the performance of click-through rate prediction models. Unfortunately, feature engineering is a very time and resource consuming process. It demands more time and resource when you take more features into account, and could become the bottleneck of the whole modeling process.

• •

  Low latency : The runtime latency spent on CTR prediction could not be large without negative impacts on user experience. In mobile advertising scenario, our latency budget for CTR prediction is about 10 ms. Considering the number of candidate ads, this is a very limited budget.

• •

  Big Data : Click-through rate prediction is a massive-scale learning problem from all perspectives with no doubt. Billions of training samples are generated each day, leaving out the number of users and ads involved.

In the face of the above challenges, we set the following design goals for our solution:

• •

  Rapid Model Update : We think rapid model update is one of the most effective ways to deal with constantly changing recommendation scenarios. Considering the scale of the learning problem, online learning (bayesian online learning in particular) paradigm is chosen as a corner stone of our solution.

• •

  Nonlinear Model : Linear models depends heavily on feature engineering, which is both time and resource consuming. Besides that, the size of a linear model explodes quickly while introducing more higher-order features. Inspired by the end-to-end training idea of deep learning, we choose deep nonlinear models as another corner stone of our solution.

• •

  Balance of Complexity and Latency : Since the latency budget for CTR predictions per request is limited (10ms), deep models with dozens or hundreds of layers are impractical for real-life deployment. In fact, deep models with 3   5 hidden layers are proposed currently. In addition, each layer/operation involved should be designed and implemented as efficiently as possible (scale up).

• •

  Scale Out : The solution should be able to scale out so as to support billions or even tens of billions of training samples per day.

Logistic regression with cross-features is an early well-known solution to CTR problem(M. Richardson 2007; O. Chapelle & Rosales 2015). But it depends on complex features engineering and suffers curse of dimensionality. OWLQN(G Andrew 2007) proposed by Microsoft can effectively pruning model and has been widely applied in many recommendation scenarios.

In recent years, nonlinear models and online learning have got great attention in the field of CTR prediction. Factorization Machines (FM)(Rendle 2010; Rendle 2012) depends less on features engineering and is widely used in various competitions. A variant of FM called Field-aware Factorization Machines (FFM) has been used to win two click-through rate prediction competitions hosted by Criteo and Avazu(Lin 2016). GBDT is another common solution in click-through rate prediction competitions and also widely used in industry.

Inspired by deep learning(A. Krizhevsky & Hinton 2012; A. Graves & Hinton 2013), deep neural network is getting more popular in both competitions and industrial scenarios. But traditional deep neural networks cannot be directly applied to high-dimensional feature spaces, so many studies have focused on constructing better embedding layers. Sampling-based Neural Network (SNN) uses a regular embedding layer and Factorisation Machine supported Neural Network (FNN) initializes the embedding layer with the result of a pre-trained FM model(W. Zhang & Wang 2013). Unlike FNN, Product-based Neural Network (PNN)(Yanru Qu 2016) can end-to-end learn the local dependencies which is similar to FM. Besides studis in embedding layers, Wide and Deep model(Cheng et al. 2016) proposed by Google combines deep neural networks and linear models and significantly increases app acquisitions compared with wide-only and deep-only models in Google Play store. Convolutional neural networks (CNN) or Recurrent neural networks (RNN) are also tried with applications in CTR systems(Q. Liu & Wang 2015; Yuyu Zhang 2014).

Although the models above have achieved state of art performance at their time, their experiments were mainly done on offline datasets with batch training. In real world online advertising scenarios, the system has to balance the accuracy of the model on history (training) data and the latency before pushing the model online. So many companies have tried online learning paradigm. AdPredictor(T. Graepel & Herbrich 2010) proposed by Microsoft and FTRL(H. B. McMahan & Kubica 2013) proposed by Google are two of the most famous models. But these two models are both linear models and suffer features engineering related costs. Facebook introduces a hybrid model which combines decision trees with online linear model and the new model outperforms either of these methods on their own by over 3%(X. He & Candela 2014). MatchBox(David Stern 2009) proposed by Microsoft is an online matrix factorization model but it’s hard to port its learning algorithm to more complex models like DNN.

In this paper, we describe a parallel bayesian online deep learning framework used for click through rate prediction in Tencent advertising system. Section 2 introduces the framework with a bayesian deep probit model trained with probabilistic backpropagation in the mode of assumed Gaussian density filtering(Herna ́ndez Lobato & Adams 2015; Minka 2001), then extends the model famility with some novel online deep models and presents a parallel model updating framework(Broderick & Jordan 2013; Minka 2001). Section 3 compares these models with common models such as AdPredictor, FTRL, FM and MLP in both offline datasets and online experiments. Section 4 gives some notes when we apply this framework. Finally, Section 5 presents conclusions and future work.

The main contributions of the paper are:

• •

  A general parallel bayesian online deep learning framework sutiable for a variety of bayesian online models.

• •

  Some novel bayesian online deep model with efficient training and predicting operations.

• •

  We applied the framework and these bayesian online models in the Tencent Advertising system and get significant improvements over commonly used models.

The deep probit model is a feed-forward neural network with a probit link function like Figure 1.

Given a data point$(\boldsymbol{x},y)$, made up of a input feature vector $\boldsymbol{x}$ and a label variable $y$, we assume that $y$ is obtained as :

where $f(\boldsymbol{x},\boldsymbol{\omega})$ is the output of the feed-forward neural network with weights given by $\boldsymbol{\omega}$ and $\boldsymbol{x}$, and $\epsilon$ is an additive noise, where $\epsilon\sim N(0,1)$.

Because of sparse input features, the first layer of the model is a embedding layer, where $\boldsymbol{\omega}_{e}$ is its weight and $\boldsymbol{z}_{e}$ is its output. For get better embedding of sparse input features, we add a common embedding op layer which is similar to inner product layer in PNN. In general, this layer is parameterless. Considered the amount of calculation, we introduce three effective embedding op layer. Then the output of embedding op layer is used as the input of a common deep neural model with multiple hidden layers. The number of hidden layers is $L$. There are $V_{l}$ hidden units in layer $l\in{[1,L]}$ and $\boldsymbol{\omega}_{l}$ is the weight matrices between hidden layer ${l-1}$ and hidden layer $l$. We denote the output of layer $l$ by $\boldsymbol{z}_{l}$ and the input of layer $l$ by $\boldsymbol{a}_{l}=\boldsymbol{\omega}_{l}\boldsymbol{z}_{l-1}/\sqrt{V_{l-1}+1}$. Specially, $\boldsymbol{z}_{0}$ is the output of embedding op layer and $\boldsymbol{z}_{L}=f(\boldsymbol{x},\boldsymbol{\omega})$. The activation functions for each hidden layer are rectified linear units (ReLUs).

We have described the general structure of our CTR model. Then we need to know how to inference on the model. Given input feature vector $\boldsymbol{x}$ and $\boldsymbol{w}$, the CTR can be written as :

To complete our probabilistic model, we specify a Gaussian prior distribution for each of $\boldsymbol{w}$. In particular,

where $m_{e},m_{l}$ is prior mean and $v_{e},v_{l}$ is prior variance.

Considered the prior distribution, predictive distribution can be derived as

Given $p(y|\boldsymbol{x},\boldsymbol{w})$ and the prior $p(\boldsymbol{w})$, the posterior distribution for the parameters $\omega$ can then be obtained by applying Bayes’ rule:

Both the exact predictive distribution and exact posterior over weights can’t be calculated in simple closed form. So we use probabilistic backpropagation(PBP)(Herna ́ndez Lobato & Adams 2015) in the mode of assumed Gaussian density filtering to approximate them. PBP is a scalable method for learning Bayesian neural networks. Similar to classical backpropagation, PBP works by computing a forward propagation of probabilities through the network and then doing a backward computation of gradients. First, PBP propagates distributions forward through the network and approximates each new distribution with a Gaussian when necessary. Then PBP computes the gradients of weights and update each of $\boldsymbol{\omega}$ with following rules(Minka 2001; Herna ́ndez Lobato & Adams 2015):

where $Z$ is the the normalization constant :

When predicting with model, we only need to do a forward computation. Because PBP will approximates each new distribution with a Gaussian when necessary. So $\boldsymbol{z}_{L}$, the output of last hidden layer will be be approximated as a Gaussian variable with mean $m^{\boldsymbol{z}_{L}}$ and variance $v^{\boldsymbol{z}_{L}}$. So the predictive distribution can be approximated as:

To update the weights of model, we need to compute the normalization constant $Z$. With equations (8) and (9), we can approximate $Z$ as :

However, it remains to compute the mean and variance parameters $m^{\boldsymbol{z}_{L}}$ and $v^{\boldsymbol{z}_{L}}$ through the network. In embedding layer, $\boldsymbol{x}_{i}$, the $i$-th element of $\boldsymbol{x}$ is transformed to a dense value vector $\boldsymbol{\omega}_{e,\boldsymbol{x}_{i}}$. Then all dense value vectors are concated as $\boldsymbol{z}_{e}$.

In embedding op layer, we can design a variety of operations to get better embedding. The simplest operation is copying input to output :

More operations will be introduced in Section 2.2.

In $l$-th hidden layer, using moment match, the mean and variance of $\boldsymbol{a}_{l}$ can be approximated as :

Let $\boldsymbol{z}_{l}=max(0,\boldsymbol{a}_{l})$, the mean and variance of the $i$-th element of $\boldsymbol{z}_{l}$ can be approximated as :

where

Finally, we can get the mean and variance of $\boldsymbol{z}_{L}$.

In 2.1, we have described a deep probit model with a simplest embedding op layer and how to online learn it and predict with it. In our practice, we find that we can get better performance in online experiment by using better embedding operation layer such as FM and FFM. Here we introduce three novel embedding operations : DimensionAwareSum, FM and FFM. Corresponding to the three operations, we propose three novel models called Sparse-MLP, FM-MLP and FFM-MLP.

Every day there are ten of billions of samples in tencent advertising system. The CTR model must be trained in parallel mode. Here we introduce a simple and effective parallel update framework for our Bayesian online models.

We store global parameters in parameter server and use multiple workers in storm for training data. In time $T$, assumed the parameters in parameter server as prior $P(\theta)$ and minibatch data in $i$-th worker as $D_{i}$, then the posterior can be obtained by applying Bayes’ rule:

Because $p(\theta|D_{1},D_{2},...,D_{M})$ and $p(\theta)$ are Gaussian, the likelihood $p(D_{i}|\theta)$ can be approximated as Gaussian. In $i$-th worker, we can first compute the $p(\theta|D_{i})$ using PBP. The $p(D_{i}|\theta)$ can be obtained by equation 27 and then is updated to parameter server.

In the case of Gaussian, the flow of training worker and parameter server is as algorithm 1 and algorithm 2.

Above framework can deal with big data in the real world. We have applied it into Tencent advertising system and got a steady performance.

However, offline evaluation can only be used as reference and it isn’t easy to compare online models with batch models in offline datasets. Online environment is more complex, so we generally using A/B test to evaluate the real effect of the models. We have applid our framework and serveral models into tencent advertising system and got a good online effect. Because of the difficulty of feature engineering, we only compare our models with FM and DNN using batch training which has achieved the best effect. The online effect is shown in Table 3. Our online models are better than DNN. An important reason is that our online models can update every 15 minutes while DNN can only update every several hours. So the online models can perform much better in new ads, just as Tabel 4. Meanwhile, the percentage of new Ads using online models is higher than that using batch models, so it can get long term profit to use our bayesian online models. In addition, our online models also achieve cpm lift comparable with ctr lift.

Typical CTRs are much lower than 50%, which means that positive examples (clicks) are relatively rare. Thus, simple statistical calculations indicate that clicks are relatively more valuable in learning CTR estimates. We can take advantage of this to significantly reduce the training data size with minimal impact on accuracy. We remain all positive examples and subsampling negative examples with a sample rate $w$. Then the predicted CTR as p, the recalibrated CTR $q$ should be $q=p/(p+(1-p)/w)$. In general, we will select a suitable $w$ to make the ratio of positive and negative samples being about 0.1.

Online models need long day’s data to get stable effect. In practice, we want to know the real effect of experiment such as a new model and a new feature as soon as possible. For this purpose, we need to burn a new model with last serveral day’s data. In general, we use 15-30 day’s data to burn a new model.Because the complexity of deep model, we propose training the first several day’s data with a single thread in burning and then with muti multiple threads. Otherwise the effect of new model is often unstable.

Because our model can be online learning, so the prior is not very important. The prior mean can be set to zero and prior variance can be set to 0.01 because more accurate and stable effect can be got by using smaller prior variance.

If the model has run a long time, some variances of weights would converge towards zero and learning would come to a halt. Like AdPredictor and MatchBox, we make these variances converging back to the prior variance.

where $v$ is the current variance and $v_{prior}$ is the prior variance.

Though our models only need to one pass of data, we can use more data or learn same data with multiple times for better performance. In fact, we test two methods to improvement effect but keep realtime update.

• •

  Multiple Data : We can use two or more sets of training workers with same config. These sets training the data with same positive examples and different negative examples because of negative sampling. The then get and update parameters with a same parameter server.

• •

  Batch Training : We only use one set of training workers. But in every minibatch data, we run multiple ADF or EP on the data.

In our online experiments, both above methods can lead to better performance. However, the two methods is less safe when encountering abnormal data. So it is greater challenges to daily operations.