A Reinforcement Learning-driven Translation Model for Search-Oriented Conversational Systems

Our reinforcement learning-driven translation model allows to formulate a user’s information need $x$ expressed in NL into a keyword-based query $y$. The user’s information need $x$ is a sequence of $n$ words ($x=x_{1},...,x_{i},...,x_{n}$). To fit with our word selection objective, the query $y$ is modeled as a binary vector $y\in\{0,1\}^{n}$ of size $n$ (namely, the size of the natural language expression $x$). Each element $y_{j}\in y$ equals to 1 if word $x_{i}\in x$ exists in query $y$ and 0 otherwise. For example, if we consider the NL as ”Identify documents that discuss sick building syndrome or building related illnesses.” and the key-words query as ”sick building syndrome.”, the expected query will be formulated as follows: $y=(0,0,0,0,1,1,1,0,0,0,0)$ .

The objective of our model $f_{\theta}$ (with $\theta$ being the parameters of our model) is to estimate the probability $p(y|x)$ of generating the binary vector $y$ given the NL expression $x$. Since terms are not independent within the formulation of NL expressions and queries, it makes sense to consider that the selection of a word is conditioned by the sequence of decisions taken on previous words $y_{<i}$. Thus, $P(y|x)$ could be written as follows:

This probability is first learned using a maximum likelihood estimation (MLE) on the basis of NL-query pairs (Section 2.2). Then, this probability is refined using reinforcement learning techniques (Section 2.3). We end up with the network architecture used in the translation model.

The translation model works as a supervised word selection model aiming at building queries $y$ by using the vocabulary available in NL expressions $x$. To do so, we use a set $D$ of $N$ NL-query pairs $D=\{(x^{1},y^{1}),...,(x^{k},y^{k}),...,(x^{N},y^{N})\}$.

The objective of the translation model is to predict whether each word $x_{i}^{k}$ in the NL expression $x^{k}$ is included in the expected query $y^{k}$. In other words, it consists in predicting the probability $p(\hat{y}_{i}^{k}=y_{i}^{k}|\hat{y}_{<i}^{k},x^{k})$ that the $i^{th}$ element $\hat{y}_{i}^{k}$ of vector $\hat{y}^{k}$ is equal to the same element $y_{i}^{k}$ in the original query $y^{k}$ (namely, that $\hat{y}_{i}^{k}=y_{i}^{k}$) given the state of previous elements $\hat{y}_{<i}^{k}$ and the NL expression $x^{k}$. This probability $p(\hat{y}_{i}^{k}=y_{i}^{k}|\hat{y}_{<i}^{k},x^{k})$ is modeled using a Bernoulli distribution in which parameters are estimated through the probability distribution.

Let’s define for a NL-query instance $(x^{k},y^{k})$, $f_{(\theta,x^{k})}=\sum_{y_{i}^{k}\in y^{k}}log(p(\hat{y}_{i}^{k}=y_{i}^{k}|\hat{y}_{<i}^{k},x^{k}))$. The translation model is trained by maximizing the following MLE over the set $D$ of NL-query pairs $(x_{k},y_{k})$:

To inject the task objective in the translation model, we consider that the process of query building could be enhanced through reinforcement learning techniques. Therefore, the word selection could be seen as a sequence of choices of selecting word $x_{t}$ at each time step $t$. The choices are rewarded at the end of the selection process by a metric measuring the effectiveness of the query building process within a retrieval task. Particularly, the predicted query $\hat{y}$ obtained from the binary vector $\hat{y}$ is fed to a retrieval model to rank documents. For each NL expression $x$ (and accordingly the associated predicted query $\hat{y}$), we dispose of a set $\mathcal{D}_{x}$ of relevant documents (also called ground truth). We note $GT$ the set of $n$ pairs $(x;\mathcal{D}_{x})$. With this in mind, the effectiveness of the obtained ranking could be estimated using an effectiveness-driven metric (e.g., the MAP). Thus, the reward $R$ for a generated query $\hat{y}$ given the relevance feedback pair $(x,\mathcal{D}_{x})$ is obtained as follows:

At the end of the selection process, the objective function aims at maximizing the expectation of the search effectiveness over the predicted queries:

where $\hat{y}$ is given by the translation model $f_{\theta}(x)$. This objective function is maximized using gradient descent techniques Baxter et al. 1999.

The model is based on an encoder-decoder building a query $\hat{q}$ from the input $x$. Particularly, each element $x_{i}$ of $x$ is modeled through word embeddings $w_{x_{i}}$; resulting in a sequence $w_{x}$ of word embeddings for input $x$. As shown in Figure 2, the encoder is a bi-directional LSTM Hochreiter and Schmidhuber 1997 aiming to transform the input sequence $w_{x}$ to its continuous representation $h_{n}$. The decoder is composed of a LSTM in which each word $x_{i}$ is injected to estimate the word selection probability $p(y_{i}|y_{<i},x)$ using the hidden vector $h_{n}$ learned in the encoder network and the current word $x_{i}$; leading to estimate probability $p(y_{i}|y_{<i},x_{i},h_{n})$.

Since there does not exist yet SOCS-driven datasets including NL-query pairs, we use TREC tracks (namely, Robust 2004 and Web 2000-2001). In these tracks, query topics include a title, a topic description and a narrative text; the two latter being formulated in natural language. To build query-NL pairs, we use the title to form the set of keyword queries and the description for the set of information needs expressed in NL. An example of a query-NL pair is: Title Lewis and Clark expedition Description What are some useful sites containing information about the historic Lewis and Clark expedition?

This NL-query building process results in 350 pairs in total as presented in Table 1.

We are aware that the use of TREC datasets is biased in the sense that it does not exactly fit with the expression of NL information need in the context of conversational systems, but we believe that the description is enough verbose to evaluate the impact of our query building model in this exploratory work. Further experiments with generated datasets, as done in Song et al. 2017, will be carried out in the future.

We also analyze the issue of duplicate words into TREC descriptions since it can directly impact the query formulation process based on word selection in the word sequence of TREC descriptions. In practice, this might lead to select several times the same word to build the query, and, therefore, directly impact the retrieval performance. As shown in Table 1, the ratio of duplicate words in TREC descriptions over the whole set of queries is very low (1.1 duplicate words in average in each query for TREC Robust and 0.65 for TREC Web). This suggests that this issue is minor in the used datasets. We, therefore, decided to skip this issue for the moment.

To evaluate our approach, we measure the retrieval effectiveness of the predicted queries. To do so, for each predicted query, we run the BM25 model through an IR system (namely, PyLucene¹¹ 1 http://lucene.apache.org/pylucene/) to obtain a document ranking. The latter is evaluated through the MAP metric.

To show the soundness of our approach (namely, translating information needs expressed in NL into queries), we compare our generated queries to scenario NL feeding the natural language information needs (TREC descriptions in our protocol) to the IR retrieval system.

Since the objective of our model is to formulate queries, we also evaluate the effectiveness of original TREC titles (scenario Q). This setting rather refers to the oracle that our model must reach.

We mentioned that before training the selection model we transformed each $x$ to its binary representation $y$ based on the presence of the words in the ground truth query. The dataset being slightly biased by this binary modeling, we observed that not all the words existing in the query do exist in $x$. To analyze this bias, we also compare our approach with these binary queries (scenario Q bin) referring to the projection of queries Q on the vocabulary available in the NL description.

We also compare our model to a random approach which randomly selects 3 words from $x$ to build queries (scenario Random).

Different variants of our model are also tested:

• •

  SMT which only considers the first component of our model based on a supervised machine translation approach (Section 2.2). This variant could be assimilated to the approach proposed in Song et al. 2017 in the sense that the machine translation is performed independently of the task objective.

• •

  RL which only considers the reinforcement learning objective function (Section 2.3) without pre-training of the supervised translation model.

• •

  SMT+RL which is our full model in which we start by pre-training the model using the supervised translation model (Section 2.3), and, then, we inject the reward signal in the translation probabilities (Section 2.4).

To transform each word $x_{i}$ to its vector representation $w_{x_{i}}$, we use Fasttext ²² 2 https://github.com/facebookresearch/fastText/ Bojanowski et al. 2017 pre-trained word embeddings. The encoder and decoders have one hidden layer with 100 hidden units each.

To train our model, we perform 10-fold cross-validation. For the SMT+RL model, we start by a pre-training using the supervised translation model for 100 iterations. The training is then pursued by 1000 iterations while including the reinforcement learning approach. In the latter, the reward, namely the MAP metric, is estimated over document rankings obtained by the BM25 model in PyLucene. We use a minibatch Adam Kingma and Ba 2014 algorithm to pre-train the model and SGD for the reinforcement learning part. Each update is computed after a minibatch of 12 sentences.