Pointwise HSIC: A Linear-Time Kernelized Co-occurrence Norm
for Sparse Linguistic Expressions

Computing the co-occurrence strength between two linguistic expressions is a fundamental task in natural language processing (NLP). For example, in collocation extraction Manning and Schütze 1999, word bigrams are collected from corpora and then strongly co-occurring bigrams (e.g., “New York”) are found. In dialogue response selection Lowe et al. 2015, pairs comprising a context and its response sentence are collected from dialogue corpora and the goal is to rank the candidate responses for each given context sentence. In either case, a set of linguistic expression pairs $\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{n}$ is first collected and then the co-occurrence strength of a (new) pair $(x,y)$ is computed.

Pointwise mutual information (PMI) Church and Hanks 1989 is frequently used to model the co-occurrence strength of linguistic expression pairs. There are two typical types of PMI estimation (computation) method.

One is a counting-based estimator using maximum likelihood estimation, sometimes with smoothing techniques, for example,

where $c(x,y)$ denotes the frequency of the pair $(x,y)$ in given data $\mathcal{D}$. This is easy to compute and is commonly used to measure co-occurrence between words, such as in collocation extraction¹¹ 1 In collocation extraction, simple counting $c(x,y)\propto\widehat{\mathbf{P}}(x,y)$, rather than PMI, ranks undesirable function-word pairs (e.g., “of the”) higher Manning and Schütze 1999.; however, when data $\mathcal{D}$ is sparse, i.e., when $x$ or $y$ is a phrase or sentence, this approach is unrealistic. The second method uses recurrent neural networks (RNNs). Li et al. 2016 proposed to employ PMI to suppress dull responses for utterance generation in dialogue systems²² 2 In dialogue response selection or generation, a simple conditional probability $\widehat{\mathbf{P}}(y|x)$, rather than PMI, ranks dull responses (e.g., “I don’t know.”) higher Li et al. 2016.. They estimated $\mathbf{P}(y)$ and $\mathbf{P}(y|x)$ using RNN language models and estimated PMI as follows:

This way of estimating PMI is applicable to sparse language expressions; however, learning RNN language models is computationally costly.

To eliminate this trade-off between robustness to data sparsity and learning time, in this study we propose a new kernel-based co-occurrence measure, which we call the pointwise Hilbert–Schmidt independence criterion (PHSIC) (see Table 1). Our contributions are as follows:

• •

  We formalize PHSIC, which is derived from HSIC Gretton et al. 2005, a kernel-based dependence measure, in the same way that PMI is derived from mutual information (Section 3).

• •

  We give an intuitive explanation why PHSIC is robust to data sparsity. PHSIC is a “smoothed variant of PMI”, which allows various similarity metrics to be plugged in as kernels (Section 4).

• •

  We propose fast estimators of PHSIC, which are reduced to a simple and fast matrix calculation regardless of whether we use linear or nonlinear kernels (Section 5).

• •

  We empirically confirmed the effectiveness of PHSIC, i.e., its robustness to data sparsity and learning time, in two different types of experiment, a dialogue response selection task and a data selection task for machine translation (Section 6).

Let $X$ and $Y$ denote random variables on $\mathcal{X}$ and $\mathcal{Y}$, respectively. In this paper, we deal with the tasks of taking a set of linguistic expression pairs

which is regarded as a set of i.i.d. samples drawn from a joint distribution ${\mathbf{P}_{\!{XY}}}$, and then measuring the “co-occurrence strength” for each given pair $(x,y)\in\mathcal{X}\times\mathcal{Y}$. Such tasks include collocation extraction and dialogue response selection (Section 1).

In this section, we give the formal definition of PHSIC, a new kernel-based co-occurrence measure. We show a summary of this section in Table 2. Intuitively, PHSIC is a “kernelized variant of PMI.”

This section gives an intuitive explanation for the first feature of PHSIC, i.e., the robustness to data sparsity, using Table 3. In short, we show that PHSIC is a “smoothed variant of PMI.”

First, the maximum likelihood estimator of PMI in Equation (1) can be rewritten as

where $\mathbb{I}[\text{condition}]=1$ if the condition is true and $\mathbb{I}[\text{condition}]=0$ otherwise. According to Equation (16), $\widehat{\mathrm{PMI}}(x,y)$ is the amount computed by repeating the following operation (see the first row in Table 3):

collate the given $(x,y)$ and the observed $(x_{i},y_{i})$ in $\mathcal{D}$ in order, and add the scores if $(x,y)$ and $(x_{i},y_{i})$ match exactly or deduct the scores if either the $x$ side or the $y$ side (but nor both) matches.

Moreover, an estimator of PHSIC in data space (Equation (15)) is

where $\widehat{\widetilde{k}}(\cdot,\cdot)$ and $\widehat{\widetilde{\ell}}(\cdot,\cdot)$ are similarity functions centered on the data⁴⁴ 4 To be exact, $\widehat{\widetilde{k}}(x,x^{\prime}):=k(x,x^{\prime})-\frac{1}{n}\!\sum_{j=1}^{n}k(x,x_{j})-\frac{1}{n}\!\sum_{i=1}^{n}k(x_{i},x^{\prime})+\frac{1}{n^{2}}\!\sum_{i=1}^{n}\!\sum_{j=1}^{n}k(x_{i},x_{j})$, which is an estimator of the centered kernel $\widetilde{k}(x,x^{\prime})$ in Equation (13). . According to Equation (17), $\widehat{\mathrm{PHSIC}}(x,y)$ is the amount computed by repeating the following operation (see the second row in Table 3):

collate the given $(x,y)$ and the observed $(x_{i},y_{i})$ in $\mathcal{D}$ in order, and add the scores if the similarities on the $x$ and $y$ sides are both higher (both $\widehat{\widetilde{k}}(x,x_{i})>0$ and $\widehat{\widetilde{\ell}}(y,y_{i})>0$ hold)⁵⁵ 5 In addition, the scores are added if the similarity on the $x$ side and that on the $y$ side are both lower, that is, if $\widehat{\widetilde{k}}(x,x_{i})<0$ and $\widehat{\widetilde{\ell}}(y,y_{i})<0$ hold. or deduct the scores if the similarities on either the $x$ or $y$ sides are similar but those on the other side are not similar.

As described above, when comparing the estimators of PMI and PHSIC from the viewpoint of “methods of matching the given $(x,y)$ and the observed $(x_{i},y_{i})$,” it is understood that PMI matches them in an exact manner, while PHSIC smooths the matching using kernels (similarity functions).

With this mechanism, even for completely unknown pairs, it is possible to estimate the co-occurrence strength by referring to observed pairs through the kernels. Therefore, PHSIC is expected to be robust to data sparsity and can be applied to phrases and sentences.

Recall that we have two types of empirical estimator of PMI, the maximum likelihood estimator (Equation (1)) and the RNN-based estimator (Equation (2)). In this section, we describe how to rapidly estimate PHSIC from data. When using the linear kernel or cosine similarity (e.g., cosine similarity between sentence embeddings), PHSIC can be efficiently estimated in feature space (Section 5.1). When using a nonlinear kernel such as the Gaussian kernel, PHSIC can also be estimated efficiently in data space via a simple matrix decomposition (Section 5.2).

In this section, we provide empirical evidence for the greater effectiveness of PHSIC than PMI, i.e., a very short learning time and robustness to data sparsity. Among the many potential applications of PHSIC, we choose two fundamental scenarios, (re-)ranking/classification and data selection.

• •

  In the ranking/classification scenario (measuring the co-occurrence strength of new data pairs with reference to observed pairs), PHSIC is applied as a criterion for the dialogue response selection task (Section 6.2).

• •

  In the data selection/filtering scenario (ordering the entire set of observed data pairs according to the co-occurrence strength), PHSIC is also applied as a criterion for data selection in the context of machine translation (Section 6.3).

The NLP community has commonly employed PMI to estimate the co-occurrence strength between linguistic expressions; however, existing PMI estimators have a high computational cost when applied to sparse linguistic expressions (Section 1). We proposed a new kernel-based co-occurrence measure, the pointwise Hilbert–Schmidt independent criterion (PHSIC). As well as defining PMI as the contribution to mutual information, PHSIC is defined as the contribution to HSIC; PHSIC is intuitively a “kernelized variant of PMI” (Section 3). PHSIC can be applied to sparse linguistic expressions owing to the mechanism of smoothing by kernels. Comparing the estimators of PMI and PHSIC, PHSIC can be interpreted as a smoothed variant of PMI, which allows various similarity metrics to be plugged in as kernels (Section 4). In addition, PHSIC can be estimated in linear time owing to the efficient matrix calculation, regardless of whether we use linear or nonlinear kernels (Section 5). We conducted a ranking task for dialogue systems and a data selection task for machine translation (Section 6). The experimental results show that (i) the learning of PHSIC was completed thousands of times faster than that of the RNN-based PMI while outperforming it in ranking accuracy (Section 6.2); and (ii) even when using a nonlinear kernel, PHSIC can be applied to a large dataset. Moreover, PHSIC reduces the amount of training data to one third without sacrificing the output translation quality (Section 6.3).

We are grateful to anonymous reviewers for their helpful comments. We also thank Weihua Hu for useful discussions, Kenshi Yamaguchi for collecting data, and Paul Reisert for proofreading. This work was supported in part by JSPS KAKENHI Grant Number JP15H01702 and JST CREST Grant Number JPMJCR1513, Japan.