Modeling Topical Coherence in Discourse without Supervision

Based on the motivation laid above, a prima-facie formulation of coherence of a paragraph (denoted as $CS$) can be proposed assuming that $CS$ varies linearly and positively with: (a) to what degree of certainty topics expressed by the paragraph (denoted as $T$) are supported by its constituent sentences (denoted by variable $E(T)$), and (b) to what extent these topics are related to each other (denoted by variable $\lambda(T)$). Mathematically,

For empirical purposes, $\kappa$, the constant term can be set to $1$. The two components of the above formula require topics expressed by the paragraph to be extracted first. This process is explained below.

The first step before extracting the topics (which are abstract “concepts”) is to define a set of wide variety of generic topics which could be mapped to any given paragraph. This one-time process is carried out by performing word-clustering on a large scale mixed domain unlabeled data. The assumption here is that clusters extracted from large scale data represent generic topics in the universal space of English language. After defining generic topics, the next step is to infer topics discussed in a given input paragraph. We explain each of these steps in detail below.

For word clustering, we consider embeddings learned on a large scale corpora Pennington et al. 2014 and cluster them via k-means clustering. The assumption here is that words connected to a particular topic would exhibit strong syntagmatic and paradigmatic relations. Since word embeddings are good at capturing such relations, topically connected words may eventually lie in the same cluster in the vector space. Each cluster in this space hence represents a topic. Topic models such as the ones based on Latent Dirichlet Allocation Blei et al. 2003 are other alternatives for topic extraction. However, it is well known that the behavior of topic models changes drastically¹¹ 1 basic LDA based experiments did not show good performance in our setting based on the prior distribution, hyper-parameters and document processing applied Chang et al. 2009. Hence, we rather keep the topic extraction process simplistic by using word embeddings.

Each sentence in the input paragraph is POS tagged and only nouns are selected for topic extraction. The intuition behind choosing only nouns is that nouns are most representative of the topic in a given sentence. For each noun, we determine the topic-cluster to which it belongs. To avoid noise, for each sentence, we choose a dominant cluster by assigning cluster scores to all clusters identified within a given sentence (the reason we try to extract topics for each sentence separately is that sentences are the atomic units that are capable of discussing independent topics from others). The cluster topic score of a cluster is (a) directly proportional to the fraction of nouns within the sentence that belong to the cluster and the mean pairwise cosine similarity of nouns in the sentence, and (b) is inversely proportional to the diameter of the cluster. The diameter of the cluster is the maximum distance between any two points in the cluster. The dominant cluster for a sentence is the one which has the highest cluster topic score. After determining the dominant cluster, we take all points within the cluster and check for its existence in WordNet. This eliminates highly specific terms, jargon, named entities which could potentially add noise to the clusters. Therefore, each topic can now be represented as a subgraph of the WordNet where the nodes in the subgraph are bag-of-words representing a topic (topicBOW).

Given the topics relevant to the paragraph, we now explain the two components $E(T)$ and $\lambda(T)$ of Equation 1 in the following subsections.

For an input paragraph $P$, with $M$ sentences ($s_{1},s_{2},...,s_{M}$), expressing a set of topics ($T=t_{1},t_{2},...,t_{N}$), the component $E(T)$ can be expressed as:

where $p(t_{i}|P)=p(t_{i}|s_{1},s_{2},...,s_{M})$ represents the probability of the topic $t_{i}$ conditioned over the paragraph. $E(T)$ corresponds to the conditional entropy of topics here. Intuitively, if sentences in the paragraph are well distributed across all the topics emerging out of the paragraph, i.e. each sentence in the paragraph is somewhat related (even though loosely) to all the topics, the paragraph may exhibit more coherence, with less topic-shift. On the other hand, if the paragraph can be divided in such a way that each segment of the paragraph is related to a unique topic, the paragraph will be less coherent. This is adequately captured by the conditional entropy formula which rewards when the distribution $p(T|P)$ is smooth and penalizes when it is sparse. Moreover, this formulation of $E(T)$ captures another essential aspect: if the number of topics in the paragraph become very large, it inherently pulls the conditional probabilities $p(t_{i}|P)$ down, making the distributions sparse²² 2 it is highly improbable to have sentences discussing a large number of topics and thus, reducing the entropy. So, when number of topics grow, $E(T)$ is penalized more.

The term $p(t_{i}|P)$ in Equation 2 can be expanded further, with the help of Bayes’ rule as follows:

Furthermore, term $p(P|t_{i})$ can be expanded using chain rule, as:

Here $C_{k-1}$ can be considered as the context that appears before the the $k_{th}$ sentence.

Since, no topic in the paragraph can be given more importance over another, the probability term $p(t_{i})$ can be uniformly distributed across $T$ i.e., $p(t_{i})=\frac{1}{N}$. For estimating $p(s_{k}|C_{k-1},t_{i})$ (without applying any simplifying assumptions), we follow the distributed bag-of-words (BOW) model by Le and Mikolov 2014. The idea here is to train distributed bag-of-words model using a large number of sentences (covering good number of topics); and later on use the model for inferring $p(s_{k}|C_{k-1},t_{i})$. The inference time snapshot of the model is given in Figure 1 and the probability estimation algorithm is given in Algorithm 1.

As discussed earlier in Section 2.3, since topics are also treated as bag of words (topicBOW), it is easy to do conditional inference using a bag-of-words based distributed representation model. In such a setting, both sentences and topics are treated as bag-of-words, and hence, there is no special signal to be passed to the model with respect to the topic bag-of-word representations. We agree that bag-of-words based techniques can be agnostic to within-sentence sequentiality and natural order of sentences. However, since modeling topical coherence involves inferring sentence-topic associations, and does not necessarily aim to model within-sentence properties; not preserving sentence order would not adversely affect our goal. We now describe the second component of Equation 1.

The component $\lambda(T)$ aims to capture relatedness between topics expressed by the paragraph. For two paragraphs with same number of topics, the coherence score should be more if topics are strongly related with each other as compared to the case where the correlation between topics of the paragraph is less. This helps us to refrain from penalizing the coherence scores even though the paragraph contains large number of topics if such topics are strongly related. For example, a topical shift from the topic carnivore to mammals in general should be penalized less than that from carnivore to electronics. The inter-relatedness between topics is captured by $\lambda(T)$, for which we rely on lexical knowledge networks such as WordNet Fellbaum 1998, that preserve various forms of conceptual-semantic and lexical relations between words and are well-curated. Note that, we do not opt for simplistic topic-relatedness measures such as inter-cluster distance between topic-clusters, as such measures are significantly affected by the bias in the data used for clustering, and noise introduced by imperfect algorithm and parameter selection.

Each word in the topic bag-of-words (for the whole paragraph) is mapped to a node in the WordNet (if its lemma exists in WordNet), according to its most frequent sense. The mapped nodes in the WordNet are then connected with each other through other intermediate nodes to form a subgraph. Figure 2 illustrates a subgraph extracted for Example 2 discussed in Section 1. From the WordNet subgraph, we compute $\lambda(T)$ as follows:

• •

  NodeSim(T): This corresponds to the average similarity between nodes representing the topics in the subgraph. This is obtained by calculating the average cosine similarity between the corresponding node embedding, obtained via the TransE multi-relational embedding learning technique Bordes et al. 2013. TransE operation on a graph results in low-dimensional embeddings of the nodes that capture its relationship with the other nodes in the graph in a distributional manner. In our setting, a higher similarity between two node embeddings indicate higher inter-relationship between them. We compute cosine similarity between the embeddings of each pair of nodes in the sub-graph and then average the similarity scores.

• •

  ND(T): This represents the average neighborhood degree of the nodes in the subgraph. Intuitively, a higher average neighborhood degree indicates higher degree of connectedness amongst nodes, indicating higher topic relatedness.

• •

  ED(T): This denotes the edge-density of the graph³³ 3 https://en.wikipedia.org/wiki/Dense_graph. The notion behind using this measure is that if topics are distantly placed in the WordNet graph (a case of higher incoherence), the subgraph generated will be denser with more number of nodes and even more number of edges established through various WordNet relationships. So, higher $ED(T)$ should penalize $\lambda(T)$ and the overall coherence score.

• •

  TC(T): We define this term as:

  $$TC(T)=\frac{\#edges~in~the~subgraph}{\#edges~in~its~transitive~closure}$$

  Since transitive closure of a graph indicates the node reachability, the term $TC$ reduces the reachability of nodes in the subgraph. For a graph like Wordnet where there are limited relations between nodes, subgraphs with non-ambiguous reachable paths are indicative of stronger topic-relatedness. Hence, a higher $TC$ score should penalize $\lambda(T)$ and the overall coherence score.

We provide an end-to-end algorithm for coherence score calculation in Algorithm 2.

We carry out our experiments on two sets of data as described below:

To cluster the GloVe word vectors we experimented with different values of K for K-means clustering algorithm. Finally, we chose K=1000 based on the values of average inter-cluster and intra-cluster distances. The conditional probability inference model discussed in Section 2.4 above was trained on a mixed domain corpus, i.e., UMBC⁵⁵ 5 http://ebiquity.umbc.edu/blogger/2013/05/01/umbc-webbase-corpus-of-3b-english-words/ WebBase corpus of 3 billion English words. The pre-computed transE embeddings trained on the Wordnet graph were obtained from Bordes et al. 2013. Our coherence scores were in the real-valued range [0.1-1000].

We obtained coherence scores $CS$ using Algorithm 2 for each of the four sets of human graded essays (referred to as $Set1$, $Set2$, $Set7$ and $Set8$ henceforth); and the synthetic data generated (referred to as $Synthetic$). We obtained Spearman’s rank correlation coefficient between the gold labels and calculated coherence scores to see how they are correlated. Since, for most practical purposes, the relative ranking of paragraphs based on coherence may be more important than computing the absolute coherence scores, we chose the Spearman’s rank correlation metric instead of Pearson’s correlation. We also computed the rMSE between the scaled coherence scores and gold labels for each of the five sets of data.

The results for the Spearman’s rank correlation and rMSE between the coherence scores and gold labels are reported in Table 3. As it can be seen, in all cases our coherence scores obtain a positive correlation with the corresponding gold labels, suggesting that we are indeed modeling coherence, which plays an essential role in human essay grading. The low values of rMSE indicates that the predicted coherence scores are quite acceptable. To see the importance of each component in our coherence score, we calculated the component-wise Spearman’s rank correlation coefficients w.r.t. the gold labels for each of the five datasets (Table 4). It can be seen that entropy and average neighborhood degree are positively correlated and TC score and edge density are negatively correlated as expected. The correlation with $NodeSim(T)$ is mostly positive, though for essay sets 2 and 7 we get slightly negative correlations. This might be due to the specific nature of prompts of these essay sets. Set 1 and Set 2 are persuasive essay prompts. Hence, the responses may contain complex relations between words which might not be captured by a WordNet graph which models very few specific kinds of relations. The comparatively high correlation of entropy component shows that it is an essential part of our coherence score formulation.

We ran our pipeline for Example 1 given in Table 1 (Coherent) and the example shown in Figure 3 (Incoherent). We got coherence score values of $17.29$ and $0.964$ respectively, for the two cases. In each case, out of all the topics obtained for the paragraph, we obtained the topic membership of each sentence. Interestingly for the second case, as shown in Figure 3, out of the two topics obtained for the paragraph, the first, third and fourth sentences belong to one topic (coloured blue), and the second sentence belongs to the second topic (coloured red), indicating a significant drift in topic and hence incoherence.

We conducted human evaluation on a random group of 27 essays (9 sets) from our synthetic dataset. The task was to rank each paragraph within a set (1 original and 3 perturbed variants) based on the coherence. We compared these rankings done by human subjects with the ranking produced by our system based on the values of coherence scores. We found an agreement of 79.3% between the two rankings, which suggests that the rankings produced by our system are acceptable.

Intrigued to see how our system performs as compared to supervised techniques for measuring local coherence, we tested our system on the Earthquakes and Accidents datasets released by Barzilay and Lapata 2008. We obtained coherence scores for the essays given in the test set of the two datsets. Then the accuracy was measured by considering the fraction of test pairs ranked correctly based on the values of our coherence scores. If the relative difference between the coherence scores was less than a fixed tolerance, we marked them as positive. We got accuarcies of 77.3% and 71.5% on Earthquakes and Accidents datasets, respectively. These values are competitive to the reported accuracy figures of 81.4% and 86.0% (row Coreference-Syntax-Salience- in Table 5 of Barzilay and Lapata 2008) on the two datasets. Considering the fact that our system is unsupervised and does not need additional complex meta-information like dependency parses and coreferences, syntax and saliency information etc., which the best supervised systems use today; our system offers significant advantages and is more generalizable compared to the popular supervised techniques.