Hypernyms Through Intra-Article Organization in Wikipedia

Abstract

We introduce a new measure for unsupervised hypernym detection and directionality. The motivation is to keep the measure computationally light and portatable across languages. We show that the relative physical location of words in explanatory articles captures the directionality property. Further, the phrases in section titles of articles about the word, capture the semantic similarity needed for hypernym detection task. We experimentally show that the combination of features coming from these two simple measures suffices to produce results comparable with the best unsupervised measures in terms of the average precision.

1 Introduction

Given two words $w_{1}$ and $w_{2}$, the hypernym detection task is to determine if there is a hypernym relation between the two words. If a hypernym is known to exist, the directionality task is to determine if $w_{1}$ is a hypernym or hyponym of $w_{2}$. More precisely, due to polysemy, the detection task asks if, there is some meaning of $w_{1}$, in which it is a hypernym or hyponym of some meaning of $w_{2}$.

The first approaches were pattern based Hearst 1992; Snow et al. 2004. However, these suffered from poor recall. This led to the development of methods based on the distributional hypothesis Harris 1954 or the Distributional Inclusion Hypotheses Geffet and Dagan 2005. The method used in these techniques was to take a very large corpus, and using either window based, or dependency path based approaches, along with measures like frequency, PPMI Church and Hanks 1990, LPMI Evert 2005, to find vectors to represent the words. In supervised settings, the vectors for two words are combined suitably and a classifier is trained Baroni et al. 2012; Roller et al. 2014; Weeds et al. 2014; Shwartz et al. 2016 to predict the existence of a hypernym relation and later directionality. However, recently there has been deeper research on what exactly is learned by these techniques Levy et al. 2015. In the unsupervised setting a suitable measure, motivated by either the distributional inclusion hypothesis or the distributional informativeness hypothesis is used for hypernyms Santus et al. 2016; Weeds et al. 2004; Santus et al. 2014; Geffet and Dagan 2005.

In this paper we present a simple and computationally light unsupervised technique for hypernym detection and directionality which is a combination of two measures, called as depth measure and heading measure. We start off with a large corpus, but instead of finding window or dependency path based contexts, which are very expensive to compute; we argue that the internal organization of descriptive and explanatory documents naturally leads to strong signals that are indicative of hypernyms. By exploratory documents, we mean documents that have been produced with the express purpose of making the reader understand the concepts that the document is describing; text books, research papers and Wikipedia articles are prime examples of this. We verify this intuition empirically, wherein we achieve results comparable and in some cases better than prior techniques in both the tasks of hypernym detection and directionality. One salient feature of our measures is that they exploit how humans organize information in explanatory documents making them portable across all languages. This offers us an advantage over prior techniques which depend on the intricacies of syntax and semantics of the language of the documents.

2 Methodology

We will be using Wikipedia as the source of descriptive and explanatory documents in this paper. For ease of exposition, we define the concept of units. Given an article from Wikipedia, each page title, section title, sub-section title etc., irrespective of the depth of section, will be referred to as a heading. Each heading on the article usually consists of a title that describes what the text following the heading is about. Following the heading, are usually a few paragraphs that describe in more detail the heading. This may be followed by another heading, and this pattern repeats. We refer to each heading and the text following it, till (but not including) the next heading, as a unit. Thus the article is physically organized as a sequence of disjoint units. We represent a unit $u$ as a pair $(h,S)$, where $h$ is the heading, and $S$ is the sequence of sentences in the unit.

2.1 Depth Measure

Given a hypernym-hyponym pair, $(w_{1},w_{2})$, consider the organization of a Wikipedia article containing both of them. The very first unit at the top of the page is usually a broader introduction of the main topic of the article. It will use words that are more popular. However, from the next unit onward, the articles tend to be more specialized with higher detail in content. Thus the linguistic contexts used in the units occurring lower in the article tend to be more detail oriented than those occurring earlier (except for the first introductory unit). Since more detailed context are indicative of a hyponym, they tend to occur later in the article than the hypernyms. This same reasoning applies within a unit. In this case the hypernym will tend to occur in earlier sentences in the unit than the hyponym. We generalize this to the case in which $w_{1}$ and $w_{2}$ do not co-occur in the same article, as follows. We take a large corpus of articles (e.g. all articles in Wikipedia), and check the depth at which $w_{1}$ and $w_{2}$ tend to occur (individually). If $w_{2}$ tends to occur at larger depth than $w_{1}$, we conclude that $w_{2}$ is a hyponym of $w_{1}$.

Let $\mathcal{P}$ be the set of articles. Let $a\in\mathcal{P}$ be an article, and let $w$ be a given word or phrase. To formally define depth, we will assume that the article has a fixed rooted tree like topology with the units of $a$ as its vertices, denoted by $G(a)$. The root will be the first unit of the article, and the depth will be the distance from the root. We experiment when $G(a)$ is a Star-like tree topology, as indicated by the depths of its sections and sub-sections, or a Linear-like topology with a unit being a parent of the immediate next unit in the physical layout of the article. We define a function $\lambda(a,w)$ that captures the depth of each occurrence of $w$ in $a$. Let $\mathcal{I}(a,w)$ denote the set of occurrences of $w$ in $a$. Each occurrence consists of a pair $(u_{i},s_{j})$, where $u_{i}$ denotes a unit, and $s_{j}$ is the sentence in which it occurs. Multiple instances of $w$ in the same sentence is treated as one instance. Let $d(G(a))$ denote the total depth of $G(a)$. If $d(u_{i})$ is the depth of unit $u_{i}$ in $G(a)$, and $|u_{i}|$ is the number of sentences in it, then we define the

$$\lambda(a,w)=\sum\limits_{(u_{i},s_{j})\in\mathcal{I}(a,w)}\left(1-\frac{d(u_{i})}{d(G(a)}\right)\left(1-\frac{j}{|u_{i}|}\right)$$

The first factor gives a normalized measure (to ensure same scale across all articles, of different sizes) of the depth of each occurrence of $w$ in $a$. Similarly, the second factor gives a normalized depth of the instance within a unit. Larger the $\lambda(a,w)$, more likely is it to be a hypernym. To aggregate this measure across all articles:

$$\lambda(w)=\underset{a\in\mathcal{P}}{\mathrm{median}}\hbox{~}\lambda(a,w)$$ (1)

2.2 Heading Measure

For testing relatedness between words, we define the heading measure, inspired by  Do and Roth 2012. We search in Wikipedia for the article on the given phrase $w$ (e.g., if $w$ is the word jumping, then we get the article https://en.wikipedia.org/wiki/Jumping). Since the page is about $w$, it is organized into sections that explain every property of $w$. We can thus represent $w$ simply by the collection of headings (titles, sub-titles at every possible level).

If the page on $w$ turns out to be a disambiguation page, then the page lists different possible meanings of $w$, along with the corresponding links. We follow each of the links to get possible articles on different meanings of $w$. In case any of the pages is again a disambiguation page, we iterate further. For each of the pages that are articles, we form a set of headings. Each set corresponds to a different meaning of $w$. We let $S_{w}$ denote the collection of the headings for different meanings of $w$ (See Algorithm 1). Note here, that $S_{w}$ is a set of sets. One advantage of this method is that we get the different meanings of the words up front, whereas, in context feature based approaches, there can be a mixing of the different contexts for polysemous words.

input : Word or phrase $w$

output : $S_{w}$, a collection of headings of pages on $w$

Function ExtractHeadings(w)

$\mathcal{P}=\{P_{1},\ldots,P_{k}\}$ be the set of articles on $w$, $S_{w}\leftarrow\phi$

    while $\mathcal{P}\not=\phi$ do

    Select any $P\in\mathcal{P}$

       $\mathcal{P}\leftarrow\mathcal{P}\setminus P$

       if $P$ is not a disambiguation page then

       Let $C$ be the collection of headings on page $P$

          $S_{w}\leftarrow S_{w}\cup\{C\}$

          else

       Let $\mathcal{D}$ be the collection of articles that $P$ points to as possible meanings of $w$

          $\mathcal{P}\leftarrow\mathcal{P}\cup\mathcal{D}$ end if

       end while

   return $S_{w}$

   

Algorithm 1 Extract Heading Sets

After computing $S_{w_{1}}$ and $S_{w_{2}}$ as shown in Algorithm (1), we compute the $SimScore(w_{1},w_{2})$ as the maximum similarity between an element of $S_{w_{1}}$ and $S_{w_{2}}$. For the similarity, we experimented with two measures, the Jaccard Similarity, and the cosine of the corresponding word2vec Mikolov et al. 2013 vectors. For using word2vec, for each heading set $C$, we take the mean of the vectors for each heading. Since word2vec uses the context of words, this combines our features with the contextual features. The final measure we use for the pair of words is:

$$\left(\frac{1+\lambda(w_{1})-\lambda(w_{2})}{2}\right)\left(SimScore(w_{1},w_{2})\right)$$ (2)

3 Experiments and Results

3.1 Datasets and Corpus

We experimented with four datasets widely used in literature: BLESS Baroni and Lenci 2011, EVALution Santus et al. 2015, Lenci/Benotto Benotto 2015, and Weeds Weeds et al. 2014 taken from the repository provided by Shwartz et al. 2017. The corpus of articles we use is a complete xml dump of the English Wikipedia dated 3 Nov 2017.

3.2 Testing Directionality

We extracted out the pairs marked hypernyms from each of the four data sets and computed the depth measure for each word in the pair. If the difference $\lambda(w_{1})$ - $\lambda(w_{2})$ is less than zero, we mark these pairs as False and compute precision. To identify the articles containing $w$, we indexed the corpus of Wikipedia using Elasticsearch Gormley and Tong 2015 and used the top thousand articles returned as the set for computing $\lambda(w)$. We experimented with the Star and Linear topologies.

Dataset		Star Topology	Linear Topology
	Total	Precision	Precision
BLESS	1198	0.918	0.536
Weeds	1321	0.974	0.429
Evalution	3303	0.980	0.566
LenciBenotto	1728	0.974	0.439

Table 1: Testing directionality. Total= total number of pairs present in the test set.

It is seen that with Star topology our precision in very high on each of the datasets. The worst performance is on BLESS. However, here too, it is $91.8\%$. For BLESS, as seen in  Santus et al. 2014, the performance of SLQS is $87\%$. Similar to SLQS, our depth measure is motivated by distributional informativeness hypothesis Shwartz et al. 2017. However, without using the extensive computation of context vectors and entropy, we are able to demonstrate good performance. As can be seen by physically examining Wikipedia articles, many of them tend to have a Star topology. This is also indicative that the topology used plays a major role in this feature. More sophisticated techniques will be needed to identify the topology of individual articles.

3.3 Testing Detection

For this experiment, we aim to discriminate pairs of words connected by the hypernym relation, from words connected by other relations (meronym, coord, attribute, event, antonym, synonym). For each pair, we evaluate our scoring function given in expression (2). We compared our numbers with those given in  Shwartz et al. 2017. In that paper, multiple measures are used, and the best performing measure for every row of the table is presented. We conducted the experiments for both, Star as well as the Linear topology. However, the results for Star topology were slightly better, hence we present these in Table (2).

Dataset	Hyper vs Rel	AP word2vec	AP Jaccard	Best AP	Best Measure
BLESS	all other relations	0.084	0.065	0.051	$invCL$
	meronym	0.446	0.355	0.760	$SLQS_{sub}$
	coord	0.203	0.235	0.537	$SLQS_{sub}$
	attribute	0.581	0.509	0.740	$SLQS_{sub}$
	event	0.453	0.315	0.779	$APSyn$
Weeds	all other relations	0.514	0.506	0.441	$clarkeDE$
	coord	0.514	0.506	0.441	$clarkeDE$
EVALution	all other relations	0.273	0.290	0.353	$invCL$
	meronym	0.629	0.678	0.675	$APSyn$
	attribute	0.556	0.614	0.651	$APSyn$
	antonym	0.520	0.526	0.550	$SLQS-row$
	synonym	0.606	0.593	0.657	$SLQS-row$
Lenci/Benotto	all other relations	0.401	0.389	0.382	$APSyn$
	antonym	0.548	0.530	0.624	$APSyn$
	synonym	0.599	0.593	0.725	$SLQS-row_{sub}$

Table 2: $AP=$ average precision. The Best AP and Best Measure is taken from  Shwartz et al. 2017.

For the case of hypernym vs all other relations, except for EVALution, in all other data sets, our average precision ($AP$) using both Jaccard and word2vec (Shwartz et al. 2017 call this as $AP$@all) is better than the best unsupervised measure as reported in Shwartz et al. 2017. For comparing hypernyms against individual relations, we find that with Jaccard similarity, it performs better than the best measures on meronyms in EVALution, and coordinates in Weeds. However, it performs worse for both the relations in BLESS. Our systems performs worse than the best measure whenever an Informativeness Measure  Shwartz et al. 2017, like $SLQS$ and its variants perform well. It performs better, or at least competitive, when the best performing measure is an Inclusion Measure or Similarity Measure (except for hypernym-vs-event in BLESS). A possible explanation of this is that the heading features that we use do not capture how informative a phrase is. However, having common headings is an indication of shared features, implying similarity, which is also indicated by inclusion measures. However, it should be noted, that we are comparing our single system against the best performing one in each case. For finding the best measure, Shwartz et al. 2017 finds the best by varying the measures as well as the features, whereas we have a fixed system. Our system took a day to set up (including coding effort), and a few mins to run. This is in contrast to methods mentioned above that rely on the computation of context vectors; calculation of dependency parse tree based features alone, from ukWack and Wackypedia corpus, took several days on the same machine.

3.4 Error Analysis

One of the sources of error in our technique is a semantic drift due to disambiguation pages. For example, for the pair $(alligator,wild)$, which is marked as attribute in BLESS, our system follows disambiguation links to wildlife, and then marks it as a hypernym. We find this pattern repeatedly. e.g. $(scale,lizard)$ is a meronym in BLESS, but is classified as hypernym in the hypernym vs. meronym experiments. While $(scale,snake)$ is a meronym in BLESS, and is marked correctly as not a hypernym. One reason for this is that among the disambiguation pages, a word is often generalized to related terms. For the hypernym v/s all experiment, the proportion of false positives when at least one word needed a disambiguation page was $37\%$ for BLESS, $29\%$ for Weeds, and about $31\%$ for EVALution and Lenci/Benotto. Selective link following during the disambiguation step can potentially solve this problem.

4 Conclusion

We showed that the organization of articles is an important feature for the task of both hypernym detection and directionality. Using just this simple and computationally cheap measure suffices to give performance that is comparable to the state of art unsupervised measures in these tasks. The proposed measure can also be trivially extended to any languages with a Wikipedia. We believe future work in this area will benefit by using this feature in complex systems that can improve performance.