Segmentation-free Compositional $n$-gram Embedding

Most existing word embedding models Mikolov et al. 2013; Pennington et al. 2014; Bojanowski et al. 2017 take a sequence of words as their input. Therefore, the conventional models are dependent on word segmentation Yang et al. 2017; Shao et al. 2018, which is a process of converting a raw corpus (i.e., a sequence of characters) into a sequence of segmented character $n$-grams. After the segmentation, the segmented character $n$-grams are assumed to be words, and each word’s representation is constructed from distribution of neighbour words that co-occur together across the estimated word boundaries. However, in practice, this kind of approach has several problems. First, word segmentation is difficult especially when texts in a corpus are noisy or unsegmented Saito et al. 2014; Kim et al. 2018. For example, word segmentation on social network service (SNS) corpora, such as Twitter, is a challenging task since it tends to include many misspellings, informal words, neologisms, and even emoticons. This problem becomes more severe in unsegmented languages, such as Chinese and Japanese, whose word boundaries are not explicitly indicated. Second, word segmentation has ambiguities Luo et al. 2002; Li et al. 2003. For example, a compound word

線形代数学(linear algebra) can be seen as a single word or sequence of words, such as

線形$|$代数学(linear $|$ algebra).

Word segmentation errors negatively influence subsequent processes Xu et al. 2004. For example, we may lose some words in training corpora, leading to a larger Out-Of-Vocabulary (OOV) rate Sun et al. 2005. Moreover, segmentation errors, such as segmenting

きのう(yesterday) as

き$|$のう(tree $|$ brain), produce false co-occurrence information. This problem is crucial for most existing word embedding methods as they are based on distributional hypothesis Harris 1954, which can be summarized as: “a word is characterized by the company it keeps” Firth 1957.

To enhance word segmentation, some recent works Junyi 2013; Sato 2015; Jeon 2016 made rich resources publicly available. However, maintaining them up-to-date is difficult and it is infeasible for them to cover all types of words. To avoid the negative impacts of word segmentation errors, Oshikiri 2017 proposed a word embedding method called segmentation-free word embedding (sembei). The key idea of sembei is to directly embed frequent character $n$-grams from a raw corpus without conducting word segmentation. However, most of the frequent $n$-grams are non-words Kim et al. 2018, and hence sembei still suffers from the OOV problems. The fundamental problem also lies in its extension Kim et al. 2018, although it uses external resources to reduce the number of OOV. To handle OOV problems, Bojanowski et al. 2017 proposed a novel compositional word embedding method with subword modeling, called subword-information skip-gram (sisg). The key idea of sisg is to extend the notion of vocabulary to include subwords, namely, substrings of words, for enriching the representations of words by the embeddings of its subwords. In sisg, the embeddings of OOV (or unseen) words are computed from the embedings of their subwords. However, sisg requires word segmentation as a prepossessing step, and the way of collecting co-occurrence information is dependent on the results of explicit word segmentation.

For solving the issues of word segmentation and OOV, we propose a simple but effective unsupervised representation learning method for words, phrases and sentences, called segmentation-free compositional $n$-gram embedding (scne). The key idea of scne is to train embeddings of character $n$-grams to compose representations of all character $n$-grams in a raw corpus, and it enables treating all words, phrases and sentences seamlessly (see Figure 1 for an illustrative explanation). Our experimental results on a range of datasets suggest that scne can compute high-quality representations for words and sentences although it does not consider any word boundaries and is not dependent on any human annotated resources.

Our method scne successfully combines a subword model Zhang et al. 2015; Wieting et al. 2016; Bojanowski et al. 2017; Zhao et al. 2018 with an idea of character $n$-gram embedding Oshikiri 2017; Kim et al. 2018. In scne, the vector representation of a target character $n$-gram is defined as follows. Let $x_{1}x_{2}\cdots x_{N}$ be a raw unsegmented corpus of $N$ characters. For a range $i,i+1,\ldots,j$ specified by index $t=(i,j)$, $1\leq i\leq j\leq N$, we denote the substring $x_{i}x_{i+1}\cdots x_{j}$ as $\bm{x}_{(i,j)}$ or $\bm{x}_{t}$. In a training phase, scne first counts frequency of character $n$-grams in the raw corpus to construct $n$-gram set $V$ by collecting $M$-most frequent $n$-grams with $n\leq n_{\rm{max}}$, where $M$ and $n_{\rm{max}}$ are hyperparameters. For any target character $n$-gram $\bm{x}_{(i,j)}=x_{i}x_{i+1}\cdots x_{j}$ in the corpus, scne constructs its representation $v_{\bm{x}_{(i,j)}}\in\mathbb{R}^{d}$ by summing the embeddings of its sub-$n$-grams as follows:

where $S(\bm{x}_{(i,j)})=\{\bm{x}_{(i^{\prime},j^{\prime})}\in V\mid i\leq i^{\prime}\leq j^{\prime}\leq j\}$ consists of all sub-$n$-grams of target $\bm{x}_{(i,j)}$, and the embeddings of sub-$n$-grams $z_{s}\in\mathbb{R}^{d}$, $s\in V$ are model parameters to be learned. The objective of scne is similar to that of Mikolov et al. 2013,

where $\sigma(x)=\frac{1}{1+\exp(-x)}$, $\mathcal{D}=\{(i,j)\mid 1\leq i\leq j\leq N,j-i+1\leq n_{\rm{target}}\}$, and $\mathcal{C}((i,j))=\{(i^{\prime},j^{\prime})\mid\bm{x}_{(i^{\prime},j^{\prime})}\in V,j^{\prime}=i-1\text{ or }i^{\prime}=j+1\}$. $\mathcal{D}$ is the set of indexes of all possible target $n$-grams in the raw corpus with $n\leq n_{\rm{target}}$, where $n_{\rm{target}}$ is a hyperparameter. $\mathcal{C}(t)$ is the set of indexes of contexts of the target $\bm{x}_{t}$, that is, all character $n$-grams in $V$ that are adjacent to the target (see Figures 1 and 2). The negative sampling distribution $P_{\rm{neg}}$ of $\tilde{s}\in V$ is proportional to its frequency in the corpus. The model parameters $z_{s},u_{\tilde{s}}\in\mathbb{R}^{d}$, $s,\tilde{s}\in V$, are learned by maximizing the objective. We set $n_{\rm{target}}=n_{\rm{max}}$ in our experiments.

Although we examine frequent $n$-grams for simplicity, incorporating supervised word boundary information or byte pair encoding into the construction of compositional $n$-gram set would be an interesting future work Kim et al. 2018; Sennrich et al. 2016; Heinzerling and Strube 2018.

In this section, we perform two intrinsic and two extrinsic tasks at both word and sentence level, focusing on unsegmented languages. The implementation of our method is available on GitHub¹¹ 1 www.github.com/kdrl/SCNE.

We proposed a simple yet effective unsupervised method to acquire general-purpose vector representations of words, phrases and sentences seamlessly, which is especially useful for languages whose word boundaries are not obvious, i.e., unsegmented languages. Although our method does not rely on any manually annotated resources or word segmenter, our extensive experiments show that our method outperforms the conventional approaches that depend on such resources.

We would like to thank anonymous reviewers for their helpful advice. This work was partially supported by JSPS KAKENHI grant 16H02789 to HS and 18J15053 to KF.