End-to-End Argument Mining for Discussion Threads
Based on Parallel Constrained Pointer Architecture

First, we introduce the input representation. Given $N$ threads $(T_{1},\dots,T_{N})$, we denote $T_{i}$’s posts which are sorted in thread depth order, and then timestamp order as described in Section 4.1 as $(P^{(i)}_{1},\dots,P^{(i)}_{N_{i}})$, where $N_{i}$ represents the number of posts in $T_{i}$. In addition to the thread and post representations, write $(S^{(i,j)}_{1},\dots,S^{(i,j)}_{N_{i,j}})$ for sentences in post $P^{(i)}_{j}$, where $N_{i,j}$ represents the number of sentences in $P^{(i)}_{j}$. Note that separator representations are not considered in the notation.

Then, $\bm{w}_{n}$ is given initially, an embedding vector of $n$th word in a sentence $S^{(i,j)}_{k}$, a sentence representation for an input of LSTM is represented as: $A_{k}=\sum_{n}\bm{w}_{n}$, where $\bm{w}_{n}$ is gained from bag-of-words (BoW) or word embeddings Mikolov et al. 2013; Pennington et al. 2014; Stab and Gurevych 2017. In our study, we employed BoW and a fully connected layer with a trainable parameter to learn word embeddings. Subsequently, we provide Bidirectional LSTM (BiLSTM) Graves and Schmidhuber 2005 because PN requires encoding steps. At each time step of the encoder BiLSTM, PCPA considers a representation of an ACC. Thus, the hidden representation $e_{i}$ of BiLSTM becomes the concatenation of forward and backward hidden representations. To simplify the explanation, we denote the hidden representations of $(S^{(i,j)}_{1},\dots,S^{(i,j)}_{N_{i,j}})$ as $(e^{(i,j)}_{1},\dots,e^{(i,j)}_{N_{i,j}})$. For better understanding, we show notations in Figure 3.

The general PN of Potash et al. 2017 uses all hidden states $e_{i}$. Alternately, PCPA can limit the states to improve the accuracies, since each premise has a single outgoing link to another sentence in its post. Hence, we provide an approach to discriminate IPR using only inner-post hidden states of the BiLSTM.

Figure 3 shows the example IPR discrimination in thread $T_{i}$; for example, we assume that the inner-post relation of the sentence written as "7" in the 3rd ACC of post $P^{(i)}_{2}$ is classified. The general PN needs to consider all $e_{i}$, therefore, the search space is large. On the other hands, our proposed PCPA can consider $(e^{(i,2)}_{1},e^{(i,2)}_{2},e^{(i,2)}_{3})$, which needs to use the hidden states of its post only. Therefore, our constrained architecture can reduce the search space significantly.

In general, given $W_{1}$, $W_{2}$, and $v_{1}$, parameters of attention model Luong et al. 2015 for PN,

As the definition of target and callout in our scheme, IPI exists between a parent post and child post that replies to the parent. Thus, PCPA can discriminate IPI with no need to use all of the hidden representations of the LSTM. In other words, it can discriminate IPI without searching outside of the two posts.

Hence, we design an output layer that requires only a set of reply pairs in thread $T_{i}$. Specifically, we assume that $R^{(i)}=\{(j_{1},j_{2}),\cdots\}$ where $j_{1}\neq j_{2}\land j_{1}<j_{2}$ for a set of parent-child pairs in thread $T_{i}$. Supposing $j_{1}$ is the index of a parent post and $j_{2}$ represents the index of the child post that replies to the $j_{1}$. Note that when thread $T_{i}$ does not have any reply pairs, $R^{(i)}=\varnothing$. Considering the above, a technique that is similar to the IPR’s technique is introduced.

Figure 4 shows the example IPI discrimination in thread $T_{i}$; supposing that we are going to discriminate a target that is called out from ACC "5" in the figure. In this case, the search space is limited by the parent post $(e^{(i,1)}_{1},\dots,e^{(i,1)}_{4})$. Moreover, we add an element $e^{(i,2)}_{1}$ so that a callout can point itself if there’s no target in its parent post. The left four outputs in the "Interaction Pointer Distribution" indicate a discrete probabilistic distribution that the callout ACC "5" links to target sentences in its parent post, and an output on the far right represents a probability that the callout links to itself.

The equation (1) uses a query in the PN, so we in turn concentrate on using a query vector for the callout in IPI. Herein, we introduce an additional PN for IPI using new attention parameters, $W_{3}$,$W_{4}$ and $v_{2}$, as:

At each time step of the BiLSTM, the type classification task predicts whether it is claim, premise, or NonArg. The ACC type of sentence $S^{(i,j)}_{k}$ can be classified by taking $\mathop{\rm softmax}\limits$ of $z^{(i,j)}_{k}=W_{type}e^{(i,j)}_{k}+b_{type}$, where $W_{type}$ and $b_{type}$ are parameters. An objective for the type classifier can also be described by taking the sum of log-likelihoods for all posts as:

Combining objectives of IPR (equation (3)), IPI (equation (5)) and the ACC type (equation (4.2)), the training objective of PCPA is shown as follows:

For the evaluation of ACC types, IPR and IPI discrimination, we adopt precision, recall and F1 scores. To obtain the precision and recall, we introduce a way to compute positive and negative cases by creating relations Stab and Gurevych 2017, excluding self-pointers. ⁸⁸ 8 For example, supposing there is a post which contains three sentences, $(S_{1},S_{2},S_{3})$, and two gold standard IPRs, $(S_{1}\leftarrow S_{2})$ and $(S_{1}\leftarrow S_{3})$. This is exactly the case that positive cases of IPR are $\{(S_{1}\leftarrow S_{2}),(S_{1}\leftarrow S_{3})\}$, and negative cases are all sentence pairs excluding self-pointers. That is, negatives are $\{(S_{2}\leftarrow S_{1}),(S_{2}\leftarrow S_{3}),(S_{3}\leftarrow S_{1}),(S_{3}\leftarrow S_{2})\}$. In this case, self-pointer cases are $\{(S_{1}\leftarrow S_{1}),(S_{2}\leftarrow S_{2}),(S_{3}\leftarrow S_{3})\}$.⁹⁹ 9 For IPI, we are also able to create sentence pairs. For instance, suppose there is a parent post which contains three sentences $(S_{1},S_{2},S_{3})$, a child post that contains two sentences $(S_{4},S_{5})$, and a gold standard IPI, $(S_{2}\leftarrow S_{5})$. The positive case of IPI is exactly $\{(S_{2}\leftarrow S_{5})\}$, and negative cases are all sentence pairs excluding self-pointers, that is, $\{(S_{1}\leftarrow S_{4}),(S_{1}\leftarrow S_{5}),(S_{2}\leftarrow S_{4}),(S_{3}\leftarrow S_{4}),(S_{3}\leftarrow S_{5})\}$.

First, we employ state-of-the-art PN techniques from Potash et al. 2017 as baselines. The use of these baselines was decided because our model PCPA (Our Model) employs pointer architectures. As the authors proposed two techniques, sequence-to-sequence model (PN with Seq2Seq) and w/o sequence-to-sequence model (PN without Seq2Seq), we have the two models for comparison.

To analyze how a non PN model works, multi-task learning is employed to the baseline Søgaard and Goldberg 2016 (STagBLSTM) by Eger et al. 2017. STagBLSTM is composed of shared BiLSTM layers for subtasks, and output layers for each subtask. In Eger et al. 2017, the authors provided a BIO tagging task, however, the task is not required in our work because BiLSTM handles an input as sentence representation rather than as word representation. In this paper, we use one BiLSTM.¹⁰¹⁰ 10 Though there are some variation models other than the single BiLSTM model, our preliminary experiments show a non-significant improvement.

To show end-to-end learning models are effective for AM on thread structures, we provide the following three task specific baselines. First, feature-based SVM Stab and Gurevych 2017 (SVM - T) is introduced. $T$ indicates each subtask of the claim classifier, premise classifier, IPR classifier, and IPI classifier. In addition, random forest (RF - T) and the logistic regression technique Peldszus and Stede 2015 (Simple - T) are also introduced. For each task specific model, BoW features the top 500 most frequent words ¹¹¹¹ 11 In fact Stab and Gurevych 2017 and employs rich features such as structural features. We only use BoW for comparison because the properties of COLLAGREE corpus substantially differ from their corpus..

We assume that each output of PN with Seq2Seq, PN without Seq2Seq or STagBLSTM does not satisfy the constraints as a self-pointer. This is because inappropriate outputs with constraint violations of IPR and IPI by these approaches will happen, i.e., they can predict IPI out of parent and child posts. The assumption maintains the false positive (FP) of baselines, since a self-pointer which results from a chance is not counted as FP. This condition gives the baselines the advantage of precision over our models. Therefore, this assumption is convincing.

The following describes our implementation details. The implementation of neural models are by Chainer Tokui et al. 2015. The hyperparameters are the same as Potash et al. 2017 for the PN baselines and our models¹²¹² 12 Hidden input dimension size 512, hidden layer size 256 for the BiLSTMs, hidden layer size 512 for the LSTM decoder of PN without Seq2Seq, and high dropout rate of 0.9 Srivastava et al. 2014; Zarrella and Marsh 2016. All models are trained with the Adam optimizer Kingma and Ba 2014 with a mini batch size of 16.. In the interest of time, we ran 50 epochs, and used the trained model for testing. The COLLAGREE dataset is divided into training threads and testing threads at $8:2$. In addition, we use the following hyperparameters in equation (4.2): $\alpha=\beta=1/3$. However, total loss of $L^{ipr}$ and $L^{ipi}$ tends to enlarge since they have to calculate a sum of the sentence pairs. Hence, we provide a model with tuned hyperparameters $\alpha=\beta=0.15$ (Our Model - Hyp) for comparison.

To demonstrate the effectiveness of the separator representations, we conducted an experiment. In Table 2 (Bottom), the models without the separator input representations are indicated as "w/o separator". It shows that separator representations dramatically improve scores of PN based models. This remarkable result is from the ability to learn the structural information of a thread by encoding separators in the BiLSTM.

To analyze the stability of our models, we compare standard deviations among three selected models. Table 3 shows standard deviations for the three models. These results indicate that our model has lower standard deviations for IPR than baseline PN models. The reason for this is the size of search space: our models can effectively limit the search space based on thread structures.

Next, we show how our models improve their performance by employing our parallel pointer architecture. Herein, we provide a new model of PCPA with a single PN (Our Model with Param Share), which shares $v_{1}$, $W_{1}$ and $W_{2}$ in equation (1) and $v_{2}$, $W_{3}$ and $W_{4}$ in equation (4), respectively. Table 4 demonstrates the mean of F1 scores for our model and Our Model with Param Share. Note that the average performances are lower than the best performances in Table 2. The scores indicate that sharing the two pointer architecture parameters is not effective in our proposed model. We estimate this is because poor association Caruana 1997 between the IPR and IPI identification tasks exists. Therefore, our approach of using two parallel pointer architectures is effective.

We examine how our models are specialized in thread structures. Specifically, we limit the threads in test datasets by specific thresholds, and then analyze performance transitions. We conduct two experiments as the thread depth is limited (Figure 5 and 5). While the baselines performances decrease as the thread depth increases, our model keeps its F1 score because of the separators and the search space. The separator representations for an input increase according to the thread depth, and the baseline PN models need to use wider range of hidden states in comparison with the PCPA model. In other words, our models are extremely effective, even for deeper threads.

We also limit the threads that we can use in test data by the number of posts (Figure 6, and 6). For discriminating IPR, our model increasingly outperforms others in accordance with the number of posts. Figure 6 indicates that the difference between our model and baselines is minimal. This is because the number of posts does not affect the thread depth, necessarily. Most of COLLAGREE’s threads have a depth of at most 2. In other words, Figure 6 also implies the depth of threads affects the improvement of IPI identifications.