Pedagogical sentiment analysis based on the BERT-CNN-BiGRU-attention model in the context of intercultural communication barriers

BERT-based word embedding layer

To train word vectors, BERT employs the Masked Language Model (MLM) and Next Sentence Prediction (NSP). MLM randomly designates a portion of linguistic elements as masked lexical elements for prediction during training, allowing the model to acquire knowledge of these masked lexical elements using global contextual information. While MLM effectively encodes bidirectional contextual information for representing lexical elements, it doesn’t explicitly convey logical relationships between text pairs. On the other hand, NSP can be perceived as a sentence-level binary classification problem, addressing the logical relationships between sentences by determining the plausibility of the latter sentence as a continuation of the preceding one. To address the problem, the BERT model integrates both MLM and NSP approaches, achieving an effective representation of word vectors and successfully extracting semantic features from the text. This amalgamation furnishes the model with abundant contextual information and inter-textual logical relations. The structure of the BERT model, as depicted in Fig. 2, comprises multiple transformer layers, each encompassing both self-attention and cross-attention mechanisms. This architecture empowers BERT to capture long-distance dependencies in the text, enhancing its comprehension of textual content.

Figure 2, e₁, e₂, …, e_n is the input sequence of the BERT model, Trm is the Encoder model of Transformer; and x₁, x₂, …, x_n is the output word vector sequence of the BERT model. The input sequence of the BERT is the sum of the lexical meta-entry, fragment embedding, and positional embedding information, in which the lexical meta-embedded text sequence needs to use lexeme <cls>as the start tag.

Two-channel model based on CNN and BiGRU

This approach of combining CNN and the attention mechanism leverages the advantages of CNN in feature extraction while preserving the temporal sequence of text through the attention mechanism, thus enhancing the model’s performance. Additionally, integrating the feature matrices extracted by CNN with the outputs of BiGRU (Bidirectional Gated Recurrent Unit) can further capture global and local information in the text, improving the model’s ability to comprehend deep-level textual features. The model’s structure is illustrated in Fig. 3, demonstrating how CNN, the attention layer, and BiGRU work together to process the input text and complete the language analysis task.

In Fig. 3, x₁, x₂, …, x_n is the word embedding vector of the text, and feature extraction is carried out by three convolution kernels with different window sizes, each with dimensionality dim = 768, number of channels M = 256, and the step size is set to 1. The local feature c_i is obtained by the i ^th convolution of the convolution kernel with a window size of k: (1)c i = f W x i , i + k − 1 + b

where f is the nonlinear activation function ReLu, W is a parameter in the convolution kernel matrix, x_[i,k+i−1] denotes the vector between row i and (i+k-1) of the word vector matrix, and b is a bias term. A convolution kernel with a window size k undergoes convolution (n-k+1) times, yielding the local feature vector. (2)c i k = c 1 , c 2 , … , c n − k + 1

where c i k the convolutional outputs of each convolutional kernel, resulting in the local feature matrix R: (3)R = r 1 , r 2 , … , r m

The GRU has evolved from LSTM, addressing to some extent the challenges of distance dependence and gradient vanishing encountered in RNN. GRU not only effectively handles the issue of distance dependence but also simplifies the model structure. The specific model configuration is depicted in Fig. 4.

GRU incorporates only two sigmoid functions, resulting in two gates: the update gate and the reset gate. Specifically, the update gate dictates how much state information from the previous time step should be transferred. A higher value of the update gate signifies a greater need to transfer state information. The calculation formula for the update gate is expressed as follows: (4)z i = σ W s ⋅ h t − 1 , x t + b z

The reset gate plays a pivotal role in deciding how much information from the previous time step should be forgotten. A smaller reset gate implies a greater need to discard information. The expression for the reset gate is analogous to that of the update gate, differing only in the parameters of the linear transformation. Let h_t−1 be the state information at moment t-1, and x_t denote the input vector at moment t. Put them through a linear transformation first, and then use the sigmoid activation function to get the output activation value. The new memory content can be stored using the reset gate to store past-related information, which is calculated below.

(5)r i = σ W r ⋅ h t − 1 , x t + b r (6)h ˜ t = tanh W h ⋅ r t ∗ h t − 1 , x t + b h

The formula for the final memory of the current moment is as follows: (7)h t = 1 − z t ∗ h t − 1 + z t ∗ h ˜ t

Given that the gated loop cell efficiently conveys information from the preceding cell to the subsequent one and is computationally more streamlined due to its simpler structure compared to the LSTM model.

The BiGRU is a bidirectional extension rooted in the fundamental gated recurrent unit structure. In this configuration, each input word vector undergoes processing in both a forward-facing GRU unit and a backward-facing GRU unit. The outputs of these two GRU units are then combined computationally. The bidirectional model captures both forward and backward information, endowing it with greater potency than its unidirectional counterpart. It yields two output values for the hidden layer—one for the forward output and the other for the backward output. The calculation is presented below:

(8)h → t = G R U f o r w o r d h t , h t + 1 (9)h ← t = G R U b a c k w o r d h t , h t − 1

where h → t denotes the text features before position t computed by the GRU, h ← t denotes the text features after position t computed by the GRU, and h t = h → t , h ← t denotes the features of the context of the word vectors at position t as the outputs of the bi-directionally gated recurrent unit layer. Each word vector corresponds to a bi-directionally gated recurrent unit, contributing to the collective representation.

Attention layer

The attention mechanism prioritizes information with significant importance while disregarding irrelevant details. The attention mechanism adeptly filters out irrelevant text information by assigning distinct weight values to text features exhibiting varying degrees of emotional tendency. This selective focus allows for more efficient learning of crucial textual emotional features, thereby enhancing the performance of implicit sentiment classification. Typically, textual information incorporates words with emotional tendencies, and these words frequently exert a pivotal influence on the overall emotional tone of the text. Consequently, this article introduces the attention mechanism to spotlight text features associated with emotional tendencies in implicit sentiment sentences. In this article, the attention mechanism is used to enhance the text sentiment polarity for the local feature matrix R = [r₁, r₂, …, r_m] and the global feature matrix H = [h₁, h₂, …, h_m], respectively, and the computational formula deduced after the study is:

(10)u r i = tanh W r r i + b r (11)u h i = tanh W h r j + b h

where W_r and W_h are the weight parameter matrices, b_r and b_h are the bias terms, tanh is the nonlinear activation function. By normalising the weight vectors u_ri and u_rh , the attention scores a_i and a_j on the local feature r_i and the global feature h_i are obtained as follows:

(12)a i = exp u r i ∑ i = 1 m exp u r i (13)a j = exp u r j ∑ j = 1 m exp u r j

To calculate the weighted sum of the attention scores a_i and the corresponding sub-vectors of the local feature matrix R = [r₁, r₂, …, r_m] , the local feature vector of the text optimised by the attention mechanism can be obtained s_i , and the weights of the global feature matrix H are similarly optimized to obtain the global feature vector of the text s_h. The specific calculation formula is as follows:

(14)s i = ∑ i = 1 m a i r i (15)s h = ∑ j = 1 m a j r j

The dual-channel attention mechanism layer allocates suitable attention weights to pivotal emotion word vectors within the text, thereby enhancing the performance of the BCBA model. This improvement, in turn, elevates the accuracy of the emotion recognition and analysis module within the human–computer dialogue teaching model. Furthermore, the output layer of the BCBA model formulated in this article is structured as a fully connected neural network with a softmax function. Firstly, the local feature vector s_i optimized by the attention mechanism is fused with the global feature vector s_h to obtain the final feature representation of the text s: (16)s = s r , s h

The dropout method is introduced before the fully connected neural network layer to mitigate model overfitting. The output of the fully connected layer is obtained through the softmax function for classification calculation:

(17)o = f W s + b (18)g = s o f t max o

where o is the output vector of the fully connected layer, W is the weight matrix, b is the bias term, and g is the final output vector of the model.

Parameter settings

The main parameters of the experimental environment in this article include a CPU with Intel(R) Core(TM) i7-7700HQ CPU @ 2.80 GHz, 8 GB of CPU Memory (RAM), and an Operating System of Windows 10 (64-bit operating system based on x64 processor). The experimental platform is Python 3.6. Before model training, it is crucial to preprocess the raw data appropriately. This article primarily focuses on cleaning text and removing noise from the text, such as HTML tags, special characters, URLs, etc. Given that the deep neural network architecture designed in this article combines the BERT model, CNN, BiGRU, and attention mechanism, the hidden layer consists of 180 neurons, with Leaky-ReLU serving as the activation function. The output layer comprises 60 neurons, employing Leaky-ReLU as the activation function. Typically, a softmax layer follows the output layer for multi-class classification tasks. We introduce several hyperparameters, as shown in Table 1, to optimize the model’s performance. Among them, the word embedding dimension is set to 300, the maximum length of a sentence is 120, the batch size for text processing is 32, the number of hidden layer neurons in BiLSTM is 384, the number of neuron layers is 2, the filter sizes for the convolutional neural network are (2, 3, 4), and the number of filters is 256. To mitigate overfitting during the experiment, we adopted the Dropout method. The Dropout method randomly “turns off” a portion of the neurons in the network during training, thus preventing overfitting.

Evaluation indicators

In the task of sentiment analysis, the key metrics for evaluating the classification model’s performance are TP (true positive), TN (true negative), FP (false positive), and FN (false negative). TP represents the number of samples correctly identified as having positive sentiments among all true positive sentiment samples. TN denotes the number of samples correctly identified as having negative sentiments among all true negative sentiment samples. FP signifies the number of samples incorrectly classified as positive sentiments among all true negative emotion samples. Finally, FN indicates the number of samples incorrectly categorized as negative emotions among all true positive emotion samples.

These metrics offer a comprehensive assessment of the model’s performance in sentiment analysis tasks, allowing for optimization and improvement tailored to specific requirements. As the sentiment analysis in this article focuses on binary classification into positive and negative sentiments, the evaluation metrics employed are accuracy and F1 value. These metrics are calculated as follows:

(19)A c c = T P + T N T P + F P + F N + T N (20)Pr e = T P T P + F P (21)Re c = T P T P + F N (22)F 1 = 2 ∗ Pr e ∗ Re c Pr e + Re c

The accuracy rate signifies the model’s ability to classify correctly; the precision rate denotes the proportion of positive samples correctly classified among all samples classified as positive, and the recall rate indicates the proportion of positive samples correctly classified among all actual positive samples. Higher precision and recall rates are typically desired in evaluation, but these two metrics often exhibit a trade-off in emotion classification tasks. The F1 value is employed to reconcile this, providing a balanced consideration of precision and recall. Hence, this article adopts the accuracy rate and F1 value as the primary evaluation indices for the sentiment analysis task in the human–computer dialogue teaching model.

In addition, we adopt the confusion matrix as an evaluation criterion to observe the model’s performance in various categories. Also known as an error matrix, it is a standard format for accuracy evaluation, represented as an n-row and n-column matrix. The confusion matrix is a visualization tool in artificial intelligence, especially for supervised learning. Specifically, each column of the confusion matrix represents the predicted category, and the total number in each column indicates the number of data points predicted for that category. Each row represents the true category of the data, and the total number of data points in each row indicates the number of data instances belonging to that category.

Model comparison under different datasets

Table 2 and Fig. 5 present each model’s performance metrics on the STS dataset. As depicted in Tab. 2 and Fig. 5, the model proposed in this article achieves an accuracy of 0.859 and an F1 value of 0.808 in classifying negative emotions, surpassing the performance of other models in classifying both positive and negative emotions. Notably, the performance of the TextCNN and BiLSTM models exhibits a similar trend. BiLSTM and BiLSTM+Attention demonstrate comparable performance due to their shared model structure. However, introducing the attention mechanism in BiLSTM+Attention improves accuracies of 0.721 and 0.742 for positive and negative emotion classification, with corresponding F1 values of 0.808 and 0.622, respectively. Despite this, there remains a significant gap with the model constructed in this article.

The proposed model integrates the strengths of BERT, CNN, BiGRU, and the attention mechanism, enabling a more comprehensive utilization of semantic information from both sides of the word than traditional word vector generation techniques. This facilitates a deeper understanding of implicit sentiment sentences’ lexical, syntactic, and semantic features. Consequently, the proposed model proves effective in textual sentiment analysis, achieving precision rates of 0.725 and 0.839 and recall rates of 0.695 and 0.788 for positive and negative sentiment classification on the STS dataset, respectively.

Simultaneously, experimental analyses are conducted on the Amazon dataset to validate the model’s efficacy and simplicity further; the results are illustrated in Table 3. To show the performance comparison more directly, we draw the bar chart in Fig. 6 according to the data in Table 3. The performance of each model exhibits a similar pattern, with TextCNN demonstrating comparable performance to BiLSTM due to the prevalence of short text data in the Amazon dataset. The multi-scale convolution in TextCNN allows it to effectively capture feature information between words in short text sequences, aligning its performance with BiLSTM.

A comparison between BiLSTM and BiLSTM+Attention reveals that incorporating the attention mechanism boosts the accuracy of the BiLSTM+Attention model by approximately 2.7% and the F1 value by about 1.8%. This indicates that the attention mechanism effectively enhances the model’s accuracy. During this analysis, the proposed model in this article achieved an average accuracy, precision, recall, and F1 value of 0.837, 0.782, 0.742, and 0.753, respectively, for sentiment classification on the Amazon dataset, affirming the effectiveness of the proposed model.

The two sets of experiments revealed subpar performance in classifying positive emotions. To comprehend the reasons behind the inadequate recognition of positive emotion categories by each classification model, an investigation was conducted into the confusion matrices of the predicted results. The confusion matrices for the four comparison models are presented in Fig. 7.

Figure 7 shows that all models tend to incorrectly predict statements with the actual emotion category of positive as negative emotion sentences. The probability of misclassification for negative emotions is significantly higher than for positive ones. Additionally, there are variations in the number of mispredicted emotion sentences among different models. Still, they share a commonality—the proportion of mispredicted emotion sentences with a positive emotion category classified as negative is roughly consistent. Further exploration into implicit texts with positive sentiment categories reveals that the positive sentiment features in these texts are not very prominent. This lack of distinct positive sentiment features contributes to the suboptimal performance of all classification models in this category.

Ablation experiments

Ablation experiments were conducted to assess the performance of each module in the proposed model; the results are shown in Fig. 8 and Table 4. The model denoted as BERT+CNN+BiGRU+AT was compared with models created by removing either the BERT model or the pre-attention mechanism, resulting in CNN+BiGRU+AT and BERT+CNN+BiGRU models, respectively. The change curves of the recall rate under different iteration numbers for these three models were plotted.

When considering the presence or absence of the attention mechanism, it becomes apparent that the sentiment classification model with an attention mechanism outperforms the model without one. The experimental effectiveness of BERT+CNN+BiGRU+AT and CNN+BiGRU+AT can reach recall rates of 0.701 and 0.682, respectively. However, the recall of the CNN+BiGRU+AT model fluctuates downward with the number of iterations. Whether the BERT pre-training model is included or not, it has been observed that BERT+CNN+BiGRU+AT and BERT+CNN+BiGRU achieve the best recall at the fastest iteration number. Additionally, the recall curve becomes smoother due to the utilization of more data for model training. The recall of the BERT+CNN+BiGRU model stabilizes at 0.667, while the recall of the BERT+CNN+BiGRU+AT model remains stable at the same value.

In summary, the BERT+CNN+BiGRU+AT model constructed in this article exhibits minimal fluctuations with the increase in iteration times attributed to implementing the Dropout method, which mitigates the occurrence of overfitting.