Enhancing dance education through convolutional neural networks and blended learning

Data extraction

To ensure student facial data collection completeness and protect student privacy, we need to preprocess their facial data. An MVML algorithm based on the DWH model is proposed to meet the emotional expression needs in dance teaching. The algorithm attempts to embed multiple views into a single low-dimensional latent variable space by extracting different information from multimodal data, minimizing the distance between similar data pairs, maximizing the distance between dissimilar data pairs, and learning the most appropriate distance metric in the subsequent supervised learning process. This article constructs the model based on pairwise constraints and optimizes it based on the Hinge loss function. Then, the model is initialized using random sampling.

Given a set of data M = (x, z), a new function is obtained after mapping the data: (1)T = E p h | x , z ; Ψ h

where T represents the latent representation of embedding the data points M = (x, z) in a shared low-dimensional latent variable space, while h represents hidden unit nodes, which can be regarded as observation results obtained from different sources for a given dataset by dividing it into similar or dissimilar two labels, we hope that the results obtained through metric learning are that the distance between the collection of similar data pairs in the mapping space is close, while the collection of dissimilar data pairs is far, to achieve better classification and clustering and assist teachers in the evaluation of course standards. This article uses the Euclidean distance to measure the distance between data points in the new mapping space. Since all data sets have the same representation and are denoted as y i , y j in this article, we define S = y i , y j as the set of similar data pairs and D = y i , y j’ as the set of dissimilar data pairs, and the following optimization problem arises: (2)min Ψ ∑ y i , y j ∈ D t i − t j 2 s . t . ∀ y i , y j ∈ D , t i − t j 2 ≥ 1 .

In the equation, y represents all the data that appears in S and D; t i is the representation of y i in the low-dimensional latent variable space. This optimization problem aims to make the distance between similar data pairs in the mapping space as close as possible and keep the distance between dissimilar data pairs as far as possible, as shown in Fig. 1. The low-dimensional latent variable space range is defined as a constant of 1. In this way, all the parameters that need to be learned in the model are only Ψ, which is already embedded in t.

We provide excellent standard emotional expression data through continuous observation and increasing the amount of observed data. Based on this data, we use the unsupervised DWH model in the framework of the supervised algorithm MVDML to learn the parameter Ψ through maximum likelihood estimation, which promotes the comparison of similarity measures between different data point sets. Combining distance metric learning and maximum likelihood estimation, the optimization problem is rewritten in the following form: (3)min Ψ 1 Y L Y ; Ψ + λ 1 S ∑ y i , y j ∈ S t i − t j 2 s . t . ∀ y i , y j ∈ D , t i − t j 2 ≥ 1

L(Y; Ψ) is the negative log-likelihood value of data Y, where Y is determined based on the parameter Ψ, λ is the weight parameter, and ⋅ represents the cardinality of the set.

To solve the constraint problem presented in the optimization problem, a loss function can be used to eliminate the constraints. The Hinge loss can be used to eliminate the constraints of the optimization problem, and under the hinge loss, an unconstrained optimization problem can be obtained. Therefore, the subgradient algorithm can be used to optimize the problem. Under the Hinge loss, we can obtain a new optimization problem: (4)min Ψ 1 Y L Y ; Ψ + λ 1 1 S ∑ y i , y j ∈ S t i − t j 2 + λ 1 1 D ∑ y i , y j ∈ D max 0 , 1 − t i − t j 2

Here, λ₁ and λ₂are weighting parameters. The Hinge loss in Eq. (4) is 0 when the optimization problem satisfies the constraints. When the optimization problem in Eq. (3) does not satisfy the constraints, the Hinge loss is not 0.

Contrastive divergence

To make the solution of the optimization problem more appropriate, the Hinge loss should be minimized, thus forcing the constraints to be satisfied. Since the Hinge loss is non-differentiable, this article uses the subgradient algorithm for optimization. Therefore, the contrastive divergence (CD) algorithm is adopted to roughly estimate the gradient of Ψ, which corresponds to the negative log-likelihood value 1 Y L Y ; Ψ. It is very easy to derive the gradient (or subgradient) based on the distance metric loss caused by similar data pairs and dissimilar data pairs here, so the subgradient solution formula is: (5)∇ θ i = E p ϕ x i − E p ˆ ϕ x i ∇ η j = E p φ z j − E p ˆ φ z j ∇ λ k = E p χ h k − E p ˆ χ h k (6)∇ W i k = E p ϕ x i χ h k T − E p ˆ ϕ x i χ h k T + λ 1 2 S ∑ y m , y n ∈ S t k m − t k n ∂ t k m ∂ W i k − ∂ t k n ∂ W i k + λ 2 2 D ∑ y m , y n ∈ D t k m − t k n 2 < 1 t k n − t k m ∂ t k m ∂ W i k − ∂ t k n ∂ W i k (7)∇ U j k = E p φ z j χ h k T − E p ˆ φ z j χ h k T + λ 1 2 S ∑ y m , y n ∈ S t k m − t k n ∂ t k m ∂ U j k − ∂ t k n ∂ U j k + λ 2 2 D ∑ y m , y n ∈ D t k m − t k n 2 < 1 t k n − t k m ∂ t k m ∂ U j k − ∂ t k n ∂ U j k

where E_p[⋅] is the expectation with respect to the true distribution, E p ˆ ⋅ is the expectation with respect to the empirical distribution. However, the exact calculation of E_p[⋅] with respect to the true distribution is very difficult, so it is necessary to start with the expectation with respect to the empirical distribution and use Gibbs Sampling to approximate E p ˆ ⋅. The sampling process can be completed through iterative steps, as follows, where l is the number of iterations for the cycle. (8)E h k l = E p h k | x , z h k | E x l − 1 , E z l − 1 E x i l = E p x i | h x i | E h l − 1 E z j l = E p z i | h z j | E h l − 1 .

MVML algorithm

When introducing MVML algorithms in dance teaching, collecting information on students’ movements and facial expressions becomes particularly important. Figure 2 shows the devices and their arrangements that we used in the information capture process. The blue square outlined in the middle is the information capture area where students stand to dance. Precise recording and capture of students’ dance movements and facial expressions are possible through the seven yellow infrared cameras and four green RGB cameras mounted on the walls around the capture area, as shown in Fig. 2. The yellow infrared cameras are used to capture ground truth posture data, and they are equipped with a motion capture (MOCAP) system to capture data through motion analysis. The green red-green-blue (RGB) cameras are used to record multi-view data. When the devices are activated, the images are captured from one viewpoint, and the data is quickly captured. When connected to the same hub, the cameras will automatically synchronize. Through this method, we can collect data on changes in students’ movement and expression during their dance performances.

We preprocessed the motion capture data we collected to facilitate the subsequent processing by CNN. One method of preprocessing is to standardize the data. If the data is X, with a mean ofµanda standard deviation of σ, then the formula for standardizing the data can be represented as: (9)X s t d = X − μ σ .

In the Eq. 9, X_std represents the normalized data.

The collected data is input into a CNN, and after processing by the convolutional and pooling layers, image features are extracted. The input data is represented as X_std, where X_std(i) represents the data collected at the i-th frame. After the convolution operation, the resulting feature map data is represented as C_i as follows: (10)C i = f W ∗ X s t d i + b

where W is the convolution kernel, b is the bias, and f is the activation function. The convolution operation can extract local features of the image, preserving the correlation between adjacent pixels. The pooling operation further reduces dimensionality and improves computational efficiency. The pooling formula is: (11)M = g p o o l C i

where “pool” represents the pooling operation, and “g” is the activation function. After convolution and pooling operations, the feature map is input into the fully connected layer: (12)F = f W c ∗ C + b c .

In the equation, W_c represents the weights of the output layer and b_c represents the bias. Finally, classification or regression tasks are performed through the output layer: (13)Y = f W F ∗ F + b F

where W_F represents the weight of the output layer, and b_F represents the bias. Finally, the output Y represents the evaluation result of the action, which can be used for classification or regression tasks.

Evaluation process

We designed a quantifiable course evaluation system during the specific evaluation process of dance movements and emotional expressions. Based on the scores of each evaluation item, we classify students’ levels in this module into three categories: poor, good, and excellent, to promote more hierarchical and targeted dance teaching activities. In our course evaluation system, a scoring system is used, which measures the standardization of dance movements and the accuracy of emotional expressions.

In the experiment, professional dance performers provided the standard data for scoring. As dance movements play a dominant role in the entire performance, we allocate weights to the dance movements and emotional expression modules in a ratio of 6:4 separately during scoring. By dividing the whole dance into several sections, we compare and analyze the student’s movements and expressions with the professional dancer’s data in each section, which obtains the initial scores from the system. Then, we add up the scores of the student’s movement skills and emotional expression in each section and distribute them with a weighting ratio of 6:4. The final result is the total score. Assuming a dance can be divided into n sections, the score of movement skills in each section is D₁; D₂…D_n, and the score of emotional expression in each section is Q₁; Q₂…Q_n, then the total score can be described by the following formula: (14)S Score = 6 / 10 D 1 + D 2 + … + D n + 4 / 10 Q 1 + Q 2 + … + Q n .

Data collection and preprocessing

The preprocessing pipeline for analyzing facial expressions and motion capture data involved a series of meticulously executed steps to ensure accuracy and consistency in the data. For facial expression analysis, high-resolution camera feeds were initially captured from multiple angles to provide comprehensive coverage of the dancer’s face. Image preprocessing was performed to standardize the input data, which included resizing images to a consistent resolution and normalizing them to adjust for varying lighting conditions. Techniques such as brightness and contrast adjustment were applied to mitigate the impact of diverse lighting environments.

Face detection was accomplished using a pre-trained deep learning model. This model was fine-tuned with a diverse dataset to handle variations in facial expressions and orientations effectively. Following face detection, facial landmark detection was carried out using algorithms like Dlib or MediaPipe to identify key facial features including the eyes, nose, and mouth. The detected landmarks were aligned with a standardized facial model to correct for head tilts and rotations using affine transformations. Data augmentation techniques, including rotation, scaling, and shifting, were employed to enhance the model’s robustness and generalizability.

For expression recognition, a CNN was trained using a labeled dataset of facial expressions, encompassing emotions such as happiness, anger, sadness, and joy. The dataset was carefully prepared by filtering out mislabelled and inconsistent images, ensuring high-quality and representative samples. The CNN was trained incorporating dropout, batch normalization, and learning rate scheduling techniques to optimize performance. Validation and test sets were utilized to assess the model’s accuracy and to prevent overfitting.

In the motion capture pipeline, silhouette extraction was achieved through background subtraction techniques. Gaussian Mixture Model (GMM) were employed to isolate the dancer from the background, followed by segmentation using morphological operations to refine the silhouette and eliminate noise. Post-processing steps included contour smoothing and hole-filling to ensure an accurate representation of the dancer’s outline.

Pose estimation was performed using algorithms like OpenPose to detect key body joints and poses, such as shoulders, elbows, knees, and ankles. Calibration was necessary to account for variations in camera perspectives and dancer movements, ensuring consistent keypoint mapping across frames and cameras. Table 1 outliers and erroneous detections were filtered out using statistical methods and consistency checks to maintain accuracy in pose estimation.

Movement analysis involved extracting movement trajectories by tracking key joint positions over time, with techniques such as Kalman filters or particle filters applied to smooth the data and handle noise. Angles and movement parameters were calculated to evaluate the correctness of dance movements, involving computations of angles between body segments and analysis of movement patterns. Data normalization was performed to address variations in dance styles and individual performances, standardizing trajectories and angles for consistent analysis.

This article uses the CPU of Xeon(R) E5-2640 v4, the GPU of 4*Nvidia Tesla V100 and the Ubuntu system to complete the environment setup and model training. The deep learning framework is Tensorflow. We set the learning rate of the model to 0.004, the training epoch to 40 rounds, the weight decay term to 0.001, and adopted MAE as the optimizer.

Data source

This article emphasizes the importance of emotional expression in dance teaching. Dance performance is a visual art based on emotions, and even the most excellent dance works will lose their impact without proper emotional expression. The beauty of dance comes from human beings themselves, and our emotions fluctuate due to external influences. Dancers can effectively communicate their intrinsic values by synthesizing emotional expression with choreographed movements. Humans exhibit a spectrum of affective states: happiness, anger, surprise, fear, disgust, sadness, and neutrality. Each emotion corresponds to distinct facial expressions, which can be accurately identified through advanced video analysis techniques.

We used DanceDataset (https://zenodo.org/records/5118689, DOI: 10.5281/zenodo.5118689). Cultural sensitivity is crucial in the study of emotional expression in dance. Emotions and their expression can be culturally specific, so we focus on these cultural differences to avoid misinterpreting or exploiting traditional dance practices. Cultural context should be respected, recognizing that conventional dance forms may embody unique emotional expressions inherent in their cultural heritage. It is important to approach these practices with sensitivity and respect, ensuring that cultural traditions are not distorted or inappropriately appropriated. Addressing potential bias in sentiment analysis models is another important ethical issue. Acknowledge bias in the data and remove it by ensuring that the dance dataset is diverse and representative of various demographic and cultural groups. This approach prevents the continuation of bias in the analysis and application of sentiment data. Furthermore, fairness in the application ensures that all dancers, regardless of their background, have an equal opportunity to participate in and benefit from the analysis.

Performance of expression recognition

Using the DWH model, the confusion matrix was constructed to display the recognition accuracy of different facial expressions. As shown in Fig. 3, the recognition accuracy of facial expressions is generally above 80%, and it can further improve with continuous training. The highest recognition accuracy is achieved for the happy and surprised expressions, with an accuracy rate of 92% and a probability of being incorrectly detected of only 8% for other expressions. Happiness and surprise are always displayed with an uncontrollable uplift of the corners of the mouth and widened eye pupils, whether in real life or during a dance performance. The muscles in our faces reflect our inner emotions. Therefore, recognizing happy and surprised expressions is relatively easier and more accurate. Disgust expression appears less frequently in dance performances because dance is an extreme expression of love for life. Therefore, the recognition accuracy for disgust expression is 81%. The recognition accuracy for angry, fearful, and neutral expressions is 85%, 87%, and 88%, respectively, and none of them have reached 90%, indicating a need for further training. Sadness expression is a common emotion, and its recognition accuracy is 90%. With continuous training, the recognition accuracy of facial expressions will continue to improve. By recognizing facial expressions, teachers can make scientific judgments, which not only helps improve student performance but also reduces the pressure on the teacher and increases work efficiency.

Model comparison

This article employs the F-score as the evaluation criterion for the model. Table 2 compares the model’s performance after data preprocessing and normalization with several other models.

In comparison to the graph convolutional neural network (GCNN), regularized generative adversarial network (RGAN), and stochastic Wasserstein histogram matching (SWHIM), the Deep Weighted Hybrid Multi-View Multi-Label (DWH-MVML) algorithm exhibits a significant enhancement in performance. Specifically, DWH-MVML enhances precision, recall, and F-score by 9.46%, 8.77%, 9.13% over GCNN, and 8.03%, 8.53%, and 8.29% over RGAN. When compared to SWHIM, DWH-MVML achieves increases of 4.61%, 3.10%, and 3.85% in precision, recall, and F-score, respectively. These improvements underscore DWH-MVML’s superior handling of complex prediction tasks, significantly boosting model accuracy and stability and highlighting its leading position in the current research landscape.

Learning effect evaluation

To explore the changes in student performance after introducing MVML for course evaluation, we selected twenty students as our subjects. We compared their dance learning results before and after introducing this method to evaluate its effectiveness. As shown in Fig. 4, we observed that before the introduction of MVML, the expression of student emotions followed a normal distribution trend according to the teacher’s grading standards, ranging from the 60s to the 90s with a large span. Moreover, many students were hovering around the passing mark, not because of the teacher’s insufficient level but because emotional expression is subtle and requires total attention throughout the dance performance. Given limited time and the context of mixed education with more students, many students could only receive guidance and evaluation online, and the teacher could only give a rough assessment based on their experience, often overlooking many details and causing slow progress for the students. However, through the MVML algorithm, objective evaluations can be conducted from multiple dimensions, and with the help of big data, objective evaluations can be given by identifying student emotions and providing improvement suggestions.

After the introduction of MVML, we found that the evaluation system became more objective and comprehensive, and teachers could give more reasonable suggestions for different student issues. As a result, students made significant progress in emotional expression after the introduction, and even those who had barely passed previously were now performing well in this area. Even outstanding students made progress under scientific guidance. This is evident in the graph, which illustrates a substantial improvement. This demonstrates that our method is of great help to students, providing a more objective evaluation system. Guided by empirical data, students can continuously refine their skills by addressing their weaknesses, resulting in a marked enhancement in their ability to express emotions through dance.

As the most essential part of dance, the standardization of dance movements directly affects the level and visual experience of the entire dance performance. Dancers with standard and accurate dance postures and movements often possess more skilled dance expression techniques, providing the audience with a better visual experience and sensory enjoyment. In practical dance instruction, ensuring that students’ movements are standardized is a primary focus for teachers. Traditionally, teachers teach dance movements to students by demonstrating them, and students learn gradually by observing and imitating the teacher’s movements. Following the teacher’s demonstration, corrective guidance on students’ movements and postures is provided through a one-to-many teaching approach. However, due to limited course time and human resources, it is not practical for teachers to provide detailed guidance on the dance movements of each student. Furthermore, when multiple decomposed movements are integrated and performed rapidly, the immediacy and coherence of the dance characteristics make it difficult to discern the detailed nuances of the individual movements. Due to the limitations of human eyesight and brain reaction speed, teachers’ guidance to students has also been neglected in many areas. Given the challenges above, the implementation of an MVML evaluation system offers an effective solution for addressing these issues.

Here, we selected 20 samples from the dance training class as research samples. We captured and analyzed their dance movements using an MVML system and then generated feedback reports based on the evaluation results, guiding them to improve their movements, thereby promoting the standardization of their dance movements. In Fig. 5, we compared the standardization scores of these 20 students before and after the introduction of the MVML system to measure the system’s effectiveness. Before the introduction of the MVML system (as shown by the blue dashed line in the figure), the standardization scores of these 20 students varied greatly, ranging from the 30s to the 60s, showing a great degree of difference and span. This was mainly to ensure the accuracy and credibility of the experimental results, and we followed the principle of randomness in sample selection. As the dance foundations of the students in the dance training class were not the same, it was only necessary to conduct personal longitudinal comparisons after the introduction of the MVML system to detect the system’s effectiveness in improving learning outcomes.

The scores of each student after the introduction of the system are shown by the red dots in the figure. By comparing them with the corresponding blue squares, we found that the red dots were always above the blue squares, which means that the standardization of the students’ dance movements had improved after introducing this evaluation system. In particular, students with previously weak foundational skills demonstrated substantial improvement after undergoing accurate movement recognition and assessment using the multi-view metric system. Through continuous training of body coordination based on feedback from the learning system and enhancement of muscle memory, these students achieved notable advancements in the standardization of their dance movements. On the other hand, students who had previously had a good foundation in dance showed even better standardization after being guided by the MVML system, with some students even approaching a professional level. These experimental results indicate that the MVML system significantly promotes the standardization of students’ dance movements. Leveraging the power of emerging technologies greatly liberates limited human resources. It compensates for the deficiency of human eyes in recognizing subtle movements, paving the way for future dance teaching and performance.