1 Introduction

In recent years, the surge in social media usage has transformed the landscape of digital communication, fundamentally altering how individuals express themselves and connect with others (Statista 2023). With the widespread availability of smartphones and Internet access, social media platforms have become easily accessible to a global audience, providing a seamless channel for individuals to share their thoughts and ideas. This democratization of expression, while empowering people to voice their opinions and engage in meaningful conversations, has also brought to the forefront a pressing issue: the widespread proliferation of Hate Speech (HS) (Watanabe et al. 2018), which poses a critical threat to online communities and society, in general.

There are no universally accepted and precise definitions of HS (Poletto et al. 2021), but, according to the United Nations (2019), HS is defined as any form of communication that targets and employs derogatory or discriminatory language concerning individuals or groups based on intrinsic attributes such as religion, ethnicity, nationality, race, color, descent, gender, or other identity factors. This type of discourse can lead to significant psychological and emotional distress among recipients, such as stress, anxiety and depression (Tynes et al. 2008). Beyond the immediate emotional impact, prolonged exposure to HS can also erode social cohesion, fostering an atmosphere of mistrust and polarization. This divisiveness can further perpetuate the cycle of hate and individuals may become increasingly isolated within their own echo chambers, reinforcing existing biases and prejudices (MediaSmarts 2021).

Many organizations, recognizing the urgency of addressing the proliferation of HS on social media, have initiated the release of guidelines and policies designed to mitigate this issue. However, the sheer scale of the problem, characterized by the continuous generation of vast volumes of data on these platforms, presents an inherent challenge to manual classification methods. Manual intervention is ultimately rendered impractical due to its time-consuming nature, underscoring the need to employ Machine Learning (ML) techniques to automate and streamline the classification process, thereby producing more dependable and efficient results (Qian Li et al. 2022). As a consequence of this technological shift, a dynamic landscape of research and development has emerged, aimed at harnessing the power of ML for HS detection.

Various techniques, ranging from approaches like traditional ML and Deep Learning (DL) models, have been applied with promising results, and recently, with the development of Transformer-based models (Vaswani et al. 2017), we have seen a growing expansion in the HS detection landscape.

The recent advances in Transformer-based models have introduced new possibilities in HS detection, but a comprehensive synthesis of these efforts is lacking, particularly in terms of comparing them to other ML methods.

This Systematic Literature Review (SLR) addresses this gap by exploring the current research landscape of HS detection on social media, with a specific focus on Transformer-based models. We aim to answer the following research questions:

  • Q1: What is the landscape of HS detection literature since the development of Transformer-based models?

  • Q2: How do Transformer-based models compare to other ML solutions in the context of HS detection?

  • Q3: What are the characteristics of the data used for HS detection?

This article makes three key contributions: (1) it offers a comprehensive review of HS detection methods with a focus on Transformer-based models, (2) it compares these models with other ML techniques in terms of performance and applicability, and (3) it identifies key datasets and challenges in the field to inform future research.

This document is organised as follows: Sect. 2 gives some background on what is HS and how it is defined across several organizations and research initiatives; Sect. 3 delves into the methodological aspects of the SLR, outlining the search strategy, inclusion criteria, and data extraction processes; Sect. 4 presents a comprehensive analysis of the selected studies, highlighting the key findings and principal results; finally, Sect. 5 presents the major conclusions and pinpoints current limitations and future directions.

2 Background

As previously stated, defining HS is not an easy task, since this is a complex phenomenon that is heavily reliant on the subtleties of language. It is nonetheless necessary to understand how HS is defined and what constitutes it, in order to begin to detect and combat it. Many organizations, companies and countries have defined HS in their policies and bellow we can see some examples of this definitions. Since this SLR was developed in the scope of the kNOwHATE: kNOwing online HATE speech project (kNOwHATE 2023), we also provide the definition used in the project:

  • United Nations: “any kind of communication in speech, writing or behaviour, that attacks or uses pejorative or discriminatory language with reference to a person or a group on the basis of who they are, in other words, based on their religion, ethnicity, nationality, race, colour, descent, gender or other identity factor” (United Nations 2019).

  • Meta hate speech policy: “a direct attack against people - rather than concepts or institutions - on the basis of what we call protected characteristics: race, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease. We define attacks as violent or dehumanising speech, harmful stereotypes, statements of inferiority, expressions of contempt, disgust or dismissal, cursing and calls for exclusion or segregation” (Meta 2023).

  • Twitter policy on hateful conduct: “attack other people on the basis of race, ethnicity, national origin, caste, sexual orientation, gender, gender identity, religious affiliation, age, disability, or serious disease” (Twitter 2023).

  • YouTube hate speech policy: “content that promotes violence or hatred against individuals or groups based on any of the following attributes, which indicate a protected group status under YouTube’s policy: Age, Caste, Disability, Ethnicity, Gender Identity and Expression, Nationality, Race, Immigration Status, Religion, Sex/Gender, Sexual Orientation, Victims of a major violent event and their kin, Veteran Status” Google (2019).

  • Definition in the eBook The Content and Context of Hate Speech: “is directed against a specified or easily identifiable individual or, more commonly, a group of individuals based on an arbitrary or normatively irrelevant feature... stigmatizes the target group by implicitly or explicitly ascribing to it qualities widely regarded as undesirable... casts the target group as an undesirable presence and a legitimate object of hostility” (Parekh 2012).

  • kNOwHATE project: building on scholar definitions (i.e., Siegel 2020) and guidelines provided by the Council of Europe in its latest recommendation (CM/Rec/2022/16), the project defines online HS as “bias-motivated, derogatory language that spread, incite, promote, or justify hatred, exclusion, and/or violence/aggression against a person/group because of their group membership” (Carvalho and Guerra 2023).

When examining the various interpretations of Hate Speech used by multiple organizations and research initiatives, we can identify some similarities. Firstly, all definitions mention that HS targets a specific group or individual based on his/her group membership, and not concepts or institutions. Secondly, these groups are targeted with malicious intent, based on real or attributed characteristics, and some organizations consider this characteristics as protected. Depending on the characteristic that is being targeted, there are different categories of HS. The main characteristics mentioned in the aforementioned definitions include religion, ethnicity, nationality, race, colour, descent, gender, and sexual orientation.

This work focuses on analyzing studies related to HS detection, especially those that define HS within comprehensive frameworks. It also includes studies addressing offensive and abusive speech, recognizing that these types of speech are frequently discussed alongside HS in the literature. Although offensive and abusive speech do not involve targeting individuals based on group membership (as is the case with HS) (Carvalho and Guerra 2023), the detection methods used for these types of speech are quite similar.

In order to maintain clarity, the remainder of the article refers to these collective studies (HS, offensive, and abusive speech detection) as HS detection works. Nevertheless, Sect. 4 presents specific statistics about the number of studies addressing each type of speech, as this breakdown may be of interest to certain readers. This approach provides a clear and concise way to streamline the discussion while still offering the detailed analysis and statistical information for those who may want to differentiate between the types of speech.

3 Methodology

This section presents an overview of the methodologies employed in this SLR. In developing our methodology, we drew inspiration from the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (Page et al. 2021). PRISMA provides a widely recognized framework for conducting systematic reviews, ensuring transparency and methodological rigor in the review process. The structured approach outlined in PRISMA facilitated a comprehensive overview of the methodologies employed in our SLR, covering key aspects from search criteria delineation to data extraction. Our goal was to adhere to the principles of PRISMA to enhance the reliability and reproducibility of our review and to ensure a robust and exhaustive coverage of the literature under review.

Our primary goal is to provide an analysis focusing on key trends in performance across different methods employed in the field of HS detection within the context of social media. Specifically, our review seeks to address the following key objectives: First, we aim to examine the ML and natural language processing (NLP) methods that have been utilized for the identification and classification of HS in social media platforms and how they have changed with the introduction of Transformer models, to better understand what are the current trends and future perspectives (Q1); Then, we compare the several methodologies employed with one another and with Transformer models, to identify which ones achieve better results (Q2). Finally, we analyse the characteristics of the resources being used in the scope of this task, like languages and data sources, to identify which areas can be further developed (Q3).

In the end we also delve into the current challenges and limitations that researchers face in this domain, with a focus on proposed strategies and potential solutions. By addressing these goals, we aim to offer valuable insights into the state-of-the-art in HS detection in social media, thus facilitating a better understanding of the field and its future directions.

To accomplish this, we first defined criteria to search and select studies to be examined in our SLR, relevant to our objectives. We selected two databases, Scopus and Web of Science, since they both have an extensive coverage of literature, across diverse academic fields. This is beneficial, since HS detection can be seen as multidisciplinary problem ranging from linguistics and social sciences to computer science, so it is necessary to search in databases that index a wide range of journals, in a variety of disciplines.

The search query was designed to maximise the retrieval of studies pertinent to our subject, and for that the following keywords were established: ‘hate speech’, ‘abusive’, ‘offensive’, ‘classification’ and ‘detection’. ‘Hate speech’ is the most common keyword used in this subject by the scientific community, since it is also a legal term in many countries. The terms ‘offensive’ and ‘abusive’ were also added as previously mentioned since they convey a similar idea, in the sense that HS can be seen has an extreme of abusive text, and all of them share an offensive aspect (Alkomah and Ma 2022). This terms are also present in the literature as key terms to use when finding relevant studies (Alrashidi et al. 2023; Mullah and Zainon 2023; Yin and Zubiaga 2021). These keywords were used in addition to Boolean operators to form our search query (“hate speech” OR “abusive” OR “offensive”) AND (“classification” OR “detection”). Our query was applied to the following parts of the studies: title, abstract and keywords.

To define which articles should be included or omitted from our SLR some inclusion and exclusion criteria were set to keep only the studies that fulfilled our goals for this work.

The inclusion criteria were the following: firstly, to capture the most recent developments in the field, and since we want to focus on Transformer-based models, we limited our search to studies published from 2017 to the present day, since it was in 2017 that the Transformers architecture was introduced (Vaswani et al. 2017), and with that came a growing interest in this area. Furthermore, to facilitate the comprehension and analysis of the research, we restricted our selection to studies written in the English language. To assure high-quality and peer-reviewed research, only journal articles were considered for inclusion, while conference papers, data papers, and similar publications were excluded. Additionally, we aimed to select studies that were published in journals with a high impact factor, specifically those ranked in Quartiles 1 and 2 based on ScimagoFootnote 1 journal quality rankings. Given the emphasis of this review on HS classification, we prioritized articles whose primary focus centred on this specific area of research and that proposed or discussed solutions related to this classification task.

The exclusion criteria were: articles primarily focused on other forms of media, such as images, sound, memes, and non-textual content, articles that lack a clear approach or technical content related to HS classification, and finally, studies that do not centre their main objectives on HS detection, but on another task, like the development of HS resources.

Although we decided to include only journal articles, we recognize that by excluding high impact peer-reviewed conferences we are limiting the inclusion of cutting-edge research, so in order to mitigate this side effect we decided to include the most relevant papers of two tasks held in the context of the SemEval international workshops of 2019 and 2020, published by the Association of Computational Linguistics (ACL). In these years’ editions the OffensEval task were held, that aimed at detecting offensive language. By including the most relevant studies papers of a competition with a high degree of participation, we believe we get a glimpse of that time’s best techniques for the task. Additionally, to ensure comprehensive coverage of recent innovations, we extended our search to include ACL conference papers published between 2020 and 2024 that met our inclusion criteria, specifically selecting long papers from the main conference proceedings.

Fig. 1 shows the number of records identified in the database search, and the filtering process that is applied afterwards, using a PRISMA flow diagram (Page et al. 2021). Our initial query resulted in 2876 studies, plus the 15 ACL studies selected. After the removal of duplicate entries, and the application of exclusion criteria, we were left with 105 articles for full-text analysis. After assessing the full text of the 105 articles selected from our inclusion/exclusion criteria, an additional three articles were discarded because the dataset used for HS detection was not manually annotated, but instead algorithms were used to automatically annotate the data used for building the classifiers (Ayo et al. 2021; Lee et al. 2022; Roy et al. 2023). Given the nuanced and context-dependent nature of HS, the reliance on automated processes for annotation introduces potential biases and inaccuracies that may compromise the robustness and reliability of the classifiers developed in these studies, leading to the final 102 articles considered for our SLR.

Fig. 1
Fig. 1
Full size image

PRISMA flow diagram

For the full-text analysis of our studies, data extraction is a critical component, since it helps to collect information in a methodological and comprehensive way, so we employed a rigorous and systematic approach that involved the identification and extraction of key elements from each study, that answered our initial objectives. The data collected was mainly about the datasets utilized in each study, the methods they used for the classification task (algorithms, pre-processing, feature representation, etc.), the metrics used to evaluate the performance (with the actual values obtained) and the principal findings and limitations. For this, an extraction form was used in order to ensure consistency.

4 Results

The findings of this SLR are presented in this section, divided into four distinctive categories: An overall analysis of the results of our search (Sect. 4.1), an analysis of the evolution of HS detection (Sect. 4.2), Methods and Algorithms where we will compare all different approaches employed for this task (Sect. 4.3), and Resources where both the languages and the types of data used for the detection will also be analyzed (Sect. 4.4). Through a meticulous synthesis of empirical evidence and critical evaluation, this section aims to provide a comprehensive overview of the state-of-the-art in HS detection.

4.1 Overall analysis

When analysing the initial results of the 2155 (2140 plus 15) articles not duplicated, resulting from our search query we can see in Fig. 2 a notable upsurge in the volume of studies related to HS detection, confirming the increasing significance of this topic within the research community. Over the years, we observed a considerable growth in publications, with the data indicating a substantial increase in the number of studies published annually. In 2017, 65 relevant studies were identified, which increased almost 10 times to the 588 results found in 2022. Since the search was conducted in September and the current year (2023) has not come to an end at the time of writing, the lower number of publications found (369) is not surprising. We have also added the number of documents included in our SLR from each year. This graph confirms the growth of this research topic and the need for an updated review.

Fig. 2
Fig. 2
Full size image

Number of search results (orange) and included documents (blue) by year

Our search across the Scopus and Web of Science databases yielded a substantial number of results, with 1663 studies identified in Scopus and 1213 in Web of Science. The presence of these studies across both platforms emphasizes the widespread recognition and coverage of the topic within the academic community, while also reflecting the diversity of academic sources that contribute to this discourse.

Categorically, the types of studies were delineated into two main groups: conference papers and journal articles, has shown in Fig. 3. The data demonstrated that conference papers constituted most of the studies, with 1645 identified. In contrast, 1025 studies were classified as journal articles. This can be explained in part by the number of competitions dedicated to the task of HS classification (Basile et al. 2019; Zampieri et al. 2020; Wiegand et al. 2018), from which a large number of conference articles result, since each participant has their contribution in the form of a conference paper.

Fig. 3
Fig. 3
Full size image

Number of documents by type

Our initial search results show the growing prominence of HS classification as a research field, the substantial volume of studies dedicated to the topic, and the diverse types of publications contributing to this evolving discourse. This data forms a valuable foundation for the subsequent synthesis and filtering of the findings in our initial search. Moving forward the results presented will be of the final 87 studies considered for this SLR.

4.2 Q1: Landscape of HS detection literature

Over the years, various approaches have been employed for hate speech (HS) detection, with notable evolution in the methods used. This section provides an overview of the five major approaches - Traditional ML, DL, Transformers, Generative Models, and Multi-Task Learning – and examines their progression and impact on HS detection throughout the years.

Fig. 4
Fig. 4
Full size image

HS detection approaches by year (MTL: Multi-Task Learning)

Fig. 4 illustrates the evolving trends in the application of different approaches to HS detection, highlighting a clear shift in techniques over time. By 2019 DL techniques became more prevalent, reflecting the growing interest in neural network-based methods for HS detection. This increase aligns with the first OffensEval task, where most participants employed DL models, marking them as the state-of-the-art approach at that time. In 2020 and 2021, the landscape of HS detection continued to evolve. Transformer-based models began to gain significant traction, with seven studies in 2020 and five in 2021. This surge in popularity aligns with the introduction of the Transformer architecture by Vaswani et al. (2017), which took about three years to be widely adopted for HS detection. The second OffensEval task further solidified this trend, as most competitors shifted to BERT-based models, confirming that Transformers had become the dominant approach during this period. Although traditional ML methods continued to be used, Multi-Task Learning (MTL) emerged for the first time, with one study appearing in both 2020 and 2021. In 2022 and 2023, we observed a more diverse set of approaches in HS detection. DL remained prominent, while Transformers continued to grow in popularity, becoming the go-to method with 10 studies in 2022 and 14 in 2023. Although traditional ML techniques remained relevant, their usage declined. Generative and Multi-Task Learning models, newer approaches in the field, began to gain recognition in 2023, highlighting their potential for HS detection. In 2024, two studies featuring Transformer models were published, both coinciding with ACL papers extracted after the search, explaining their presence as the only studies from that year.

Fig 5 shows the total number of studies that employed each approach. DL and Transformers are the most frequently used methods, with 40 and 38 studies, respectively, accounting for over two-thirds of the research reviewed. Traditional ML follows with 19 studies, while Multi-Task Learning and Generative Models are represented by four and two studies, respectively. These findings underscore the significant impact of Transformers on the HS detection landscape, as they have become the preferred choice for many researchers in recent years.

Fig. 5
Fig. 5
Full size image

Number of HS studies, grouped by approach

The authors of the OffensEval-2019 reported that over half of the participants explored Deep Learning models (Basile et al. 2019). In contrast, OffensEval-2020 saw most teams utilizing pre-trained Transformer models, with all of the top 10 teams employing either BERT, RoBERTa, or XLM-RoBERTa (Zampieri et al. 2020)..

Table 1 SemEval top papers
Full size table

The results presented may be limited by the relatively small number of articles included in our analysis, potentially misrepresenting broader trends. To address this, we supplemented our review with conference papers from the top participants in OffensEval-2019 and OffensEval-2020, as well as other selected ACL papers, to provide a more comprehensive representation of the state-of-the-art solutions during that period. As shown in Table 1, the results from these conferences align with our findings, demonstrating a clear transition from ML and DL approaches in 2019 to the adoption of Transformer-based models in 2020.

In summary, the evolution of HS detection methods shows a clear shift from traditional, simpler ML techniques to more advanced DL and Transformer-based models. The field has also seen a growing diversity of approaches, with Generative Models (GM) and Multi-Task Learning (MTL) gaining prominence in recent years. This progression highlights the dynamic nature of the research landscape and the continuous efforts to enhance HS detection in digital environments.

4.3 Q2: ML solutions for HS detection

As previously discussed, a wide range of approaches have been employed for HS detection, from traditional ML methods to more advanced DL and Transformer-based models. This section compares these approaches to determine which methods yield the most promising results and whether Transformers have consistently outperformed other models. To facilitate this comparison, we categorize the studies into five distinct approaches. Before examining each in detail, we provide a brief summary of each category to clarify their key differences.

ML focuses on the development of algorithms and statistical models that enable computers to perform tasks without explicit programming. The core idea is to allow machines to learn patterns and make decisions based on data. DL is a subset of ML that employs neural networks with many layers, that are more complex than traditional ML models, to analyze and learn from data.

Multi-Task Learning is an approach where a single model is trained to perform multiple related tasks simultaneously. The goal is to enable the model to learn shared representations and features across tasks, potentially leading to improved performance compared to training separate models for each task. Generative Models are a class of ML models that aim to generate new data samples that resemble a given training dataset, increasing the amount of data available for training. Finally, Transformers use transfer learning, by taking advantage of models pre-trained on large datasets for unsupervised tasks that capture general language patterns, and fine-tuning them with smaller labeled datasets on specific tasks, leveraging this pre-existing knowledge. This transfer of knowledge allows the model to generalize well to diverse tasks, enhancing performance and efficiency.

In the subsequent sections, we delve into the findings of studies adopting each of these approaches, assessing their effectiveness and making comparisons with one another.

4.3.1 Traditional machine learning

Starting with traditional ML techniques, we identified 16 studies that resorted to this type of method, and made comparisons with various algorithms. Support Vector Machines (SVM) and Logistic Regression (LR) where the algorithms that achieved better results, outperforming other ML algorithms in three different studies respectively. Pitropakis et al. (2020); Shannaq et al. (2022); Mohapatra et al. (2021) obtained better results with a combination of SVM with n-grams and pre-trained embeddings, when compared with other traditional ML models.  Indurthi et al. (2019) and Pérez and Luque (2019) managed to obtain good results with an SVM model with a RBF and linear kernel respectively, topping the standings in the OffensEval-2019 task. Arcila-Calderón et al. (2021); Vanetik and Mimoun (2022); Saeed et al. (2023) employed a LR model with pre-trained embeddings and managed to outperform other traditional ML models. Other models that obtained good results were Random Forest (RF) with count vectorizer embeddings, that managed to outperform Bagging and Adaboost models (Turki and Roy 2022), and the j48graft classifier, a type of Decision Tree (DT) model, combined with text features (Watanabe et al. 2018).

Recently pre-trained Transformer embeddings have been used in combination with traditional ML models to improve performance. By using these embeddings as input features for traditional ML models, they benefit from their ability to capture intricate relationships and context in the text data, which can be challenging for traditional feature engineering methods. (García-Díaz et al. 2023; Vanetik and Mimoun 2022) combined Bidirectional Encoder Representations from Transformers (BERT) embeddings with a Multi-Layer Perceptron (MLP) and LR models respectively, and managed to outperform ML and EM. In addition to this, (Raut and Spezzano 2023; Vanetik and Mimoun 2022) showed that combining traditional ML models with BERT embeddings can even outperform DL and Transformers on its own.

Ensemble Models have gained prominence in the realm of HS detection, as a strategic approach to overcome limitations associated with individual models. This models involve combining predictions from multiple models to enhance overall performance, making them a compelling alternative for addressing challenges posed by the use of single models in HS detection. seven studies used an ensemble of ML models, and although these models did not outperform Transformers and DL models, they managed to outperform single ML models, showing that they can enhance the performance of these simpler models, by combining them. four of this models used majority voting to get the predictions (Khairy et al. 2023; Aljero and Dimililer 2021; Rajalakshmi et al. 2023; Plaza-Del-Arco et al. 2020), two studies used a LR meta classifier (Agarwal and Chowdary 2021; Oriola and Kotze 2020), and one study used a stacking approach (Mullah and Zainon 2023).

Traditional ML models can be used effectively for the task of HS detection, and recent improvements show that this type of simpler model, when combined with a richer textual representation, or in an ensemble with other simple models, can even surpass more complex models like Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), Bidirectional Gated Recurrent Unit (BiGRU) and BERT based models (Saeed et al. 2023; Raut and Spezzano 2023; Vanetik and Mimoun 2022).

4.3.2 Deep learning

Jumping to DL techniques, these have been extensively used for the task of HS detection, with 37 studies employing this method. These studies have explored a variety of DL models, including CNNs, LSTMs, GRUs, and hybrid or ensemble models that combine multiple DL architectures as we can seen in Fig. 6.

Fig. 6
Fig. 6
Full size image

Different DL models for HS detection

CNNs have been used to effectively capture the local patterns and features of text, making them well-suited for identifying HS. They have been applied in several HS detection studies (Karayiğit et al. 2021; Akhter et al. 2022; Roy et al. 2020; A. T. Kabakus 2021; Zhang and Luo 2019; Duwairi et al. 2021; Alshalan and Al-Khalifa 2020; Mozafari et al. 2020) with promising results, even outperforming Transformers (Alshalan and Al-Khalifa 2020), and getting 3rd place in OffenseEval-2019.

LSTMs are another class of recurrent neural networks (RNNs) that are capable of capturing long-range dependencies in text. This makes them well-suited for handling the sequential nature of language, which can be important for identifying HS. They have also been used in several HS detection studies (Priyadarshini et al. 2023; Ayo et al. 2020; Dascălu and Hristea 2022; Pronoza et al. 2021; Pereira-Kohatsu et al. 2019; Madhu et al. 2023; Montejo-Ráez et al. 2019).

Both CNNs and LSTMs are two of the most widely used DL architectures for HS detection. CNNs can capture local patterns and features in text, while LSTMs are adept at handling long-range dependencies. The results of using CNNs and LSTMs for HS detection are somewhat mixed with some studies have shown that CNNs outperform LSTMs (Roy et al. 2020; A. T. Kabakus 2021), while others have found the opposite (Madhu et al. 2023; Dascălu and Hristea 2022; Ayo et al. 2020).

Taking advantage of these mixed results, hybrid models that combine these two types of models have consistently shown strong performance. These models leverage the strengths of each architecture, leading to improved results and generalizability. For example, CNN-BiLSTM models have been shown to outperform even Transformers in some studies (Mundra and Mittal 2022; Fazil et al. 2023). This suggests that hybrid models may be able to more effectively capture the complexities and nuances of HS. In addition, CNN-BiGRU models have also shown promising results, by combining the local feature extraction ability of CNNs with the long-range dependency modeling ability of BiGRU’s they managed to outperform all other single DL models (Kamal et al. 2023; Aarthi and Chelliah 2023). Nine other studies used an ensemble approach of DL models managing to outperform single DL and ML models, and in some cases even the state-of-the-art Transformers. A majority voting ensemble of several LSTM models with different features (Pitsilis et al. 2018), a meta classifier of several combinations of models with different embeddings (Cruz et al. 2022), a combination of a BERT, BiLSTM and BiGRU models (Mazari et al. 2023) and finally a deep neural network with several text features (Al-Makhadmeh and Tolba 2020) all managed to outperform ML and DL models with good results. In addition, five other studies managed to get better results than all other approaches (ML, DL and TL). These studies employed an ensemble of CNN models (Zhou et al. 2020), BERT models (Mridha et al. 2021), bagging of BiGRU, BiLSTM, CNN (Mahajan et al. 2024), a stacking of BiLSTM, LSTM, CNN and CNN-LSTM models (Muneer et al. 2023) and a combination of a BERT, MuRIL and DNN models (Roy et al. 2022).

Ensembles emerge as a compelling solution to HS detection, especially when individual models like CNNs or LSTM’s do not perform well. By leveraging the strengths of diverse architectures and addressing limitations in generalization and imbalanced datasets, ensembles offer a robust and effective approach for enhancing the accuracy and reliability of HS detection systems even managing in some cases to outperform the state-of-the-art models.

Similarly to LSTM’s, GRU’s are also type of RNN’s that are capable of capturing short-term dependencies in text. They were used in three HS detection studies, even doe the comparisons were made with traditional ML models, that they outperformed  (Keya et al. 2023; Albadi et al. 2019; Kar and Debbarma 2023). Another study used a BiGRU model managing to place top two in the OffensEval-2019 task (Ding et al. 2019). Other DL models used where Bidirectional RNNs (BiRNNs) (Anezi 2022) and a Softmax clasifier combined with text features (Sharmila et al. 2022).

These studies demonstrate the versatility and effectiveness of DL techniques for HS detection. DL models can capture complex patterns in text, making them well-suited for identifying subtle and nuanced forms of HS. Additionally, hybrid models can combine the strengths of different DL architectures to further improve performance.

4.3.3 Transformer-based models

The Transformers were by far the ones that achieved the most promising results, surpassing the state-of-the-art models in almost all studies that employed them, outperforming all other approaches in most cases. It was also the most used approach with 29 studies. The success of the basic BERT model on a plethora of different NLP tasks lead to the widespread use of this models and a large number of variants. This is mirrored on the large number of studies that employed this models for HS detection.

A fine-tuned version of the basic BERT model for the English language was used in nine studies (Boulouard et al. 2022; Casavantes et al. 2023; Arcila-Calderón et al. 2022; Toliyat et al. 2022; Vashistha and Zubiaga 2021; Fan et al. 2021; Shanmugavadivel et al. 2022; Pamungkas et al. 2021; Sotudeh et al. 2020), outperforming all DL and ML models compared in the respective studies. Other variants of the BERT model that were retrained in other languages were also implemented, like BETO for spanish (Benítez-Andrades et al. 2022; Plaza-del Arco et al. 2021; Perez et al. 2023; Valle-Cano et al. 2023), RuBERT for Russian (Bilal et al. 2023; Pronoza et al. 2021), RoBERTuito also for Spanish (Molero et al. 2023), UmBERTo for Italian (Ramponi et al. 2022), MARBERT for Arabic (Alrashidi et al. 2023), HindiBERT for Hindi (Bhardwaj et al. 2023), Arabic BERT-mini also for Arabic (Almaliki et al. 2023), MuRIL for seventeen indian languages (Kapil et al. 2023) and NAIJAXLM-T for English and Nigerian (Tonneau et al. 2024). It is also relevant to mention that this list goes beyond the set of articles found by our SLR and includes models such as BERTimbau widely used for Portuguese (Santos et al. 2022; Matos et al. 2022) and BERTje for Dutch (Markov et al. 2022). Besides this BERT models retrained for other languages, there are also multilingual models being developed like mBERT and XLM-RoBERTa that were trained with multilingual data and can be used in many languages. The mBERT model was used in four studies (Rodriguez-Sanchez et al. 2020; Dowlagar and Mamidi 2022; Kapil et al. 2023; Bigoulaeva et al. 2023) and the XLM-RoBERTa was used in five studies (Liu et al. 2023; Awal et al. 2023; Subramanian et al. 2022; Wang et al. 2020; Dadu and Pant 2020). In addition to the models retrained on other languages, there have also been models with different architectures or hyperparameters than BERT, also used for HS detection like RoBERTa (Dowlagar and Mamidi 2022; Arshad et al. 2023; Kaminska et al. 2023; Hartvigsen et al. 2022; Wiedemann et al. 2020; Bansal et al. 2020), ELECTRA (Aurpa et al. 2021) and AlBERT (Wiedemann et al. 2020). More recently, models like GPT\(-\)3.5 are also being used for this task, like the case of Zhang et al. (2024).

Transformers emerged as the most promising strategy for HS detection, consistently outperforming other methods across all studies. The versatility and adaptability of TM, coupled with the development of specialized variants and hybrid approaches, have significantly advanced the field of HS detection, paving the way for more comprehensive and effective measures to combat online HS.

4.3.4 Generative models

As we have seen, there has been a recent surge in the use of Generative Models, with two studies employing this method in the year 2023. Su et al. (2023) utilized a Semi-Supervised Learning Generative Adversarial Network (GAN) architecture. The model incorporates RoBERTa sentence features as the backbone, combining them with a generator that introduces random noise and a discriminator for adversarial training. In this study the authors also used vast amounts of unlabelled data from another related domain, and demonstrated that the generative model outperformed the baseline RoBERTa model without the additional data generation. In another study, Cohen et al. (2023) combined multiple generative models for HS detection. This model utilizes DeBERTa Large as a foundational element and incorporates back-translation augmentation to enhance the diversity of the training dataset. Furthermore, the integration of Generative Pre-trained Transformer (GPT) and Test-Time Augmentation demonstrated superior performance compared to baseline models, highlighting the effectiveness of generative models in achieving state-of-the-art results in HS detection.

The combination of pre-trained language representations, in this case RoBERTa and DeBERTa, and generative capabilities allows these models to capture intricate patterns and nuances present in HS texts. Generative techniques facilitate the augmentation of the training dataset, addressing issues related to limited labeled data in HS detection scenarios, like is the case with low-resource languages. This, in turn, enhances the generalization capabilities of the models, ensuring better performance on unseen HS text. In addition, adversarial training allows models to discern subtle differences between authentic and deceptive HS content, contributing to heightened discriminative power in HS detection. The utilization of Generative Models in HS detection has the potential to address one of the most common challenges in HS detection scenarios, being the lack of training data, that needs to be manually collected and annotated. With the introduction of this models, HS detection in low-resource languages can be done, without the need of extensive collection and annotation of data.

4.3.5 Multi-task learning

Previous studies have established the relevance of sentiment features in aiding HS detection tasks (Al-Makhadmeh and Tolba 2020; Sharmila et al. 2022; Watanabe et al. 2018). Recognizing the potential benefits of incorporating sentiment-related features, researchers have extended their exploration into Multi-Task Learning. The prevalent idea is that HS is a negative type of discourse, that has associated emotions like anger, rejection and criticism, so in the Multi-Task Learning framework, the model is designed to simultaneously learn and optimize multiple tasks during training, through shared representations. Specifically, in the context of HS detection, the model is tasked with emotion and sentiment classification in addition to HS detection. Shared representations are employed across these interconnected tasks, allowing the model to leverage common knowledge and patterns present in the data, aiming to enhance the overall performance of HS detection models.

Studies referenced earlier have highlighted the informative nature of sentiment features in HS detection. This recognition has spurred further investigation into Multi-Task Learning, where sentiment and emotion classification tasks are jointly addressed to bolster HS detection capabilities. Recently four studies have employed Multi-Task Learning for HS detection task. Two studies leveraged Multi-Task Learning to concurrently address emotion and sentiment classification alongside HS detection (Plaza-Del-Arco et al. 2021; Zhou et al. 2021). By sharing information across these related tasks, the model aimed to capture linguistic nuances associated with HS. This integrated approach demonstrated notable improvements over ML and DL models. Following this work, Min et al. (2023) also developed a Multi-Task Learning model that tackled emotion classification in conjunction with HS detection, obtaining a better performance when compared with the baseline Single-Task Learning model. The last study that employed Multi-Task Learning diverged from the previous two, choosing to develop a model that addressed simultaneously post level and token level aggression (Zampieri et al. 2023).

Multi-Task Learning, specifically integrating emotion and sentiment classification with HS detection, emerges as a promising avenue for HS detection. The studies discussed underscore the effectiveness of Multi-Task Learning, leading to improved model performance. However there’s a downside to this approach, since the quality of corpora is important in a Multi-Task Learning environment, and having enough data with quality is not always possible, especially in low-resource languages.

4.4 Q3: Data characteristics for HS detection

In this section we look into the different languages where studies have been developed to detect HS, and also what are the different sources where researches look to gather data for the development of their models. This information will allow us to understand which languages researches have focused their work on, and which languages are less explored and may be more vulnerable to the negative effects HS. By looking at the data used we will also be able to see if data has been collected from a vast plethora of places, or if studies have all converged to the same sources, thus making the models less likely to be able to perform well outside their scope.

4.4.1 Information sources

The majority of studies use data collected from different social media platforms as seen by Fig. 7. They are a rich source of data for HS detection, given the extensive volume of user-generated content. Twitter,Footnote 2 in particular, stands out as the dominant source in HS detection research, with a staggering 73 studies using Twitter data. The brevity and public nature of tweets make them highly accessible for research purposes. The Twitter platform has been a focus due to the ease of collecting and processing large datasets. While Twitter leads the way, other social media platforms also contribute to the HS detection landscape. Facebook,Footnote 3 YouTube,Footnote 4 InstagramFootnote 5 and RedditFootnote 6 are also present with 10, 11, three and three studies respectively. These platforms, although less prevalent, offer insights into the multifaceted nature of HS across different online environments.

HS detection research also explores data outside of social media, like news sites and alternative platforms that cater to specific communities. Sites like Fox News and others provide eight instances and niche platforms like GABFootnote 7 and Stormfront,Footnote 8 known for its association with far-right ideologies, contributes eight instances. The inclusion of such sources allows for a more comprehensive examination of HS across diverse online spaces.

Fig. 7
Fig. 7
Full size image

Data sources for HS detection

It is important to note that not all data sources are created equal. Twitter, with its character limit, differs significantly from platforms like Facebook or YouTube, where users have more space to express their views. Furthermore, news websites and comments may not share the same characteristics as tweets, as they often involve more formal language and context. Researchers must consider these nuances when developing and evaluating HS detection models to ensure their applicability across various platforms.

HS detection research draws data from a wide range of sources, with Twitter being the primary contributor. The prevalence of Twitter data highlights its accessibility and suitability for large-scale studies. However, it is essential to recognize the distinctions among different sources in terms of content, context, and user behavior. Future research in this field should continue to explore a diverse array of sources to gain a more comprehensive understanding of HS in the digital landscape.

4.4.2 Languages

HS is a pervasive problem that transcends geographic and linguistic boundaries. It is a global issue, and researchers have recognized the need to address it in various languages. However, the research landscape in the domain of HS detection has exhibited a notable focus on the English language, as evidenced by Fig. 8. A significant portion of research efforts, resources, and datasets have been concentrated on English, with 60 studies focusing on this language. Nonetheless other languages were explored, like Spanish, Arabic and Hindi, with 16, 11 and eight studies respectively.

Fig. 8
Fig. 8
Full size image

Languages where HS detection was conducted

Recognizing the need to combat HS in various linguistic environments, researchers are increasingly turning their attention to low-resource languages. These languages often lack the extensive datasets and resources that are readily available for English, but has we can see, some work is beginning to be made in order to include this languages in this field. For instance, even though Portuguese has only one study in our research, there have been recent attempts to create curated datasets for HS detection (Carvalho et al. 2022, 2023).

One promising avenue for addressing HS in low-resource languages is the utilization of Transformer-based models, since they can leverage knowledge from languages with more extensive resources, like English, and fine-tune it on the limited available data for a specific language, bridging the resource gap to some extent. Transformer-based models, particularly those pre-trained on multilingual data, have shown promise in cross-lingual HS detection. These models can generalize across multiple languages, learning universal language features that enable them to detect HS irrespective of the language used. They can be effective on zero-shot cross-lingual HS detection where by using a high resource source language for training, like is the case with English has we’ve seen, models can classify low-resource target languages with promising results (Bigoulaeva et al. 2023; Pamungkas et al. 2021). Additionally, by fine-tuning these models on a small dataset in the low-resource target language, researchers can effectively extend HS detection capabilities to languages with limited resources (Awal et al. 2023; Liu et al. 2023).

4.4.3 Types of speech

As noted earlier, not all of the included studies focus solely on HS, as our search criteria also encompassed offensive and abusive speech. As illustrated in Fig. 9, the majority (83%) studies included address either HS alone or a combination of HS with other types of speech. The remaining 17% were split between 11% of studies focused on offensive speech and 6% on abusive speech.

Fig. 9
Fig. 9
Full size image

Percentage of works dealing with the different types of speech included in our SLR

Tables 2, 3, 4, 5, 6, 7, 8, 9, 10 in Appendix A, present all the studies that were included in this SLR with the most important information about the approach each work followed for the task of HS detection.

5 Impact of transformer-based models

As we have seen throughout this SLR, Transformers have had an impact on almost all areas of HS detection. Firstly, theses models have been gaining traction in HS detection tasks and since 2022 have been the most used models, which clearly indicates their popularity and success among researchers. These models, characterized by their ability to capture intricate linguistic patterns and contextual nuances, have consistently demonstrated superior performance compared to traditional ML techniques and other DL architectures. Studies highlighted in our review consistently show that Transformers outperform other highly used models such as CNN’s, LSTM’s, SVM’s and Ensemble models. Moreover, besides Transformers having a better standalone performance they have also been incorporated into other models to further enhance detection accuracy. They have been used to enhance the performances of other models, or by taking advantage of their rich text representation, has features, or by combining them into hybrid or ensemble models. Furthermore, the advent of Transformers has catalyzed the development of HS literature and research, particularly in addressing challenges posed by low-resource languages. By leveraging pre-trained multilingual representations and fine-tuning on target languages with limited resources, Transformers have significantly expanded the scope of HS detection to encompass a broader array of linguistic contexts. In summary, the impact of Transformers on HS detection cannot be overstated. Their superior performance, integration into hybrid models, and facilitation of research in low-resource languages underscore their significance as the cornerstone of modern HS detection methodologies. Moving forward, continued advancements in Transformers hold immense promise in furthering our understanding of online HS dynamics and fostering safer digital environments for all users.

6 Conclusion

This work provides a comprehensive review of the evolution of hate speech (HS) detection, particularly focusing on the shift from traditional machine learning (ML) approaches to the dominance of Transformer-based models. Our review has shed light on several key lessons that will shape future efforts in this field. First, while Transformer models consistently outperform traditional ML and deep learning (DL) approaches in terms of performance, the trade-offs in computational demands highlight the need for context-specific solutions. Transformers excel in large-scale, multilingual applications, but DL models may offer faster, resource-efficient alternatives for specific tasks.

Second, our review reveals a growing yet underexplored interest in generative models and multi-task learning for HS detection. These approaches, while still in their infancy, show promise for handling more complex linguistic features of hate speech and addressing cross-platform variations in data. Moreover, the multilingual and cross-lingual capabilities of Transformer models present a significant advance, particularly for low-resource languages, suggesting a positive shift toward a more inclusive and globally applicable HS detection framework.

This review unveils several relevant insights: (1) Transformer models consistently outperform other methods, but their high computational requirements suggest that hybrid approaches, combining deep learning with traditional machine learning, may be more appropriate in certain contexts; (2) Although significant strides have been made in addressing low-resource languages, there is still a need for further work to improve inclusivity across a wider range of linguistic and cultural contexts; and (3) transparency and reproducibility remain critical challenges in the field, as the lack of publicly available code and datasets in many studies limits progress, hindering replication efforts and the development of generalizable models.

Looking ahead, we identify several key directions for future research. First, addressing algorithmic bias is imperative. Our review shows that despite advances in HS detection, bias mitigation remains under explored, especially for low-resource languages and marginalized communities. Future research should prioritize the development of fair and ethical models that avoid reinforcing societal inequalities. Second, there is a clear need for more standardized benchmarks and open-access resources. The difficulty of comparing results across studies due to inconsistencies in code and dataset availability is a major barrier to progress. Establishing common benchmarks, promoting data sharing, and ensuring transparency in methodology will be crucial in driving the field forward. Third, further exploration of emerging technologies such as multi-task learning and generative models could unlock new possibilities in HS detection. These techniques, which allow models to learn from multiple tasks simultaneously or generate more contextualized responses, may offer solutions to the inherent challenges of capturing the subtle and evolving nature of hate speech.

Our vision for the future of HS detection is one of interdisciplinary collaboration. As the scope of hate speech expands across different platforms and cultures, contributions from linguistics, computer science, ethics, and social sciences are essential to create holistic, reliable, and ethically sound solutions. We envision a future where HS detection systems are not only highly accurate but also transparent, fair, and adaptable to the needs of diverse online communities. By fostering such interdisciplinary efforts, we can ensure that HS detection tools contribute meaningfully to creating safer, more inclusive digital spaces.

One limitation of our review is its primary focus on journal articles, along with recent ACL papers and selected contributions from the OffensEval task at SemEval. This approach may have overlooked some cutting-edge research typically presented at conferences. While peer-reviewed journals provide a rigorous evaluation process, conferences are often hubs for the dissemination of innovative ideas and emerging trends. Consequently, the exclusion of a broader range of conference papers may have resulted in certain dimensions of the topic being underrepresented.

Although we decided to include only journal articles, we recognize that by excluding high impact peer-reviewed conferences we are limiting the inclusion of cutting-edge research, so in order to mitigate this side effect we decided to include the most relevant papers of two tasks held in the SemEval international workshops of 2019 and 2020 published in ACL. In these years’ editions the OffensEval task was held, aimed at detecting offensive language. By including the most relevant studies papers of a competition with a high degree of participation, we believe we get a glimpse of that time’s best techniques for the task. Additionally, to ensure comprehensive coverage of recent innovations, we extended our search to include ACL conference papers published between 2020 and 2024 that met our inclusion criteria, specifically selecting long papers from the main conference proceedings

In summary, this work provides a comprehensive overview of the current research landscape in HS detection, with a particular focus on the increasing impact of Transformer-based models. It highlights key insights, identifies gaps in the existing literature, and suggests directions for future research. We aim for this review to serve as a foundation for further progress in the field, equipping researchers to tackle the complex and evolving challenges of detecting online hate speech.