Social Network Analysis and Mining (2022) 12:30

[Link]

ORIGINAL ARTICLE

Characterizing the role of bots’ in polarized stance on social media

Abeer Aldayel1 · Walid Magdy1

Received: 16 May 2021 / Revised: 6 January 2022 / Accepted: 8 January 2022

© The Author(s) 2022

Abstract
There is a rising concern with social bots that imitate humans and manipulate opinions on social media. Current studies on
assessing the overall effect of bots on social media users mainly focus on evaluating the diffusion of discussions on social
networks by bots. Yet, these studies do not confirm the relationship between bots and users’ stances. This study fills in the
gap by analyzing if these bots are part of the signals that formulated social media users’ stances towards controversial top-
ics. We analyze users’ online interactions that are predictive to their stances and identify the bots within these interactions.
We applied our analysis on a dataset of more than 4000 Twitter users who expressed a stance on seven different topics. We
analyzed those users’ direct interactions and indirect exposures with more than 19 million accounts. We identify the bot
accounts for supporting/against stances, and compare them to other types of accounts, such as the accounts of influential and
famous users. Our analysis showed that bot interactions with users who had specific stances were minimal when compared to
the influential accounts. Nevertheless, we found that the presence of bots was still connected to users’ stances, especially in
an indirect manner, as users are exposed to the content of the bots they follow, rather than by directly interacting with them
by retweeting, mentioning, or replying.

Keywords Stance · Bots · Social media

1 Introduction commonly been used to manipulate views by posting con-

tent and interacting with real users (Mendoza et al. 2020;
Social media platforms are used as a primary source for Bessi and Ferrara 2016; Boichak et al. 2018). For example,
the dissemination of news (Center 2013). A vast number of these programs were used during the 2016 US presidential
users who depend on these platforms as their primary source campaign to manipulate discussions by spreading content
of information also use them as a medium to express their related to the US elections (Rizoiu et al. 2018; Ratkiewicz
stances on various events. A stance is defined as a person’s et al. 2011). In addition, in a recent study by Dunn et al.
attitude and view toward an entity or subject, either in sup- (2020), it shows that bots were used to spread fake news
port or opposition (AlDayel and Magdy 2019a). about Coronavirus (COVID19) in social media. All of these
Social media platforms are infested with social bots1 factors highlight the need to identify the role bots play in
(automated accounts) that mimic human behavior and can affecting the stance of social media users.
be used to spread inflammatory content with the aim of pro- There is no concrete method to analyze the role of bots
moting a specific view or stance (Shao et al. 2018; Bessi and in affecting the social media users’ stances (Garimella and
Ferrara 2016). West 2019; Pulido et al. 2018). Nevertheless, there have
Due to the prevalence of bots on social media, humans been several attempts to gauge the effect of bots on vari-
are not the only players on these platforms, and bots have ous events such as elections (Boichak et al. 2018; Santia
et al. 2019; Shao et al. 2018). The focus of these studies was
to evaluate content diffusion on social networks as a way
* Abeer Aldayel to measure the influence of bots on public stance towards
[Link]@[Link]
a topic. Most of these studies evaluated the spread of the
Walid Magdy misleading and false information by social bots to measure
[Link]@[Link]
the effect of these accounts on the discussion of an event
1
School of Informatics, University of Edinburgh, Edinburgh,
1
UK “bots” will be used henceforth to refer to “social bots”.

13
Vol.:(0123456789)
 30 Page 2 of 24 Social Network Analysis and Mining (2022) 12:30

(Tardelli et al.; Santia et al. 2019; Shao et al. 2018); for interactions between the bots and the users? Do users
instance, study by (Santia et al. 2019) and (Tardelli et al.) directly interact with bots by retweeting/replying, or
evaluated the spread of misleading content on Facebook by only by being exposed to their content by following bot
bots, and spreading misinformation related to financial top- accounts?
ics, respectively. Similarly, the work of Shao et al. (2018)
found that bots amplified the spread of fake news within a To this end, we proposed three hypotheses to assess bots’
10-month period between 2016 and 2017. Previous studies interplay with specific stances, compared to real accounts.
used the spread of bots on social networks as indicators of These hypotheses were derived from the induction phenom-
their effect on social media user’s stances. While this method enon (also known as social influence) (Friedkin and Johnsen
showed that bots are heavily present in social networks, there 1990), wherein an individual’s behavior is affected by their
are still limitations when it comes to identifying whether the social interactions (Gilani et al. 2019).
presence of bots is correlated with users’ stances towards
specific topics. H1 The presence of bots is an influential feature for predict-
In this paper, we seek to understand the interplay between ing stance in an online social network.
bots and support/against stances with respect to a given H2 For some topics, bots might have a presence in a certain
topic. We study bots’ role and define their connection with stance over the other.
stance interactions as the signals in the online social net- H3 The specific type of bot interaction moderates the rela-
work that can be predictive for stance towards a given topic tionship between bot presence and stance.
(AlDayel and Magdy 2019b). Our main hypothesis is that if
bots exist among the most influential features for predicting To answer our previous research questions and test the previ-
the user’s stance, then it can be inferred that these bots have ous hypotheses, we performed a large-scale analysis of bots
a role in pushing and/or reinforcing that stance. on Twitter. We focused on Twitter,2 as this platform has been
Our approach for measuring the relationship between bots shown to be one of the popular social media environments
and online stance interactions is to build stance-detection for sharing content and spreading news. Also, many stud-
models that are trained on users’ online social networks, then ies have indicated a substantial presence of bots on Twitter
analyze the presence of bots among the most important fea- (Abu-El-Rub and Mueen 2019; Stella et al. 2018), which
tures for predicting stance. Stance detection is a well-known makes this platform suitable for our study. Then, we built
task that has been used in multiple studies to infer a user’s a stance-detection model by using users’ interactions as the
leaning/ views towards an entity or topic. Many studies have main features to infer those whose stances were in favor of
used stance prediction to analyze the public view towards an and against a given topic. We used two types of online inter-
event or entity by analysing social media users interactions actions, direct interaction (IN) and Indirect Exposure (EXP).
(Magdy et al. 2016a; Darwish et al. 2017; Graells-Garrido The (EXP) interactions contain accounts collected from the
et al. 2020). user’s friends list (connection network (CN), while the direct
interactions (IN) include a set of retweets, mentions, and
1.1 Hypotheses and research questions replies in users’ tweets. By answering these research ques-
tions, we offer the following contributions:
Previous studies on the role and effect of bots on online
social networks (OSN) highlight the need to address human • We assessed the interaction behavior of social bots to
interactions to effectively differentiate between automated analyze their relationship with the against and in-favor
and real accounts (Abokhodair et al. 2015). of stance concerning specific topics.
In this paper, we investigate the following research • Most of the previous techniques for examining the effect
questions: of social bots evaluated synthetic datasets, and focused
on the diffusion of these accounts in social networks. In
RQ1 Do social bots have a presence among the most this work, we provide a more robust technique to analyze
influential network signals that can predict a user’s the connection between bots and polarized stance interac-
stance? tions3 by using a gold standard stance dataset that covers
RQ2 Is the interaction of social bots with users’ stances various domains.
similar on both those with supporting stances and those
with opposing stances? Or do they usually have a more
noticeable relationship in a particular direction? Does 2
We refer to Twitter as “social media” in this paper.
this change according to the topic? 3
Polarised stance refers to non-neutral stances. It does not necessar-
RQ3 How does the relationship between the presence ily imply extreme stance, but just being either against or in-favor of a
of bots and users’ stance change based on the type of given topic.

13
Social Network Analysis and Mining (2022) 12:30 Page 3 of 24 30

• We provide a comprehensive analysis of how users with Wikipedia edits bots “@EarthquakeBot”, which provides
specific stances were exposed to bot content through two updates about earthquakes that measure 5.0 or more on the
types of interactions: (1) direct interactions (IN) through Richter Scale, as they happen.4 Twitter has a clear policy
retweets, mentions, and replies to bots tweets, and (2) about the automation of accounts to regulate bots’ adoption
indirect interaction (EXP) by following bots and being on its platform.5 One of these rules is to prevent automated
exposed to their content. accounts from spamming the users or sending unsolicited
messages.
We applied our experiments on a dataset of more than 4000 Unfortunately, not all automated Twitter accounts (bots)
Twitter users who had expressed polarized stances towards got created for a noble cause. As will be discussed in the
seven different topics in multiple domains, including the next section, some bots get created to spread fake news
political, social, and religious spheres. We analyzed those (Shao et al. 2018) or to create campaigns against election
users’ online interactions and friendships with more than candidates (Bessi and Ferrara 2016), or to amplify specific
19 million accounts, among which we identified the bot stance on a topic (Stella et al. 2018). Thus, it became a cru-
accounts and the ways in which the users of specific stance cial task for many researchers to build methods to identify
(favor/against) interacted with these bots accounts. bots and measure their spread in social networks. While
Our findings showed that a relationship between social there are quite many studies in these directions, there is still
bot accounts and users’ stances does exist, but it is minimal a limited amount of work to gauge if their role on stance is
when compared to connections with human accounts, which of any effect. In this study, we fill in the gap by investigating
were more significantly tied to user stance. We also found the bots interplay with stance.
that the relationship between bots and user stance occurs
when users follow the bot accounts and are exposed to their 2.2 Bots’ role in online social networks
content; this effect was more apparent than the online signals
coming from those users who directly interacted with those Bots have been always created and used on social media
accounts through retweeting or replying. platforms for various good reasons, such as providing ser-
vices/information on given subjects (Wate 2021) or for
example, to help institutions with answering customer que-
2 Related work ries by text generating appropriate content (Seering et al.
2018). Nonetheless, recent research brought awareness of
This section provides a discussion of previous work in the issue of bots on social media, where these automated
inspecting bot’s effect on social media. Initially, we give accounts might usually employed for disruptive and mali-
some background on Twitter as a social media platform and cious reasons such as spreading misinformation and spam
its policy towards automated accounts, such as bots. Then, messages (Stella et al. 2018). This kind of bots have been
we show recent work on measuring the role of bots on the associated with the risks of manipulating public opinion by
spread of discussions on social media. Finally, we discuss artificially magnifying specific message on social media by
work related to inferring online signals on social media that retweeting specific message or spreading hateful posts (Luc-
are predictive of the stance towards a topic, which is our eri et al. 2019). Most of the previous studies assessed the
methodology’s primary instrument. effect of bots by analyzing the spread of these accounts on
social media as related to specific events (Aiello et al. 2012;
2.1 Twitter policy on bots Abokhodair et al. 2015; Bastos and Mercea 2019; Ferrara
2017). For example, a study conducted by Rizoiu et al.
Twitter is one of the largest online social networks (OSNs). (2018) used retweet diffusion to analyze the presence of bots
Users can easily create an account, which is public by in the first US presidential debate in 2016. They used syn-
default, then they can follow any other public accounts thetic data and generated an artificial social group of 1000
without their consent. Only protected accounts, which are users to model cascades of retweets diffusion and to calcu-
accounts that have their posts (tweets) seen only by their late users’ importance. Also, the study by (Ratkiewicz et al.)
followers, are the ones that need explicit approval to follow examined the diffusion of memes by bot account. In their
them. study they analyzed political memes in Twitter and focus on
Unlike many of the social media platforms, Twitter allows detect astroturfing campaigns in the context of U.S. political
accounts to post tweets automatically. This motivated many
users and/or institutions to create bots, which are accounts
that generate its content automatically and interacts on Twit- 4
More examples of interesting and creative online bots are available
ter based on predefined rules (Seering et al. 2018). Many on Botwiki: [Link]
bots accounts are created for useful causes, such as the 5
[Link]

13
 30 Page 4 of 24 Social Network Analysis and Mining (2022) 12:30

elections. The work of (Hegelich and Janetzko 2016) investi- of bot services on the Twitch platform; in this study, they
gated bot activity in the Ukrainian-Russian conflict and con- limited their analysis to the service bots provided for Twitch
cluded that autonomous bot behavior helped spread content. users. Another study analyzed the role of bots on Wikipedia
A study analyzed the spread of bots in discussions related and studied the editing behavior thereof and the effect on
to the Syrian civil war by using 3000 tweets related to the human editors; they found that the overall human interac-
topic (Abokhodair et al. 2015). They found that the growth tions with bots were more in comparison with bots, com-
and content of botnets did not aligned with the bots main pared to human-human interactions.
behaviour as these bots were spamming the hashtags with Most of the previous studies examined the effects of
topics not related to war. Another study by Bastos and Mer- social bots by measuring their presence and the spread of
cea (2019) analyzed the bots behavior in Brexit discourse on their content on social networks. However, there is a gap in
Twitter (Bastos and Mercea 2019). In their study, they used the literature to understand if the spread of these bots has a
retweets to inspect user-to-bot and bot-to-bot cascade com- presence within the signals that predict users’ stances. Our
position. They found that a botnet spread content supporting study extended the efforts to assess the relationship between
the “Leave” campaign. The study of (Stella et al. 2018) bots and users’ stance on social media by assessing the inter-
evaluated the role of bots in spreading negative content play between bots and users’ stances. Moreover, in contrast
according to social media data. In their study, they collected to previous studies, we provide a fine granularity analysis of
data related to the 2017 Catalan referendum and analyzed bots on polarized stances (i.e., against or in favor). We uti-
the diffusion of negative content by bots. They used Logistic lized the advances in stance-detection models using network
Regression (LR) along with accounts metadata to identify features to measure bots’ presence in the signals that are
bots accounts. Their results showed that bots increased the predictive for stance. Our novel approach states: the more
exposure to negative content. Along the same line, the study bots that are present among the top predictive features for a
by (Luceri et al. 2019) estimated the stance of bots on social specific stance, the stronger the relation between the pres-
media according to the content they spread. A recent work ence of bots and a given stance of users on the topic.
by (Mendoza et al. 2020) focused on retweet only and ana-
lyze the presence of bots. Their findings show that there are
different degrees of integration with different types of users. 2.3 Online factors influencing the stance prediction
Another study by (Ng and Carley 2021) studied the chang- on Twitter
ing stance on a specific topic (coronavirus vaccine) using
Twitter data from April 2020 to May 2021. In their study, A large body of work has analyzed the stance towards a
they found that a larger proportion of bots are more prone to topic on Twitter. Most of the work in this realm studied
flipping their stances compared to non-bot accounts, where public stances in the aftermath of an event and analyzed
these normal accounts did not change their stance. the interplay between these stances and other online fac-
Another study by Abu-El-Rub and Mueen (2019) ana- tors; for example, the work of (Allcott and Gentzkow 2017)
lyzed bot behavior in social media related to the US election showed the interplay between polarized stances and the shar-
and quantified the level of bots and human participation in ing behavior of unreliable news content. Another work by
social campaigns. By analyzing the retweets network, they Graells-Garrido et al. (2020) showed the temporal effect of
found that bots’ interactions can corrupt social campaigns. online interactions on discussions related to abortion. The
Also, (Schuchard et al. 2019) examined bots’ activities on work by (Musco et al. 2018) examined the trade-off between
twitter concerning the US 2016 elections and concluded disagreement and polarization using synthetic along with
that bots tend to have a hyper-social nature. Along the same Riddet data. The study demonstrates that overall users pre-
lines, (Gilani et al. 2019) provided a comparison between fer links that minimize disagreement due to the well-known
bots and human behavior with a focus on online activity. confirmation bias. Moreover, the study by AlDayel and
They used manual annotations to label the accounts as a Magdy (2019b) shows that user’s interactions and connec-
bot or not. In their study, they showed that humans have a tion networks had the best performance when it came to pre-
higher follower rate compared to bots. On the same line, dicting the user’s stances, compared to textual features; they
a study by ‘(Aiello et al. 2012) used bots accounts to sug- built a stance-detection model using three different kinds of
gest friendship to Twitter users. They finding indicates the the networks: an interaction network, a preference network,
importance of trustworthiness as in general people are aware and a connection network. Their findings highlighted that a
of the presence of the bot have been more inclined to follow user’s online interactions or connections with other accounts
its suggestions. were effective signals to predict their stances towards dif-
Another line of studies analyzed bots behavior on differ- ferent topics. A recent survey (AlDayel and Magdy 2021)
ent kinds of platforms, such as Twitch and Wikipedia. For provides a thorough overview of the most predictive features
example, (Seering et al. 2018) analyzed the social actions for stance detection in social media. It lists multiple studies

13
Social Network Analysis and Mining (2022) 12:30 Page 5 of 24 30

Table 1  Sample tweets from # Tweet Topic Stance

each dataset
1 Those of us that also have a brain will be voting against Hillary HC Against
2 The carbon clock is ticking @CarbonBubble #SurplusGas #carbon #gas CC Favor
3 Close the border with military or any means possible. If we don’t STOP IT IM Against
NOW, it will over run our country to the point of no return. We have to
send a strong message and not back down!
4 Yes it is. But how long do we want to play this game? Enough is enough. B Favor
#Brexit means Brexit, nobody voted for a deal, and so on and so forth. The
UK just needs to leave on 29th of March.

Topics: hillary clinton (HC), climate change is a real concern (CC), immigration (IM), and brexit (B)

that confirm the effectiveness of network interactions as a (B) in February 2019 using the keyword “Brexit.” For the
predictor of users’ stance in social media. topic of Immigration, we collected the tweets in October
2018 using the following keywords: “immigrant,” “refugee,”
and “border.” Tweets of both topics were submitted to the
3 Data collection crowd-sourcing platform Appen.6 We followed the same
annotation guidelines used to construct the SemEval stance
To examine the role of bots on users’ stances, we utilized dataset (Mohammad et al. 2016), and each tweet was anno-
datasets that contain ground truth labels for stances on tated as “favor,” “against,” by five annotators while taking
seven topics. This section provides a description of the data- the majority vote as the final label for each tweet. For each
sets used and explains the process of constructing users’ topic we created around 100 quality control test instances
networks. (labels for a given tweet) to verify the annotators ability of
assigning the correct labels.7 These test instances with pre-
3.1 Stance‑detection datasets answered labels are used to further qualify high-performing
contributors, remove under-performing ones, and continu-
We used two datasets that contain tweets that are labeled for ally training contributors to improve their understanding of
stances towards seven topics. These datasets are: the task. Furthermore, we choose the high qualified anno-
SemEval stance dataset. We chose this dataset because tators in Appen and accept Level 3 annotators for the task
it is considered to be one of the most well-known stance to ensures only our most experienced and highest perform-
dataset that covers topics from different domains. A recent ers will work on your task. The inter-annotator agreement
study by (AlDayel and Magdy 2019b) used SemEval stance between the annotators for Brexit was 73%, and the score
data to provide a thorough assessment of social media online for Immigration was 75%; these scores demonstrate a high
signals that affect stance prediction. This dataset contains level of agreement between the annotators, which indicates
tweets related to five topics: Hillary Clinton (HC), Cli- that different annotators frequently gave the same response
mate Change is a real concern (CC), the Feminist Move- (stance) for the same tweet.
ment (FM), Legalization of Abortion (LA), and Atheism Table 1 shows sample of the tweets in our data collection
(A) (Mohammad et al. 2016). The dataset is publicly avail- along with their stance labels to the corresponding topic.
able and consists of 4,163 tweets collected in 2016 that are
labeled with three classes of stances: in-favor, against, and 3.2 Collecting users’ online interactions
either. Since the focus of this study was to investigate the
relationship between social bots and users’ stances, we used For each tweet in our dataset, we collected all the interac-
the “Users Dataset,” a subset of the SemEval dataset that tions information for its author. For each user, we collected
was published by (AlDayel and Magdy 2019b). This dataset two types of online interactions, as defined by the work
contains 2,875 tweets, which are a subset of tweets from user of (AlDayel and Magdy 2019b). The first is IN@, which
accounts that are still available on Twitter. is the interaction network of the user that includes all the
Events dataset. We created an additional dataset that accounts the user retweet, mention or reply; and the other
covered two recent topics: Brexit (B) and Immigration (I).
These topics were selected because they were one of the
viral events at the time we were collecting data. The tweets 6
Previously know as Figure Eight and CrowdFlower. [Link]
in this dataset were all selected to be replies to other tweets com/.
to have a higher chance of showing a polarized stance as 7
[Link]
being part of a discussion. We collected tweets on Brexit Manag​ement.

13
 30 Page 6 of 24 Social Network Analysis and Mining (2022) 12:30

Table 2  The number of tweets Dataset Topic Tweets (users) IN@ CNFR
per topic in the SemEval and
Events datasets with the number SemEval Atheism (A) 550 (426) 608,399 740,878
of unique users who authored
Climate change (CC) 461 (381) 560,629 524,591
the tweets shown in brackets
Hillary clinton (HC) 670 (511) 1,151,355 1,217,426
Feminist movement (FM) 524(441) 657,411 371,700
Legalization of abortion (LA) 670 (490) 978,300 938,184
Events Brexit (B) 466 (466) 2,129,244 656,864
Immigrations (I) 1512 (1512) 5,567,226 3,274,835
Total 4853 (4227) 11,652,564 7,724,478

The total number of accounts users interacted with (IN@) and followed (CNFR) for each topic

is CNFR , which is the connection network of the user that inspecting bots’ presence within the most influential fea-
includes the list of accounts the user follows. We used Twit- tures. This section discusses our framework, which includes
ter API to collect users’ timelines, which included all the building an effective stance classifier, extracting the most
tweets they posted or retweeted in their home-timeline. 8 predictive features for a given stance, and identifying the
From the timeline, we extracted all the accounts that the bots among the accounts.
user retweeted, replied, or mentioned to represent the IN@.
We also collected the friends list (i.e., the accounts the user 4.1 Stance detection classifier
follows) using Twitter API .9
Table 11 shows all statistics related to our datasets, Stance detection is a well-known task that has been used
including the number of tweets and users for each of the to infer a user’s leaning towards an entity or topic (Magdy
seven topics, which are labeled according to stance, and the et al. 2016a; Darwish et al. 2017; Samih and Darwish 2020).
number of collected accounts in the interaction and con- For example, in their study, Grčar et al. (2017) built a
nection networks. As shown in the table, the total number stance classifier to analyze the stances toward the Brexit
of accounts collected for all the users in our datasets was referendum using 37,000 tweets. Another study by Magdy
more than 19 million accounts, which means that on aver- et al. (2016a) analyzed the stances toward Muslims in the
age, each user interacted and/or connected with more than wake of the November 2015 Paris terrorist attacks. Moreo-
4000 accounts in total. The median number of accounts the ver, (Magdy et al. 2016b) used a stance classifier to analyze
user interacts with (IN) is 1,288 (average = 2,532), and the the supporters of Hillary Clinton and Donald Trump in 2016
median number of accounts the user follows (CNFR ) is 602 US elections.
(average = 2,101). A significant amount of work has studied stance detec-
Our aim in this study is to identify which of those tion by using network interactions to predict people’s lean-
accounts are bots and to understand which of those are ings towards an event or entity (Thonet et al. 2017; Darwish
shown to have predictive features for specific stances; in et al. 2019). The study by Lai et al. (2018) used a social
this way, we can explore the relationship between bots and network community based on retweets, quotes, and replies
user stances online (Table 2). to extract network-based features to train a stance-detection
model. Similarly, a study by Darwish et al. (2019) designed
an unsupervised stance-prediction model using a clustering
4 Assessing the role of social bots algorithm with a combination of network features, namely
retweeted tweets, retweeted accounts, and hashtags. These
This section describes the methodological framework that studies highlight the importance of the social network inter-
examined the connection between bots and users’ stances actions of online users to detect their views in support of or
in social media. As mentioned earlier, our methodology for opposed to specific topics.
measuring the relationship between bots and users’ stances The first step in our methodology was to build a stance
is by building a stance classifier using network features and classifier that classifies a given user’s stance as being in
favor of or against a given topic.
To create an effective stance classifier, we replicated
8
the current state-of-the-art stance-detection model devised
[Link]
ence/​get-​statu​ses-​home_​timel​ine. by AlDayel and Magdy (2019b), which reported the best
9
[Link] results-to our knowledge-on the SemEval stance dataset. We
search-​get-​users/​api-​refer​ence/​get-​frien​ds-​list. used a binary SVM with a linear kernel, and the parameters

13
Social Network Analysis and Mining (2022) 12:30 Page 7 of 24 30

Table 3  The average F1-score Topic A CC HC FM LA B I

for stance detection on the seven
topics in our two datasets IN 71.9 48.2 71.8 61.2 70.3 47.6 55.8
EXP 68.05 48.21 72.98 66.0 66.42 69.2 49.00

were tuned using fivefold cross-validation on the training set. F1-score (Mohammad et al. 2016) (Appendix A). The macro
In the study by AlDayel and Magdy (2019b), they showed Favg is calculated as presented in Eq. 1.
that a binary classifier that is trained on the two classes of
Ffavor + Fagainst
“in favor” and “against” while ignoring the “neither” class Favg = (1)
achieved a better performance than a three-class classifier. 2
This setup was ideal for our purpose, since we were only The official evaluation measure used by Semeval2016 does
focusing on the roles of bots on influencing stance and were not disregard the ‘none’ class. As by taking the average
thus not interested in the “neither” class ; consequently, we F-score for only the ‘favor’ and ‘against’ classes, we treat
followed this same setup. A stance-detection model was ‘none’ as a class that is not of interest or ‘negative’ class
trained for each topic separately, which means we trained in Information Retrieval (IR) terms. Falsely labeling nega-
seven different models for each of our seven topics. tive class instances still adversely affects the scores of this
Regarding the features, AlDayel and Magdy (2019b) metric.
compared multiple sets of features, including content fea- Table 3 shows the performance on the seven topics
tures that are extracted from the text of tweets, different reported in the F1-score using the script provided by the
online interactions features of users’ accounts, and even SemEval task (Mohammad et al. 2016). As a validation for
the web domains that are linked in users’ tweets. To serve the model quality, our results for the five topics of SemE-
the purpose of our analysis, we focused on online network val aligns well with those reported in AlDayel and Magdy
features; namely the interaction (IN) and connection (CN) (2019b), which confirms that we succeeded in replicating
network of users, both of which achieved the highest perfor- the state-of-the-art stance detection model.10
mance on the SemEVal dataset (AlDayel and Magdy 2019b).
We trained the stance detection model on two sets of fea- 4.2 Extracting the most influential features
tures, and we refer to each of these as follows: on stance
• IN, which included all the accounts with which each user To assess the extent of the relationship between bots and
directly interacts through retweets, replies, or mentions; users’ stances, we analyze the most effective features for the
also includes the website domains the users included in stance prediction model. For each polarized stance (favor
their tweets(AlDayel and Magdy 2019b). or against), we use the weight of the coefficient generated
• EXP, which corresponded to CNFR in (AlDayel and by the stance model to identify the set of the most influ-
Magdy 2019b), and included the list of the accounts that ential features on the stance prediction. These features are
the user followed. We called it EXP, since it represented extracted from the features set which contain the accounts
the accounts the user was exposed to by following them, and the domains (URLs) the user interacted within the (IN)
and were thus affected by their content, even without feature set, and the accounts the user follows in the (EXP)
directly interacting with the content thereof by liking, feature set. We use the top 1000 most influential accounts for
replying, or retweeting. the stance prediction model from each feature set, exclud-
ing the domains from the IN features in our analysis, since
For each topic, the labeled datasets were split into 70% our focus in this study is on bots’ connection to user stance.
for training and 30% for testing to build the stance classi- In the next section, we inspect the population of bot
fier. This is the same split reported by the SemEval stance accounts that exist in those 1000 most predictive accounts
dataset (Mohammad et al. 2016). For the Events dataset, for user’s stance, and compare their population to other
we applied a random split to training and testing with the accounts.
same split percentage-70% and 30%- for training and testing,
respectively. The stance classifier was separately trained for
each topic twice: once by using the IN features of each user,
and another by using the EXP features thereof. For evaluat-
10
ing the classification performance, we used the SemEval Since our focus in this study is to provide a through analysis of
the interplay between stance and bot advised by the most well-known
stance-detection official evaluation script to calculate the benchmark stance dataset we kept the Climate change topic in our
analysis even when it has the lowest F1 score.

13
 30 Page 8 of 24 Social Network Analysis and Mining (2022) 12:30

Fig. 1  Botometer score distribu-

tion of the top 1000 accounts
that are predictive to stance for
both networks

4.3 Identification the presence of bot accounts Sometimes the Botometer API generates an error mes-
sage. This happened when it failed to access the tweets of
In the next section, we inspect the population of bot accounts an account because it is deleted, suspended, or protected.
that exist in the 1000 most predictive accounts for a user’s We considered protected accounts to be human accounts,
stance, and compare this population to other accounts. since it was unlikely that a bot would restrict its tweets to
There is a large body of work focused on the develop- only its followers (Rizoiu et al. 2018). While suspended
ment of techniques to detect bots in social media (Davis accounts could be suspended because they were bots, Twit-
et al. 2016; Puertas et al. 2019; Santia et al. 2019). The ter can also suspend an account if the user of that account
work of (Puertas et al. 2019) used a multilingual classifica- violates the platform rules.12 Common reasons to suspend
tion model to identify bot accounts based on the content a Twitter account includes abusive tweets, spamming, or if
of their posts. Another work by Dutta et al. (2021) focused the account has been hacked or compromised. For the previ-
on the fraudulent user activities where they proposed a ous reasons, we did not consider the suspended accounts in
framework to detect collusive users involved in ’follow- our dataset to be bots; instead, we treated these accounts as
ing’ activities. Similarly, the work by Cresci et al. (2015) “unknown” and labeled them as “deleted.” Therefore, the
introduced a framework to detect Fake followers on Twitter deleted and suspended accounts in our dataset were labelled
only. One of the most popular bot-detection APIs is Botom- as “deleted”.
eter11 (Davis et al. 2016; Yang et al. 2019), which provides a For accounts that have a low botometer score, which are
robust method to detect the existence of bots in social media. most likely to be non-bots (i.e., human accounts), we wanted
Botometer uses a Random Forest classification algorithm to make a distinction between famous and normal accounts,
to classify tweets as bots based on 1000 features that were since it might be expected that influential accounts will
extracted from users’ meta data along with tweets timeline. have a more prominent relationship with users’ stances than
The classification score ranges from 0 to 1, where 0 indicates normal accounts. According to the research conducted by
the likelihood that the account is human and 1 indicates that Cossu et al. (2016), the authors postulated that an account
the account is likely to be non-human (“bot”). was more influential when it had more followers.
The Botometer API has been used in various studies to Thus, we further classified the non-bot accounts accord-
detect the existence of bots in the social media (Rizoiu et al. ing to the number of followers thereof into three categories:
2018; Varol et al. 2017; Broniatowski et al. 2018). In the ultra-famous, famous, and normal.
study conducted by Rizoiu et al. (2018), the Botometer API According to Twitter users statistics, 13 only 0.05% of
was used to analyze the role and influence of bots on social Twitter accounts have more than 10,000 followers; thus
media in the 2016 US Presidential Debate. Another study by we label them as ultra-famous; the famous accounts are
Broniatowski et al. (2018) estimated the bot scores of Twit- those with a number of followers ranging between 1000
ter accounts that spread content related to the vaccine debate and 10,000, and which applies to 2% of Twitter users; and
on social media. Along these lines, we used Botometer in finally, the normal accounts were those with fewer than 1000
our study to detect the bots in our dataset. followers, which applies to 98% of Twitter users.
To identify the bots in the set of predictive accounts
extracted from the stance-detection model, we used the Bot-
ometer API (Davis et al. 2016). This API generates a score
∈ [0, 1], where 0 indicates the account of a real user, and 1
suggests the strong likelihood of a bot.

12
[Link]
twitt​er-​accou​nts.
11 13
[Link] [Link]

13
Social Network Analysis and Mining (2022) 12:30 Page 9 of 24 30

Fig. 2  Distribution of social

bots for each topic in the top
1000 most predictive accounts
for polarized stances using
direct interaction (IN) and indi-
rect exposure (EXP) features.
Atheism (A), Climate Change
Is a Real Concern (CC), Hillary
Clinton (HC), the Feminist
Movement (FM), Legalization
of Abortion (LA), Immigration
(I), and Brexit (B)

5 Results and analysis and improved the overall bot-detection accuracy (Rizoiu
et al. 2018). We followed the same setup and used the same
In this section we assess the role of the social bots in detect- threshold.15
ing online stance by analyzing the top influential accounts The next section provides a further analysis of these
that were the most predictive toward the polarized stances accounts that are likely to be bots given their high bot score
with regard to each topic. and compare them to human accounts (accounts with low
Botometer score) and deleted accounts.
5.1 The distribution of bot scores of the most
influential accounts 5.2 The role of social bots on stances

Figure 1 shows the distribution of Botometer scores for the For each type of feature (i.e., IN and EXP), we show the
top 1000 accounts that are most predictive for stance in our percentage of likely-bot accounts alongside other types of
dataset, for both the interaction and exposure network fea- accounts for each topic.
tures (IN and EXP).14 The scores generated by Botometer Direct interactions (IN). Figure 2 shows the percentage
are within a range ∈ [0, 1], where 0 indicates the account of of social bots on the in-favor or against stances with respect
a real user, and 1 suggests the strong likelihood of being bot. to the top 1000 IN features for each topic. The results show
As shown in Fig. 1, most of the accounts that are predic- the prevalence of real accounts in the set of most influential
tive to stance have a low bot score, where the majority of features on the stances, compared to the minuscule exist-
accounts have scores between [0, 0.2], indicating that these ence of social-bot accounts. This trend is consistent in each
accounts are most like to be real people. Only very few topic with respect to both in-favor and against stances. The
accounts have high bot scores ( more than 0.6). This indi- bots had an existence not exceeding 6% of the overall set of
cates that most of the accounts that have a role in predicting influential accounts in each topic, while non-bot accounts
users’ stance are for real people. constituted the majority of the most influential accounts,
To enable a more in-depth analysis for accounts that are reaching higher than 95% for some topics. As shown in
more likely to be bots, we focused on those accounts that Fig. 2a, some of the accounts in the top 1000 were deleted
got a score of over 0.6, which indicates high likelyhood of by the time we inspected them, especially those related to
being a bot, which is the same score used in previous studies the SemEval stance dataset topics, since the data was more
to analyse bot behaviour (Rizoiu et al. 2018; Ferrara 2019). than four years old. This is one of the limitations of work-
In these studies, an account was classified as a bot when ing with Twitter, since we cannot retrieve information from
the Botometer score exceeded a threshold of 0.6, where
they showed that this score decreased misclassification
15
We also examined a threshold of 0.5, but found that it increased
misclassification of the human accounts as bots without improving
14
Appendix B shows the Botometer scores distribution on topic level the detection of new bots.

13
 30 Page 10 of 24 Social Network Analysis and Mining (2022) 12:30

Fig. 3  The percentage of each

account type (X-axis) in the top
N (Y-axis) important features
of stance detection to predict-
ing the Against/Favor stances
in direct interactions (IN) and
indirect interactions (EXP)

those accounts after deletion. This kind of limitation is well atheism topics). This suggests that being exposed to bots’
known in the online social network studies (Boichak et al. posts might be more strongly tied to users’ stance than
2018; Ferrara 2017). Nevertheless, we still had the tweets direct interactions with bots.
wherein these accounts were mentioned in the collected Furthermore, these bots that people follow have a
users’ timelines, which allows us to provide further analysis stronger connection to the against stance of some of the
to these accounts. topics compared to the favor stance. For instance, users
Indirect exposure (EXP). Figure 2b illustrates the per- with against stance towards atheism tend to be affected by
centage of bots in the favor or against stance with respect bots accounts more than users supporting Atheism (12%
to the top 1000 EXP features for each topic. Again, the per- against vs. 8% in favor).
centage of bots is minimal compared to human accounts. The same trend can be seen in climate change and
However, it is worth noting that bots constitute more immigration topics. This demonstrates that bots can have
population in the EXP network compared to IN, where it larger relationship to one stance direction compared to
reaches 12% and 13% in some cases (Climate change and the other.

13
Social Network Analysis and Mining (2022) 12:30 Page 11 of 24 30

5.3 Magnitude of the bots’ role Appendix D shows the full values of the chi-squared test per
topic and stance. This result confirms our third hypothesis
The previous analysis showed that the majority of the top (H3) that people stances can be affected indirectly just by
1000 predictive accounts of predicting stance for all the top- getting exposed to bots content, as we actually showed that
ics was for humans. However, a small proportion of bots EXP network affects users’ stance more than IN network.
might still be the most influential among those 1000. Thus,
in this section, we provide a more in-depth analysis to the 5.4 Properties of the bot accounts
distribution of bots in the top N predictive accounts, where
N ranged between 10 and 1000. In addition, we analyze the We further analyze the properties of the bots identified
type of human accounts according to how famous they are. within the most predictive accounts of users’ stance, where
Figure 3 illustrates the average distribution of different we check their types and the number of followers they have
types of accounts on each polarized stance extracted from in comparison to the human accounts.
the IN and EXP networks averaged across all topics. We Types of the influential bots. The new version of Bot-
noticed that the distribution of bot accounts constituted the ometer API (V4) provides the type of bot based on of six
lowest percentage across all values of top N features. In fact, categories.16 These categories are: Astroturf, Fake follower,
the average distribution of bots never exceeded 10% at any Financial, Self declared, Spammer and Other (miscellane-
point. This was consistent across both networks and for both ous). The Astroturf bots are political bots and accounts
stances. The ultra-famous accounts, which were the human involved in follow trains that systematically delete content.
accounts with more than 10,000 followers, constituted the The Fake follower bots are bots purchased to increase fol-
majority of accounts; they consistently constituted more than lower counts. Financial bots are the automated accounts that
50% across all values of N over all stances and networks, post using cashtags. The Self declared are bots labeled using
and their influence reached over 70% in the top 10 features [Link], which is a website that keep track of useful and
for the EXP against stance. This means that following these creative bot accounts that self-declare themselves as bots.
accounts related to users’ stances being against a given topic. While the Spammer are automated accounts labeled as spam
Further detailed results for each of our seven topics are bots from several datasets. Bots labeled as ’Other’ are the
presented in Appendix C, where we show variations across miscellaneous other bots obtained from manual annotation
some of the topics. For example, for Brexit, the ultra-famous or reported by other users.
users had a noticeable connection to the in-favor stance, We used Botometer V4 to analyse the types of bots we
reaching approximately 75% of the top 100 accounts, while identified in the influential accounts. Figure 4 shows the dis-
the ultra-famous users only constituted 25% in the against tribution of bots types (IN) and (EXP) networks. It can be
stance. Moreover, for the against stance in EXP interactions, noticed that the most dominating bots are the bots of type
these accounts showed a sizable presence in the top 100 Others, that are obtained from manual annotation and user
features for six topics: atheism, climate change, the feminist feedback. The Astroturf bots constitutes most of bots in the
movement, Hillary Clinton, immigration, and Brexit. For the direct interactions networks that influence the against stance
legalization of abortion, the normal accounts with fewer than towards immigration and favor stance towards Brexit. None-
1000 followers had the most presence in the top 800 features theless, Astroturf bots shown to have the minimal presence
of people who were opposed to the legalization of abortion. in the indirect interactions (EXP). Bots that are identified as
Our analysis shows that bots have some role in relation- Fake followers have good presence in most topics, especially
ship to online stance of being in-favor/against a topic, how- the exposure network. Overall, spammer bots constitute the
ever to a much smaller degree than what we expected in our minimal percentages over direct and indirect interactions,
first hypothesis (H1). The relationship is minimal compared while Financial bots have no presence in the most influential
to that with human accounts, especially the ultra-famous accounts. These results show that bots that have an interplay
accounts, which had the most association with users’ stances role with stance are mostly the ones that got reported by
by far. We also found that bot accounts that people follow normal users, while political bots (astroturf) still have some
and are exposed to their content (EXP) has more influence role, especially in stances on political topics.
( presence in the top features), than bots with which users Followers of influential bots. As our analysis has
directly interacted. We applied a statistical significant test shown, ultra-famous accounts are the most influential in
using Pearson’s chi-squared test between the distribution of predicting stances. Thus, we further analyzed the bots’
bot accounts in the IN and EXP and found that bots pres- number of followers to understand to compare them to
ence in the EXP network is statically significantly higher the influential human accounts. Table 4 shows the split
than IN for all stances in most of the topics with p-value less
than 0.001, except the Brexit and immigration topics, where
both had the least number of bots (only 2–3%). Table 10 in 16
[Link]

13
 30 Page 12 of 24 Social Network Analysis and Mining (2022) 12:30

Fig. 4  Distribution of social

bots types for each topic in
the top 1000 most predictive
accounts for polarised stances
using direct interaction (IN)
and indirect interaction (EXP).
Atheism (A), Climate change
is a real concern (CC), Hillary
Clinton (HC), Feminist move-
ment (FM), Legalization of
abortion (LA), Immigration (I),
Brexit (B)

Table 4  Distribution of bots and human based on followers. The Ultra-famous accounts more than 10,000 followers; The famous accounts are
those with number of followers ranging between 10,000 and 1000; The normal accounts less than 1000 followers
IN
Account type Favour Against
Bots Humans Bots Humans

Normal 57.73% 16.52% 51.57% 12.53%

Famous 19.58% 16.70% 14.73% 12.55%
Ultra-Famous 22.68% 66.77% 33.68% 74.90%
EXP
Account type Favour Against
Bots Humans Bots Humans

Normal 63.39% 21.71% 59.25% 20.63%

Famous 22.00% 16.66% 20.07% 14.68%
Ultra-Famous 14.59% 61.61% 20.66% 64.68%

of the identified bot accounts by their number of follow- 5.5 The context of the bots
ers as normal, famous, and ultra-famous. As shown, the
majority of bots (50–60%) have less than 1000 follow- In this section, we analyze the context of interactions
ers. However, around 30–40% of them still have a large between bots and users for each stance. We explore some
number of followers, especially for those bots influential examples of the bots appeared in users’ timelines to under-
to the against stance in the IN, around 34% of them are stand the possible link between the bots and users’ stances.
considered ultra-famous accounts by having over 10,000 We also check the nature of some of the bots that the users
followers. This shows that some of the bots that are popu- followed to estimate the exposure that might have affected
lar on Twitter. their stance.

13
Social Network Analysis and Mining (2022) 12:30 Page 13 of 24 30

Table 5  Sample of tweets and the context of social bot interactions in relation to stance and topic
# T Stance Example tweet

1 A Against RT @FollowDMs: follow everyone who retweets this

2 A Against RT @JesusNarrowWay: 1 Peter 4:18, If it is hard for the righteous to be saved, what will become of the ungodly and the
sinner?
3 HC Against RT @VoteHillary2016: donald, are you talking about the 70K votes we lost in 3 states or the nearly 3 million popular votes
you lost despite
4 FM Against ’So @ForgetFeminism according to this...99.99% of the feminists I talk to are NOT “feminists”.ll let them
know.#WomenAgainstFeminism’
5 A Favor RT @BibleWisdoms: There’s one Lord, one faith, one baptism, and one God and Father of all - Ephesians 4:5-6
6 HC Favor RightOn! @Timoniumbill: @ReadyForHillary Mrs. Clean. [Link] #OhHillNo #WakeUpAmerica
#StopHillary2016
7 I Against RT @cookequipman1: AMERICA’S VET TRAIN #ConnectingAmericanVets #MAGAveteran

Table 6  Top bot accounts in T Favor Against

indirect interactions for each
stance towards the seven topics A HaginQuotes, RCSproul, warpawsiraq 2ayaat , lilxstyles, RTAL_D3OAH
CC Smartassy4ever, jtd_gameon12, bigboater88 AIIAmericanGirI, SassyCon, Moonbattery1
HC WhatHillaryAte, bluenationuntd, stylebysassys saynotogop, humoryoulike, UniteBlueSC
FM geekfeminism, onlyminionquote, tomily4 stopbrutality, FeministShit, SC2TopReplays
LA succesfultips1, JohnGaltTCMC, SMNW_YRC​ TheKeyisPrayer, prolife321, myjesus123
B UKPollingLive, watching_eu, Brexit_WestMids moggality, mosthauntedlive, britainsmilhist
I RealBarcaBooks, RomanCatholic36, umustknowthis1 milagrovargas14, fridayfeeiing,mrmarkel

The direct interaction with bots. Table 5 presents a accounts that promoted religious content, such as “@Jesu-
sample of the tweets generated by the bots that were the sNarrowWay”, (see Example 2).
most predictive of the stance in the IN features. We found The indirect exposure to bots. We also analyzed bots
these bots in the users’ timelines as retweets, replies, or accounts that were the most predictive of the stance in the
mentions. In general, bot interactions with social media EXP features, which were the bots that the users in our data-
users have three forms: 1) bots with content that aligned sets were following. Table6 presents the top three bots that
with the user’s stance; 2) bots with content that disagreed influenced the against and in-favor stances for each topic.
with the user’s stance; and 3) bots with content that had no In general, social media users tended to follow bots that
relation to the user’s stance. For instance, the bot account aligned with their leanings. For example, people with an
“@FollowDMs” was one of the influential accounts in against stance toward atheism tended to follow automated
predicting the against stance to atheism, yet this account accounts with religious content, such as “@2ayaat” and
had no relation to the topics of atheism or religion (see “@RTAL_D3OAH.” The same observation was made for
Example 1). Additionally, regarding bot accounts that had users who supported Hillary, who tended to follow accounts
no direct relation to immigration, the “@cookequipmant” that confirmed their leanings, such as “@WhatHillary-
account was one of the most influential accounts for pre- Ate”; this account is an automated account with retweets
dicting the stances related to the topic of immigration (see and tweets that amplifies content that support to Hillary.
Example 7). Furthermore, Example 5 shows that users Additionally, users with stances that supported the feminist
who supported atheism tended to directly interact with movement followed accounts that promoted the feminist
bot accounts that contradicted their stance. For example, movement, such as “@geekfeminism”. As it relates to Brexit
one of the influential bot accounts in predicting the in- supporters, accounts such as “@Brexit_WestMids ”, which
favor stance towards atheism was the one that promoted promoted Brexit through tweets and retweets, was one of
religious content, “@BibleWisdoms”. Additionally, Exam- the most effective accounts in the friends list to predict the
ple 3 shows that the account “@VoteHillary2016” was a in-favor stance towards Brexit. It is worth noting that, in
bot account that supported Hillary Clinton, yet it was one general, the most effective bots for stance prediction had no
of the accounts that had a strong effect on predicting the direct relation to the topic related to the stance. This can be
against stance to Hillary Clinton. Moreover, users with seen in the top three bots that interacted with users who held
an against stance to atheism interacted with religious bot against stance toward climate change; these accounts tended

13
 30 Page 14 of 24 Social Network Analysis and Mining (2022) 12:30

Table 7  The number of deleted T Favor Against

accounts and the expected
bots in the top 100 influential Deleted Deleted bots Existing bots Total bots Deleted Deleted bots Existing bots total
accounts on stance prediction
A 8 2 3 5 20 8 0 8
CC 8 2 2 4 15 0 0 0
HC 33 8 1 9 20 2 2 4
FM 23 1 1 2 27 2 5 7
LA 25 4 5 9 10 3 5 8
B 12 5 2 7 9 2 5 7
I 8 3 7 10 7 3 4 7

to cover a variety of political subjects that had no relation to that existed at the time of the collection. In an attempt to
climate change. For example, the account “@AIIAmerican- overcome part of this limitation, at least as it related to the
GirI” posted news tweets that were related to the Conserva- deleted accounts in the IN features, which had the highest
tive party . Furthermore, users with an against stance toward deletion percentage, we decided to manually inspect all the
Brexit tended to follow bots that distributed content about tweets where they were mentioned in the collected users’
political news, but had no direct relation to the withdrawal timelines; then, based on this tweets, we decided whether
of the United Kingdom from the European Union. they were bots or not. Since this process was time consum-
ing, we considered all the deleted accounts in the top 100 of
the influential features of the IN features. As Fig. 3 shows,
6 Inspecting the deleted accounts these trends can be spotted within the first 100 features of
the direct interactions.
On average, the deleted accounts constituted approximately We used the same annotation guideline of the Varol-2017
19% of the overall influential accounts in direct interactions Botometer dataset (Varol et al. 2017) to label the deleted
and about 11% of the indirect interactions. One of the limi- accounts as bots or not. The annotation guideline of the bot-
tations in our previous analysis was failing to analyze these detection study by Varol et al. (2017) was based on inspect-
deleted accounts. For some of the topics, the number of the ing the account’s profile page and looking for common flags,
deleted accounts in the top 1000 was over 30%; we can- such as a stock profile image or retweeting that occurs within
not confirm whether these accounts were bots or real users. seconds. As these deleted accounts had no Twitter profile
This limitation is usually found in the studies of social bot information, inspecting these accounts’ profile pages was
behaviors as a result of collecting tweets in the aftermath not applicable in our case. Since there was no unified rule to
of an event (Rizoiu et al. 2018; Luceri et al. 2019; Rizoiu label an account as a bot, we further retrieved a set of tweets
et al. 2018; Shao et al. 2018; Howard and Kollanyi 2016). from stance dataset where the users interacted with these
These deleted accounts have presented a hurdle in many bot- deleted accounts. These tweets provided additional clues to
detection studies (Rizoiu et al. 2018; Luceri et al. 2019); as label the set of deleted accounts and inspect their behavior.
these accounts no longer exist on the Twittersphere, so it was After manual inspection, we found that some of those
difficult to retrieve the needed information for these accounts accounts were likely bots. Table 7 presents the amount of
to examine the bot behavior of the account. For example, in existing bots and deleted bots in the top 100 IN features. In
a study conducted by Luceri et al. (2019), the dataset was general, bots constituted less than 11% of the top 100 fea-
composed of approximately 99% suspended accounts. In our tures of in-favor and against stances. The topic of immigra-
work, since we focused on user stances and using the set tion had the most proportion of bots that interacted with the
of influential accounts, the percentage became much lower, in-favor stance. As it related to against stances, the legaliza-
compared to previous studies of bots on social networks. tion of abortion and atheism contained the largest amount
The deleted accounts in friends networks (EXP) constituted of bots, compared to other topics.
a much lower percentage in comparison with direct interac- Table 8 shows a sample of tweets that demonstrated the
tions (IN). This is due to the fact that the accounts collected characteristics of the the deleted accounts. Some of the
from the direct interactions were extracted from each user’s deleted accounts had the term “bot” as part of the user name,
timeline, which may have contained obsolete mentions, such as in Example 3. In other cases, the account name indi-
while the friends set tended to only contain the accounts cates the behavior of the account, such as “@theism_sucks”

13
Social Network Analysis and Mining (2022) 12:30 Page 15 of 24 30

Table 8  Sample of tweets that interacted with deleted accounts in the top 100 features of (IN)
T S Type Tweet

1 CC – NotBot
@_PinealGland: 1984 [Link] one of the best books ever
2 HC – Bot
@srtalbot2 [Link]
3 A – Bot
RT @ArchbishopYoung: “Today is your day, your mountain is waiting, so get on your way.” - Dr Seuss #Quote
4 A + Bot
@X @theism_sucks we christians dont want dark matters to rule our world. we love the #Light #happiness #God’
5 A + Bot
@2ManyOfUs @theism_sucks pettiness? #bible speaks the truth. #owned again #atheist sucks LOL’
6 B + Bot
RT @Silentwoo: @IrishVol69th @alley167 @BarbietheBrain @LisaNiebs @NeensCa @heyitsCarolyn @Sekusa1
@ON11...
7 B + NotBot @X @coolredmac Well said, Sir.
8 B + NotBot RT @coolredmac: Which is why she is no longer prime minister Emmanuel Macron praises Theresa May for being loyal
and respectful to EU
9 FM + NotBot RT @PETTYMAMII: Not seeing your bestfriend for a long X time really hurts
10 LA – Bot RT @LiveActionFilms: Paul: “The right to life and freedom of religion preexist government.” #VVS14 #prolife’
11 LA – Bot RT @LiveActionFilms: “Humanizing,” @PBS? What is human, anyway? Watch the video:X #AfterTiller #abortionac-
cess’
12 LA – Bot RT @LiveActionFilms: Our latest video showing @PPact dangerous #SexEd for kids was featured on OReilly last night”

We used “X” to mask some users accounts and hide sensitive content

Table 9  Sample of accounts that we verified their bot likeness with explanation from Bot-Detective tool
Type V Account with explanation

1 NotBot ✓ @RichardDawkins This account is verified. Almost always, this means that the account belongs to a non-bot user.
2 Bot ✓ @GOTGeekX This account’s URL per word ratio for each tweet, is suspiciously high
3 NotBot ✓ @hqtriviafans Normal average number of characters per tweet (72.75 ). Bots usually have 143.7 characters on their tweets
4 Bot ✗ @laura_beene the average liked tweets is normal
5 Bot ✗ @saysmysister This account uses symbols rarely (11.53 symbols per tweet). Bots usually have 21.2 symbols per tweet, on
average

(V) indicates the verification of likeness

(see Examples 4 and 5). User interactions with this account a suspended account which has non-hateful tweets when we
were conducted in the sense of mentioning it to defend their retrieved the account’s timeline tweets from our dataset.
religious perspectives. Some accounts had limited content One of the obvious indicators of bot behavior was the vast
in our dataset such as in Example 6. In this example, the amount of retweets that showed the content of the account,
account “@BarbietheBrain” appeared in a retweet with other such as those in Examples 10, 11, and 12. These examples
accounts as a means of promoting these accounts. We con- showed retweets from a bot account “@LiveActionFilms”
sidered such situations as promoting an account by spreading interacting with against stance on the legalization of abor-
automated content and labeled the account as a bot.17 Other tion. This account was suspended because of spreading
deleted accounts had tweets that showed somewhat personal negative messages.
messages such as in Example 7. The account “@coolred-
mac” was a suspended account, as this account had hateful
tweets, such as those in Example 8. In this case, we labeled 7 Verifying the bot/non bot accounts
this account as not being a bot account. Other accounts had
normal content based on the retweet behavior in our dataset, In order to verify the reliability of Botometer in detecting
such as in Example 9. The account “@PETTYMAMII” was bots and non-bot accounts, we verified the propriety of the
top 10 accounts for each topic/stance and identified the
likely bot accounts. We inspect the type of the accounts and
measure the Cohen’s kappa score between the Botometer
and annotation labels to gauge the reliability between the
17
In Example 6, a combined account with “@BarbietheBrain” posed two labels. We used the same annotation guideline of the
bot behavior based on their profile characteristics. Varol-2017 (Varol et al. 2017) to verify the likelihood of

13
 30 Page 16 of 24 Social Network Analysis and Mining (2022) 12:30

bot. Also, we used Bot-Detective API (Kouvela et al. 2020) and human distributions within the top 1000 most predictive
to provide further explainable hints for bot-like accounts, accounts for their stances concerning seven topics.
that helps us to provide ground truth labels by using extra Overall, while bot accounts were present in the top influ-
information beyond inspecting Twitter page. Table 9 pro- ential accounts in predicting the stances thereof, the bots
vides some examples of accounts and examples of explana- had the lowest percentage, compared to human accounts,
tions provided by Bot-Detective in identifying the bot-likely as shown in Fig. 2. This result while it confirms our first
accounts. We found a high alignment between manual anno- hypothesis (H1) where bots have a presence in the top fea-
tation and Botometer labels in identifying human accounts tures, this presence is considered much less than what we
with Cohen’s kappa score equal to 68.81%, which indicates expected. This finding places an emphasis on the noticeable
a substantial agreement between the manual annotations and connection between human accounts and a given stance,
Botometer. Even in cases where the account seems to be a compared to that of bots. Our results align with the recent
nonpersonal account, using Bot-Detective helps in verifying study by Dunn et al. (2020) who investigate the effect of bots
those accounts. For instance, the account @hqtriviafans is in comparison with people in social media dataset related
a fan page for the trivia game show. Example 3 in Table 9, to COVID19. They found the role of bots in spreading fake
shows that this account has a high likelihood to be human news about antivaccine is limited.
as the average number of characters per tweet (72.75 ). Bots Moreover, we showed the magnitude of the effect of the
usually have 143.7 characters on their tweets. Although, for top 1000 accounts on predicting the stances related to three
some bot accounts, the score was on edge, even for the anno- kinds of real user accounts (i.e., normal, famous, and ultra-
tators. This is due to the fact that some of these accounts are famous), as shown in Fig. 3. The noticeable link between the
mostly having low tweets with the default setting. In these ultra-famous accounts and stance formation can be observed
cases, we use Bot-Detective to provide explanations based in the first top 10 accounts that influenced the given stance.
on non-profile information and further inspect the type of This finding does not align with the “million followers fal-
those accounts. For instance, the account @GOTGeekX, has lacy” theory (Avnit 2009), which was confirmed for Twitter
a score of 3.7 out of 5 using Bot-Detective, and 0.96 in Bot- by Cha et al. (2010). Throughout this study, we showed that
ometer. Using the explanations generated by Bot-Detective, the generalization of the followers’ theory is not applicable
the account is highly likely a bot, considering that the URL in the realm of measuring the influence and connection to
per word ratio for each tweet is suspiciously high (Table 9, stance.
example 2). Furthermore, we provide a finer granularity analysis of
the role of the bots on the topic level (see Figs. 8 and 7
in Appendix A). It can be observed that the relationship
8 Discussion between the ultra-famous users and the given stance repre-
sented the general trend on the topic level.
Given the prevalence of bots in social media, it is crucial to
examine the role these accounts play in affecting the online 8.2 The link between bots and supporting
users’ stances and to understand the interaction behavior of versus opposing stances
these accounts. Measuring the factors that relate to people’s
stances in social media is a complex process that is influ- When addressing the second research question, we noticed
enced by various behavioral signals (Lee et al. 2010; Cha that the role of bots on the supporting and opposing stances
et al. 2010) Motivated by this challenge, we investigated the was relatively different for a majority of the topics, which
role of bots using a gold-standard stance-labeled dataset that aligns to our second hypothesis (H2). This can be seen
contained real users’ stances on seven topics; this dataset in the proportion of bots that influenced the stances, as
contained events and topics that covered three main domains shown in Fig. 2, even though the bots presence in the two
(i.e., politics, religion, and social aspects). In this study, we topics of atheism and climate change was sizable on the
extended our understanding of the relationship between bots against stance, compared to the in-favor stance. However,
and the stances of social media users, and we highlighted by inspecting the bots in the friends set (EXP) for climate
various implications for bot studies. change, we found that most of these automated accounts had
no direct relation to climate change. As for the other topics,
8.1 Bot and human presence on stances there was at least one bot account in the top three accounts
in the friends set that were related to the topic of the stance.
To answer the first research question and to assess the asso- This finding indicates that bots can have a greater role on
ciation between bots and stance, we analyzed the most influ- a specific stance type than the others for some topics . For
ential accounts to predict users’ stances, and we inspected instance, users with stances that oppose atheism tend to fol-
the presence of bots among these accounts. We showed bot low and interact with bots that have religious content.

13
Social Network Analysis and Mining (2022) 12:30 Page 17 of 24 30

The same was observed for the friends accounts (EXP) exposed to by following them than by directly interacting
that influenced the against and in-support stances towards with them through retweets, replies, or mentions. This find-
the legalization of abortion, which had approximately 7% ing supports the third hypothesis (H3) where bots are shown
bots. Furthermore, there was a noticeable difference in bot to have presence in the direct and indirect interactions,
distribution at the topic level. The presence of bot accounts which shows that they can affect stances even indirectly by
was sizable in the direct interaction (IN) and indirect expo- having the user get exposed to their posted content without
sure (EXP) , in atheism, as is shown in Fig. 2. The fewest the need to interact directly with them.
bot accounts were seen in Brexit, where bots constituted Furthermore, we found that users with an against stance
approximately 2% of the overall interactions. When we fur- towards a given topic tended to have more indirect interac-
ther inspected the type of bots that influenced the stance tions with bots, compared to direct interactions related to
toward atheism, we noticed that these accounts had a reli- the same stance toward a topic. This kind of online behavior
gious theme that promoted faith, which supported the spe- places an emphasis on the potential hidden effect of bots,
cific type of stance. Similarly, the bots that influenced Brexit which contrasts with the existing norms of studying the
stances tended to have a political theme and a focus on news effect of social bots by solely focusing on the direct interac-
related to the withdrawal of the United Kingdom from the tions of users with the bot content “retweets”.
European Union, such as watching_eu and Brexit_WestMids.
Moreover, we inspected the stance level of interactions 8.4 Implications
with bots. We showed that users tended to directly interact
with bots that had a stance that was different from theirs (see Prior to this study, the literature has informed us that bots are
Examples 3 and 5 and Table 5). This indicates the simple present in social media, and they affect drifting discussions
direct interaction to a bot’s content does not have a direct and spreading certain information related to a given topic.
relation to a user’s stance. This behavior can be supported by However, it was not clear if their presence have any relation
the backfire effect (Nyhan and Reifler 2010), which means with users’ stance online. Our main findings suggested that
that usually exposure to this kind of contradicting content bots’ presence is linked to stance as it can be correlated to
leads to maintaining a stance despite new information that the main signals that can predict a given stance. However,
firmly contradicts it. This finding helps with gaining a better our analysis shows that bots role is minimal compared to
understanding of the effect of social bots on users’ stances influential and famous human accounts. This finding of our
on social media. Also, it is worth noting that in general study suggests that the large fear of bots spreading harmful
social media users tend to follow bots that align with their messages on social media might be overrated. This study
leanings. This can be seen in the top accounts that influenced does not deny the effect of their presence on the stances
the prediction of against stance towards atheism, where of people on a given topic, but we show that it is marginal
users tended to follow religious accounts, such as 2ayaat compared to other factors. Our findings in this study set the
and RTAL_D3OAH , as shown in Table 6. Nevertheless, the path for the research community with future research oppor-
general trend was that the top bots accounts in the friends list tunities to further examine the clear impact of bots on people
had no direct relation to the stance topic. This can be seen stances by conducting qualitative studies.
in the most influential follower bot accounts in predicting Another implication should be geared towards imple-
an against stance toward climate change. One of the top bot menting the policy of social media platforms, such as Twit-
accounts was AIIAmericanGirI, which had no direct relation ter, when dealing with these accounts. It is important to
to climate change, as is shown in Table 6. increase the awareness of social media users about the effect
of bots. As it has been shown, having users exposed to bots
content through following them is enough to predict their
8.3 Bots’ link to stance based on the interactions stance, even more than when users interact directly with bots
type content through retweeting or commenting.
Finally, stance detection on social media can enable a
The third research question was concerned with whether thorough understanding of the interplay between stance
users were influenced by being exposed to posts from the towards a topic and the online signals. The ability to fur-
social bots. We extended the effort of previous research in ther analyze the hidden effect of bots as indirect interactions
this field by looking beyond the bots diffusion and analyzing presents new territory for the current study of automated
bot interplay with online stances using two kinds of net- accounts amplification of fake news towards a certain stance
works: direct interactions with bots (IN) and indirect expo- (supporting/against). The focus of these studies needs to fur-
sure to their content (EXP). Overall, we found that users’ ther address the indirect interactions instead of solo depend-
stances were more related to bots whose content they were ence on direct interactions as a retweet.

13
 30 Page 18 of 24 Social Network Analysis and Mining (2022) 12:30

8.5 Limitations to examine their exposure to bot content. We used the gold

standard of annotated stance data that contains seven topics
Understanding bots’ effect on social media is one of the covering politics, religion, and social aspects. We showed
highly valued questions in social computing community empirical evidence of the effect of social bots on specific
(Abokhodair et al. 2015; Zheng et al. 2019; Seering et al. stances by using the state-of-the-art stance-detection model.
2018). However, it is challenging to study this kind of effect Our findings further indicate that users on social media
on the online users’ stance. In our study, we used stance tended to have limited direct interactions with social bots,
detection as the mean to link bots effect on users’ stance that famous users in terms of followers had a sizable rela-
by inspecting if those bots can act as predictive features to tionship with these stances, and that ultra-famous users
the stance. However, one limitation of our approach is that tended to have the most presence on the stances interac-
it is hard to confirm that detecting predictive bots for users’ tions of specific topics from various domains. Moreover,
stance means that the stance has been affected by the bot not social media users had indirect exposure to bots, compared
the other way around, that they interact/follow those bots to direct interaction, which suggests that users are more
since they have this stance and those bots reinforce their exposed to bot content in an indirect manner by following
leaning. This is the very typical “correlation does not mean these accounts, compared to direct interaction by retweets or
causality” problem (DeMarie-Dreblow 1991). This is a mentions. These findings help to extend the understanding
common limitation even in existing studies that identify the of the effect of bots on stances on social networks. In the
bots’ effect by analyzing their spread within OSNs (Aiello future, further analysis of the bots’ temporal interactions
et al. 2012; Schuchard et al. 2019). Nevertheless, either their with specific stances would help to further understand the
effect is by shaping users’ stance or by reinforcing an exist- overall effect of bot accounts on online stances.
ing stance, both still show that bots do have some role in
link with polarised stances to the level that they can become
predictive signals for a given stance.
Another well-known limitation on studying bots behav-
Appendix
iour in the social network is the deleted accounts in the
collected dataset. In our work, we tried to address this
limitation by inspecting some of those accounts manually.
However, our addressing to the problem has its other limita- F1‑score equations
tions by itself, since we decided an account to be bot or not
based on limited signals from the tweets mentioning them The macro Favg was calculated as presented in Equation 1:
in the users’ timelines rather than having a proper analysis
to their profiles (that do not exist anymore). Unfortunately, Ffavor + Fagainst
Favg = (2)
this will remain an issue that is difficult to resolve. Never- 2
theless, we hope that our manual inspection to the deleted where Ffavor and Fagainst are calculated as shown below:
accounts gives some indication about these accounts overall
behaviour. 2Pfavor Rfavor
Ffavor = (3)
Pfavor + Rfavor

9 Conclusion
2Pagainst Ragainst
Fagainst = (4)
In this study, we sought to understand the contemporary Pagainst + Ragainst
debate—admittedly, bots are everywhere, but what is the
role that bots play related to polarized stances? We investi-
gated this question by examining two kinds of online user
interactions: direct interactions and indirect exposure. For Distribution of the accounts scores on topic
the direct interactions, we evaluated users’ interactions with level
bots with the use of mentions. As it related to indirect expo-
sure, the analysis was carried out on the friends set of users See Figs. 5 and 6.

13
Social Network Analysis and Mining (2022) 12:30 Page 19 of 24 30

Fig. 5  The distribution of accounts scores on the top 1000 influential accounts from direct interactions (IN) in predicting the against/favor stance
(Topic level)

Fig. 6  The distribution of accounts scores on the top 1000 influential accounts from in direct interactions (EXP) in predicting the against/favor
stance (topic level)

13
 30 Page 20 of 24 Social Network Analysis and Mining (2022) 12:30

Distribution of bots on topic level

Fig. 7  The distribution of bots on the top 1000 influential accounts from the direct interactions (IN) in predicting the against/favor stance (topic
level)

13
Social Network Analysis and Mining (2022) 12:30 Page 21 of 24 30

Fig. 8  The distribution of bots on the top 1000 influential accounts from indirect exposure (EXP) in predicting the against/favor stance (topic
level)

13
 30 Page 22 of 24 Social Network Analysis and Mining (2022) 12:30

Chi‑squared test for accounts distributions References

The chi-squared test results for each topic (Table 10). Abokhodair N, Yoo D, McDonald DW (2015) Dissecting a social bot-
net: growth, content and influence in twitter. In: Proceedings of the
18th ACM conference on computer supported cooperative work
social computing, CSCW’15, New York, NY, USA. Association
Table 10  Chi-squared test for accounts distributions between IN and for Computing Machinery, pp 839-851. ISBN 9781450329224.
EXP bot accounts [Link]
Abu-El-Rub N, Mueen A (2019) Botcamp: Bot-driven interactions in
T Favor Against social campaigns. In: The world wide web conference. ACM, pp
2529–2535
A 1.69e–27*** 1.7e–33*** Aiello LM, Deplano M, Schifanella R, Ruffo G (2012) People are
CC 1.96e–06*** 1.11e–65*** strange when you’re a stranger: impact and influence of bots on
HC 5.43e–34*** 5.28e–44*** social networks. In: Sixth international AAAI conference on
weblogs and social media
FM 7.60e–12*** 3.03e–54***
AlDayel A, Magdy W (2019a) Assessing sentiment of the expressed
LA 3.02e–17*** 2.45e–17*** stance on social media. In: Proceedings of the 11th international
B 0.56 0.56 conference on social informatics (SocInfo 2019)
I 0.36 0.046* AlDayel A, Magdy W (2019b) Your stance is exposed! analysing pos-
sible factors for stance detection on social media. In: The 22nd
Overall 3.17e–69*** 1.77e–178***
ACM conference on computer-supported cooperative work and
*p<0.05; **p<0.01; ***p<0.001 social computing. ACM
AlDayel A, Magdy W (2021) Stance detection on social media: state
of the art and trends. Inf Process Manag 58(4):102897
Allcott H, Gentzkow M (2017) Social media and fake news in the 2016
election. Technical report, National Bureau of Economic Research
Distribution of baseline tweets Avnit A (2009) The million followers fallacy. Pravda Media Group
in the dataset Bastos MT, Mercea D (2019) The brexit botnet and user-generated
hyperpartisan news. Soc Sci Comput Rev 37(1):38–54
See Table 11 Bessi A, Ferrara E (2016) Social bots distort the 2016 us presidential
election online discussion. First Monday, 21 (11–7)
Boichak O, Jackson S, Hemsley J, Tanupabrungsun S (2018) Auto-
mated diffusion? bots and their influence during the 2016 us
presidential election. In: International conference on informa-
Table 11  The distribution of baseline tweets per topic in the SemEval tion. Springer, pp 17–26
and Events datasets Broniatowski DA, Jamison AM, Qi S, AlKulaib L, Chen T, Benton
A, Quinn SC, Dredze M (2018) Weaponized health communica-
Dataset Topic Favor Against Neither Total tion: Twitter bots and russian trolls amplify the vaccine debate.
Am J Public Health 108(10):1378–1384
SemEval Atheism (A) 93 349 108 550
Center PR (2013) News use across social media platforms
Climate change (CC) 284 16 161 461 Cha M, Haddadi H, Benevenuto F, Gummadi KP (2010) Measuring
Hillary clinton (HC) 113 387 170 670 user influence in twitter: the million follower fallacy. In: Fourth
Feminist movement 171 245 108 524 international AAAI conference on weblogs and social media
(FM) Cossu J-V, Labatut V, Dugué N (2016) A review of features for the
discrimination of twitter users: application to the prediction of
Legalization of abortion 98 406 166 670
offline influence. Soc Netw Anal Min 6(1):25
(LA)
Cresci S, Di Pietro R, Petrocchi M, Spognardi A, Tesconi M (2015)
Events Brexit (B) 71 395 0 466 Fame for sale: efficient detection of fake twitter followers. Decis
Immigrations (I) 232 1280 0 1512 Support Syst 80:56–71
Total 1062 3078 713 4853 Darwish K, Magdy W, Zanouda T (2017) Trump vs. hillary: What
went viral during the 2016 us presidential election. In: Interna-
tional conference on social informatics. Springer, pp 143–161
Darwish K, Stefanov P, Aupetit MJ, Nakov P (2019) Unsupervised
Open Access This article is licensed under a Creative Commons Attri- user stance detection on twitter. In: ICWSM
bution 4.0 International License, which permits use, sharing, adapta- Davis CA, Varol O, Ferrara E, Flammini A, Menczer F (2016) Bot-
tion, distribution and reproduction in any medium or format, as long ornot: A system to evaluate social bots. In: Proceedings of the
as you give appropriate credit to the original author(s) and the source, 25th international conference companion on world wide web.
provide a link to the Creative Commons licence, and indicate if changes International world wide web conferences steering committee,
were made. The images or other third party material in this article are 2016, pp 273–274
included in the article’s Creative Commons licence, unless indicated DeMarie-Dreblow D (1991) Relation between knowledge and mem-
otherwise in a credit line to the material. If material is not included in ory: a reminder that correlation does not imply causality. Child
the article’s Creative Commons licence and your intended use is not Dev 62(3):484–498
permitted by statutory regulation or exceeds the permitted use, you will Dunn AG, Surian D, Dalmazzo J, Rezazadegan D, Steffens M, Dyda
need to obtain permission directly from the copyright holder. To view a A, Leask J, Coiera E, Dey A, Mandl KD (2020) Limited role
copy of this licence, visit [Link] of bots in spreading vaccine-critical information among active

13
Social Network Analysis and Mining (2022) 12:30 Page 23 of 24 30

twitter users in the united states: 2017–2019. Am J Public Ng LHX, Carley K (2021) Flipping stance: social influence on bot’s
Health 110(S3):S319–S325 and non bot’s Covid vaccine stance. MIS2 workshop at KDD
Dutta HS, Aggarwal K, Chakraborty T (2021) Decife: detecting col- 2021
lusive users involved in blackmarket following services on twit- Nyhan B, Reifler J (2010) When corrections fail: the persistence of
ter. In: Proceedings of the 32nd ACM conference on hypertext political misperceptions. Polit Behav 32(2):303–330
and social media, pp 91–100 Puertas E, Moreno-Sandoval LG, Plaza-del Arco FM, Alvarado-
Ferrara E (2017) Disinformation and social bot operations in the run Valencia JA, Pomares-Quimbaya A, Alfonso L (2019) Bots and
up to the 2017 French presidential election. First Monday 22(8) gender profiling on twitter using sociolinguistic features
Ferrara E (2019) Bots, elections, and social media: a brief overview. Pulido CM, Redondo-Sama G, Sordé-Martí T, Flecha R (2018)
arXiv:​1910.​01720 Social impact in social media: a new method to evaluate the
Friedkin NE, Johnsen EC (1990) Social influence and opinions. J social impact of research. PLoS ONE 13(8):e0203117
Math Sociol 15(3–4):193–206 Ratkiewicz J, Conover M, Meiss M, Gonçalves B, Patil S, Flammini
Garimella K, West R (2019) Hot streaks on social media. In: Pro- A, Menczer F, Detecting and tracking the spread of astroturf
ceedings of the international AAAI conference on web and memes in microblog streams
social media, vol 13, No. 01. pp 170–180. [Link] Ratkiewicz J, Conover M, Meiss M, Gonçalves B, Patil S, Flammini
org/​ojs/​index.​php/​ICWSM/​artic​le/​view/​3219 A, Menczer F (2011) Truthy: mapping the spread of astroturf
Gilani Z, Farahbakhsh R, Tyson G, Crowcroft J (2019) A large-scale in microblog streams. In: Proceedings of the 20th international
behavioural analysis of bots and humans on twitter. ACM Trans conference companion on world wide web, WWW ’11, New
Web. [Link] York, NY, USA. Association for Computing Machinery, pp
Graells-Garrido E, Baeza-Yates R, Lalmas M (2020) Every col- 249–252. ISBN 9781450306379. [Link]
our you are: stance prediction and turnaround in controversial 92.​19633​01
issues. WebSci Rizoiu M-A, Graham T, Zhang R, Zhang Y, Ackland R, Xie L (2018)
Grčar M, Cherepnalkoski D, Mozetič I, Novak PK (2017) Stance # debatenight: the role and influence of socialbots on twitter dur-
and influence of twitter users regarding the brexit referendum. ing the 1st 2016 us presidential debate. In: Twelfth international
Comput Soc Netw 4(1):6 AAAI conference on web and social media
Hegelich S, Janetzko D (2016) Are social bots on twitter political Samih Y, Darwish K (2020) A few topical tweets are enough for effec-
actors? empirical evidence from a Ukrainian social botnet. In: tive user-level stance detection. arXiv:​2004.​03485
Tenth international AAAI conference on web and social media Santia GC, Mujib MI, Williams JR (2019) Detecting social bots on
Howard PN, Kollanyi B (2016) Bots,# strongerin, and# brexit: com- facebook in an information veracity context. In: Proceedings of
putational propaganda during the uk-eu referendum. Available the international AAAI conference on web and social media, vol
at SSRN 2798311 13, pp 463–472
Kouvela M, Dimitriadis I, Vakali A, Bot-detective: an explainable Schuchard R, Crooks AT, Stefanidis A, Croitoru A (2019) Bot stamina:
twitter bot detection service with crowdsourcing functionali- examining the influence and staying power of bots in online social
ties. In: Proceedings of the 12th international conference on networks. Appl Netw Sci 4(1):55
management of digital ecosystems, MEDES’20, New York, NY, Seering J, Flores JP, Savage S, Hammer J (2018) The social roles of
USA. Association for Computing Machinery, pp 55–63. ISBN bots: evaluating impact of bots on discussions in online commu-
9781450381154. [Link] nities. Proc ACM Hum Comput Interact 2(CSCW):1571–15729.
Lai M, Patti V, Ruffo G, Rosso P (2018) Stance evolution and twit- [Link]
ter interactions in an Italian political debate. In: International Shao C, Ciampaglia GL, Varol O, Yang K-C, Flammini A, Menczer F
conference on applications of natural language to information (2018) The spread of low-credibility content by social bots. Nat
systems. Springer, pp 15–27 Commun 9(1):4787
Lee C, Kwak H, Park H, Moon S (2010) Finding influentials based Stella M, Ferrara E, De Domenico M (2018) Bots increase exposure to
on the temporal order of information adoption in twitter. In: negative and inflammatory content in online social systems. Proc
Proceedings of the 19th international conference on World wide Natl Acad Sci 115(49):12435–12440
web. ACM, pp 1137–1138 Tardelli S, Avvenuti M, Tesconi M, Cresci S Characterizing social
Luceri L, Deb A, Badawy A, Ferrara E (2019) Red bots do it better: bots spreading financial disinformation. In: Meiselwitz G (ed)
comparative analysis of social bot partisan behavior. In: Com- Social computing and social media. Design, ethics, user behav-
panion proceedings of the 2019 world wide web conference. ior, and social network analysis. Springer, pp 376–392. ISBN
ACM, pp 1007–1012 978-3-030-49570-1
Magdy W, Darwish K, Abokhodair N, Rahimi A, Baldwin T (2016a) Thonet T, Cabanac G, Boughanem M, Pinel-Sauvagnat K (2017) Users
# isisisnotislam or# deportallmuslims?: predicting unspoken are known by the company they keep: topic models for viewpoint
views. In: Proceedings of the 8th ACM conference on web sci- discovery in social networks. In: Proceedings of the 2017 ACM
ence. ACM, pp 95–106 on conference on information and knowledge management. ACM,
Magdy W, Darwish K, Abokhodair N, Rahimi A, Baldwin T (2016b) pp 87–96. [Link] [Link]
# isisisnotislam or# deportallmuslims?: predicting unspoken org/​10.​1145%​2F313​2847.​31328​97
views. In: Proceedings of the 8th ACM conference on web sci- Varol O, Ferrara E, Davis CA, Menczer F, Flammini A (2017) Online
ence. ACM, , pp 95–106. ISBN 1-4503-4208-6 human-bot interactions: detection, estimation, and characteriza-
Mendoza M, Tesconi M, Cresci S (2020) Bots in social and interac- tion. In: Eleventh international AAAI conference on web and
tion networks: detection and impact estimation. ACM Trans Inf social media
Syst (TOIS) 39(1):1–32 Wate Y (2021) 10 best twitter bots you should follow in 2022. TechPP
Mohammad S, Kiritchenko S, Sobhani P, Zhu X-D, Cherry C (2016) URL [Link]
SemEval-2016 Task 6: detecting Stance in Tweets. In: SemE- Yang K-C, Varol O, Davis CA, Ferrara E, Flammini A, Menczer F
val@ NAACL-HLT, pp 31–41 (2019) Arming the public with artificial intelligence to counter
Musco C, Musco C, Tsourakakis CE (2018) Minimizing polarization social bots. Human Behav Emerg Technol 1(1):48–61
and disagreement in social networks. In: Proceedings of the
2018 world wide web conference, pp 369–378

13
 30 Page 24 of 24 Social Network Analysis and Mining (2022) 12:30

Zheng LN, Albano CM, Vora NM, Mai F, Nickerson JV (2019) The Publisher's Note Springer Nature remains neutral with regard to
roles bots play in Wikipedia. Proc ACM Hum Comput Interact. jurisdictional claims in published maps and institutional affiliations.
[Link]

13