Ad Delivery Algorithms:

The Hidden Arbiters of Political Messaging

Muhammad Ali∗ Piotr Sapiezynski∗ Aleksandra Korolova
Northeastern University Northeastern University University of Southern California
[email protected] [email protected] [email protected]

Alan Mislove Aaron Rieke

Northeastern University Upturn
[email protected] [email protected]
arXiv:1912.04255v3 [cs.CY] 17 Dec 2019

ABSTRACT 1 INTRODUCTION
Political campaigns are increasingly turning to digital advertising Political campaigns spend millions of dollars on ads to get their
to reach voters. It is predicted that, during the 2020 U.S. presidential message out to voters. Recently, much of that spending has mi-
elections, 28% of political marketing spending will go to online grated from traditional broadcast media (e.g, television, radio, and
advertising, compared to 20% in 2018 and 0.2% in 2010. Digital newspapers) to the Internet in the form of digital advertising. A
advertising platforms’ popularity is partially explained by how they recent study [40] predicts that during 2020, political ad spending
empower advertisers to target messages to platform users with overall will top $9.9B, with over $2.8B of that being paid to digital
great precision, including through inferences about those users’ ad platforms.
political affiliations. However, prior work has shown that platforms’ Facebook, one of the world’s largest advertising platforms, earns
ad delivery algorithms can selectively deliver ads within these target a significant portion of this revenue from candidates at the national,
audiences in ways that can lead to demographic skews along race state, and local levels. The company has a dedicated site for political
and gender lines, often without an advertiser’s knowledge. campaigns [17] which states [18]:
In this study, we investigate the impact of Facebook’s ad deliv-
Facebook advertising can help you deliver a message
ery algorithms on political ads. We run a series of political ads on
directly to constituents so you can better understand
Facebook—one of the world’s largest advertising platforms—and
and engage with them on issues they care most about.
measure how Facebook delivers those ads to different groups, de-
pending on an ad’s content (e.g., the political viewpoint featured) According to Facebook’s Ad Library [14], political campaigns have
and targeting criteria. We find that Facebook’s ad delivery algo- spent over $907M on Facebook ads worldwide since May 2018,
rithms effectively differentiate the price of reaching a user based with the Trump campaign alone currently spending over $1M each
on their inferred political alignment with the advertised content, week [47]. Furthermore, a recent study found that, at the state level,
inhibiting political campaigns’ ability to reach voters with diverse "more than 10 times as many candidates advertise on Facebook than
political views. This effect is most acute when advertisers use small advertise on TV" [25]. This adoption reflects the fact that social
budgets, as Facebook’s delivery algorithm tends to preferentially media platforms have substantially lowered the cost of advertising,
deliver to the users who are, according to Facebook’s estimation, expanding the number of campaigns who can feasibly reach voters
most relevant. Moreover, due to how Facebook currently reports ad through digital channels [25]. Given the growing importance of
performance, this effect may be invisible to political campaigns. online ads to the political discourse, it is critical to understand how
Our findings point to advertising platforms’ potential role in complex ad platforms like Facebook operate in practice.
political polarization and creating informational filter bubbles. We Much attention has been had to the ad creation and targeting
show that Facebook preferentially exposes users to political ad- phase, where the advertiser selects their desired audience and up-
vertising that it believes is relevant for them, even when other loads their ad creative. Researchers have shown that advanced
advertisers with opposing viewpoints may be actively trying to targeting features on ad platforms can be used to prevent certain
reach them. Furthermore, some large ad platforms have recently ethnic groups from seeing ads [15, 46]. For example, in 2016, the
changed their policies to restrict the targeting tools they offer to Trump campaign used these techniques to carry out "major voter
political campaigns; our findings show that such reforms will be suppression operations" aimed at lowering turnout among young
insufficient if the goal is to ensure that political ads are shown to women and black voters [27], and there is evidence that Russian
users of diverse political views. Counterintuitively, advertisers who organizations used these tools interfere with 2016 U.S. presidential
target broad audiences may end up ceding platforms even more in- elections [42, 49].
fluence over which users ultimately see which ads, adding urgency However, there is a less well-understood aspect of modern ad
to calls for more meaningful public transparency into the political platforms that may be playing an equally important role: The al-
advertising ecosystem. gorithms that ad platforms use to decide which ads get delivered
to which users. Recent work [4] has shown that an ad platform’s
∗ These two authors contributed equally. choices during the ad delivery phase—rooted in the desire to show
relevant ads to users, ostensibly to provide users with a “better expe-
rience” [43, 44]—can lead to dramatic skew in delivery along gender
and racial lines, even when the advertiser aims to reach gender- a candidate to reach their entire audience. We used long-running
and race-balanced audiences. In other words, an ad platform may ads, along with the Facebook-provided limits on how frequently a
deliver ads to a skewed subset of the advertiser’s targeted audi- given set of ads can be shown to a user, to “force” the platform to
ence based on the content of the ad alone, and do so unbeknownst consider delivering our ads to all of our targeted users. This way,
to—and out of the control of—both the users and the advertisers. we were able to “exhaust” the audience to determine how much of
As far as we are aware, we are the first to study whether such it the platform will allow a given message to be shown to. Third,
skews are introduced for political ads on real-world advertising we needed to determine how we were being charged for delivering
platforms due to the ad delivery phase alone. We focus on Facebook ads to different sub-populations of the target audience. We used
because of its critical importance to today’s digital political adver- Facebook’s advertising reporting features, combined with proxies,
tising. We hypothesize that that Facebook may choose to deliver in order to understand how our budget is split across users with
ads only to the subset of the political campaign’s target audience different political leanings.
that it predicts will be aligned with a campaign’s views, despite
attempts by the campaign to reach a diverse range of voters, and Contributions After running our ads and analyzing the results,
that this practice might play a role in political polarization by cre- we present the following contributions:
ating informational filter bubbles. Specifically, we seek to answer: First, we show that, despite identical targeting parameters, budgets,
Is a political campaign advertising on Facebook able to reach all of and competition from other advertisers, the content of a political
the electorate? Or, is Facebook preferentially delivering ads to users ad alone can significantly affect which users Facebook will show
who it believes are more likely to be aligned with the campaign’s the ad to. For example, we find Facebook delivers our ads with
political views? Additionally, does Facebook vary ad pricing based on content from Democratic campaigns to over 65% users registered
its hypothesized match between the target audience and campaign’s as Democrats, while delivering ads from Republican campaigns to
political views? under 40% users registered as Democrats, despite identical target-
These questions are particularly urgent in light of the debate ing parameters. Moreover, our “control” ads with neutral political
unfolding over the “microtargeting” of political ads. In late October content that are run at the same time are delivered to a much more
2019, Twitter decided to change its policy and ban all political balanced audience (47% Democrats), showing that preferentially
advertising on its platform [48]. In response, U.S. Federal Election delivery is a result of Facebook’s ad delivery algorithm.
Commission (FEC) Chair Ellen Weintraub publicly argued that Second, we find that this effect is surprisingly not present when we
instead of banning such ads, ad platforms should limit political target users who donated to political campaigns, rather than those
advertisers’ ability to narrowly target ads to ensure that "a broad who are registered for a given political party.
public can hear the speech and respond" [54]. Shortly after, Google
announced that it will significantly limit election ad targeting in Third, we find that that the delivery skew is present to an even
order to "promote increased visibility of election ads" [26]. At the greater degree when we use Facebook’s own political targeting fea-
time of this writing, Facebook is considering reforms of its own ad tures. For example, when we target an audience of users who Face-
platform, but details are sparse [16]. book believes have “Likely engagement with US political content
Our questions regarding ad delivery are important to these de- (Liberal)”, combined with an equal-sized audience who Facebook
velopments for at least two reasons. First, skews resulting from ad considers to have “Likely engagement with US political content
delivery can raise fundamentally similar concerns to those raised (Conservative)”, we find that our ads from Democratic campaigns
about narrow targeting: an electorate who cannot “hear and re- deliver to over 60% liberal users (compared to ads from Republican
spond” to political speech. Second, ad delivery algorithms might campaigns, which deliver to 25% liberal users).
counteract the goals of restricting microtargeting by redirecting Fourth, we find that it can be difficult and more expensive for po-
ads according to the choices of the ad platforms (in spite of broader litical campaigns to have their content delivered to those who
target audiences). Policymakers must be alert to these implications. Facebook believes are not aligned with the campaign’s views. For
To test our hypotheses, we became a political advertiser and ran example, when re-running ads for Bernie Sanders (a liberal candi-
over $13K1 of political ads under controlled conditions, and ob- date) and Donald Trump (a conservative candidate), we find that
served how Facebook’s algorithms delivered them. Unfortunately, when targeting an audience of conservative users, in the first day
Facebook makes it difficult to understand ad delivery along axes of the ad campaign, Facebook delivers our Sanders ad to only 4,772
of political affiliation; to measure these results, we had to design users, while our Trump ad is delivered to 7,588 users.2 We find
careful experiments. First, we needed to determine which users Face- that the underlying reason is that our Sanders ads targeting con-
book was delivering our ads to, and whether skew (along political servative users are charged significantly more by Facebook than
lines) exists among these users. We re-used techniques published our Trump ads ($15.39 versus $10.98 for 1,000 impressions), despite
in prior work [4, 46], using proxies based on ground-truth data being run from the same ad account, at the same time.3 Moreover,
from the voter records and political donation records. Addition- the difference cannot be attributed to some unknown underlying
ally, we created audiences according to their political leaning—as difference in liberal and conservative users’ use of Facebook, as
inferred by Facebook—and used them simultaneously as part of an our neutral ad targeting the same audiences and run at the same
ad campaign but in a way that we can explicitly see the delivery to time is delivered much more uniformly, reaching 6,822 liberal and
each subgroup. Second, we needed to determine if it is possible for
2 We find a similar, but flipped, effect if we target an audience of liberal users.
1 Throughout the paper we refer to prices in U.S. Dollars. 3 Again, we see a similar, flipped effect when targeting liberal users.
6,584 conservative users at a cost of $12.07 and $12.65 for 1,000 Limitations It is important to note the limitations of our study
impressions, respectively. (see Section 5.3 for a detailed discussion). Most importantly, we
can only report results of our own ads; we are unable to make any
Fifth, we find that when an ad creative and landing page shown statements about the extent to which any effects we observe exist
to the users is neutral, but we “trick" Facebook’s algorithm into for political ads run by real political campaigns, or political ads in
believing the ad leads to a page with content taken from a particular general. However, the fact that we observe statistically significant
candidate’s campaign web site, the skews in delivery and differential skews in our small set of ads suggests that the effects we observed
pricing are also present. This suggests that the ad delivery decisions are likely present in the delivery of other political ads as well.
made by Facebook are not driven exclusively by user reactions to The rest of this paper is organized as follows. Section 2 provides
the ad (as all such ads appeared identical to the users), but instead background on Facebook’s advertising platform and Section 3 de-
are made at least partially a priori by Facebook itself. tails related work. Section 4 gives an overview of our methodology,
and Section 5 presents the results of our experiments. Section 6
Taken together, our results indicate that Facebook preferentially offers a concluding discussion.
shows users political ads whose content it predicts are aligned
with their political views, with negative implications for both users 2 BACKGROUND
and campaigns. For users, such delivery limits users’ exposure In this section, we provide some basic background about Face-
to diverse viewpoints and—if Facebook’s inference is incorrect— book’s advertising platform, the subject of this study, necessary to
may pigeonhole them into a particular slice of political ads. For understand our methodology (described in Section 4).
campaigns, such delivery may inhibit them from reaching beyond
their existing “base” on Facebook, as getting ads delivered to users 2.1 Overview
the platform believes are not aligned with their views may become
On Facebook, like many ad platforms, there are two phases to
prohibitively expensive. Importantly, these effects may be occurring
advertising: ad creation and ad delivery.
without users’ or campaigns’ knowledge or control.
Stepping back, our findings raise serious concerns about whether Ad creation Ad creation refers to the process by which an ad-
Facebook and similar ad platforms are, in fact, amplifying polit- vertiser submits their ad to the Facebook ad platform. During this
ical filter bubbles by economically disincentivizing content they stage, advertisers provide the contents of the ad (the images, videos,
believe are not aligned with users’ political views. Put simply, Face- text, and destination link collectively called the ad creative), and
book is making decisions about which ads to show to which users specify the target audience of users on the platform to whom they
based on its own priorities (presumably, user engagement with or wish the ad to be shown (see §2.2). Advertisers often run many ads
value for the platform). But in the context of political advertising, that are related; collectively, these are called an ad campaign. Before
Facebook’s choice may have significant negative externalities on submitting their ad campaign, advertisers also specify an objective
political discourse in society at large. (i.e., what they want to achieve with the campaign, see §2.3) and
the ad budget they are willing to spend to achieve that objective
Ethics All of our experiments were conducted with careful con-
(see §2.4). The ads then enter a review process to be approved to
sideration of ethics. First, we obtained Institutional Review Board
run on the platform (see §2.5).
review of our study, with our protocol being marked as “Exempt”.
We did not collect any users’ personally identifying information Ad delivery Ad delivery refers to the process by which the
from Facebook, and did not collect any information about users Facebook ad platform selects which ads get shown to which users.
who visited our site after clicking on our ads. Second, we minimized Before displaying an ad to a user, the platform will hold an ad auc-
harm to Facebook users when running our ads by only running tion to determine which ad, from among all ads that user is eligible
“real” ads, i.e., if a user clicked on one of our ads, they were brought to see (by virtue of their inclusion in ads’ target audiences), will
to a real-world page not under our control that was relevant to be shown (see §2.6). While the ad campaign is active, the platform
the topic of the ad. In the few cases where the ads pointed to a provides a semi-live, detailed breakdown to advertisers of how their
domain we controlled, the visiting users were automatically and ads are being delivered (see §2.7).
immediately redirected to a real page that we did not control. Third,
we minimized harm to Facebook itself by participating in their ad- 2.2 Targeting
vertising system as any other advertiser would and paying for all of There are many different ways for advertisers to target ads, ranging
our ads. We registered as an advertiser in the area of “Social Issues, from users’ demographics and interests to their personally identifi-
Elections or Politics” [3], meaning our ads were subject to the same able information (PII) [46, 51]. We briefly describe these below.
review as the ads of other political campaigns. Fourth, we mini-
mized the risk of altering the political discourse through careful Detailed targeting Facebook pioneered, and is continuing to
choices of the ad content (Section 4.2), and running approximately aggressively market, ways for advertisers to target its users via
the same number of copies of ads for Republican and Democratic user attributes [46]. These attributes cover a variety of aspects of
candidates, with the same budgets. The total amount we spent on users’ lives, ranging from demographic features to online activity
political advertising while collecting data for this paper ($13.7K) and even offline information, often acquired without users’ explicit
is minuscule compared to the ad budgets of real campaigns in the consent or knowledge. In the context of politics, Facebook derives
same period (likely in the millions of dollars [14]). attributes that indicate whether users are “interested in” various
political candidates (e.g., Donald Trump, Elizabeth Warren), as well algorithm that is not publicly known.5 When using the interface,
as more general attributes about user behaviors, such as “Likely Facebook also provides advertisers with an “Estimated Daily Reach”,
engagement with US political content (Conservative)" or “Likely which Facebook defines as the estimated number of users who
engagement with US political content (Liberal)". The exact method- would be exposed to an ad given the targeting criteria and budget;
ology by which Facebook infers such attributes is not disclosed, this feature helps advertisers with an understanding of how far
but likely involves profile data provided directly to Facebook, data their budget will go for the selected audience. Although the bid
from activity on Facebook (e.g., “Liking” Pages or explicit or im- and cost control options are offered, without the knowledge of
plicit patterns of interaction with particular content), inferences the auction, they are difficult to set effectively, and therefore, it is
based on attributes of a user’s friends, and data inferred from users’ natural to only specify a budget and rely on Facebook to do the
behavior off of Facebook.4 bidding.
How the advertiser is actually charged for their ad campaign
Custom audiences Facebook also allows advertisers to target depends on the objective. For “Reach", the advertiser is charged
users directly using their PII via a tool called Custom Audiences [51]. per impression; for “Traffic" the advertiser is also charged per im-
Using Custom Audiences, advertisers can upload up to 15 different pression unless the optimization is “Link Clicks" (in which case the
kinds of PII to Facebook, ranging from names to email addresses advertiser can choose to be charged per click if they wish).
to phone numbers to dates of birth. Facebook then matches these Finally, if the advertiser chooses the “Reach" objective and
values against their database in order to build an audience for the “Reach" optimization, they are allowed to specify a frequency cap,
advertiser; the advertiser is then allowed to target their ads to just which allows them to set the maximum number of impressions a
the users who match. single user would see over a specified number of days. We use this
feature later in the paper to force Facebook to deliver our ads to
2.3 Objective many users in our audience.
When creating ad campaigns, advertisers on Facebook are asked to
specify their objective, or what they are trying to achieve. Common 2.5 Ad review
objectives include “Reach" (showing the ad to as many users as Once the ad creation phase is complete, and before the ad enters
possible), “Traffic" (showing the ad to the users most likely to click), the ad delivery phase, it is submitted to Facebook for review.6 We
and “App Installs" (showing the ad to the users most likely to install observed that most of our Facebook ads were approved within
the advertiser’s app). 30 minutes, some spent hours in review, and a few were never
Within each objective, advertisers must also specify an optimiza- approved. The criteria and internal mechanisms for approval are not
tion, indicating how Facebook should achieve their objective. For entirely clear, and precise reasons for why certain ads are rejected
the “Reach" objective, the available optimizations are “Reach" (the are not given. In this work, we only report on experiments where
default, showing ads to as many users as possible) and “Impressions" all necessary ads were approved before their scheduled start time.
(showing ads as many times as possible). For the “Traffic" objective,
the available optimizations are “Link Clicks" (the default, showing 2.6 Ad delivery
ads to the users most likely to click), “Landing Page Views" (show- Ad platforms including Facebook commonly use ad auctions to
ing ads to the users most likely to click and visit the destination select which ads to show to users. Historically, this auction took
page), “Daily Unique Reach" (showing ads to users most likely to only the advertiser’s bid price into account; more recently, Facebook
click, at most per day), and “Impressions" (showing ads to the users considers other features such as the overall performance of the
most likely to click, but maximizing the number of impressions). ad and the platform’s estimate of how relevant the ad is to the
We hypothesize that political advertisers are likely to com- browsing user [21, 43]. Facebook says as much in its documentation
monly use the “Reach" campaign objective together with default the for advertisers [21]:
“Reach" optimization—representing the goal of getting their message
[W]e subsidize relevant ads in auctions, so more rele-
out to as many people as possible—and the “Traffic" campaign ob-
vant ads often cost less and see more results. In other
jective together with the default “Link Clicks" ad set optimization—
words, an ad that’s relevant to a person could win an
representing the goal of getting as many people as possible to
auction against ads with higher bids.
visit their campaign page. In this paper we use the “Traffic” opti-
mization in exeriments targeting registered voters and donors (see Facebook explains that it measures relevance as a composite of
Section 5.1) and “Reach” optimization in all other experiments. estimated action rates ("[a]n estimate of whether a particular person
engages with or converts from a particular ad" and ad quality ("[a]
2.4 Bidding, budgets, and billing measure of the quality of an ad as determined from many sources
including feedback from people viewing or hiding the ad") [21].
When creating an ad, the advertiser must tell Facebook their bid,
5 Facebook only says: “Facebook will aim to spend your entire budget and get the most
which takes the form of an ad budget. These budgets are either a
1,000 impressions using the lowest cost bid strategy" for the “Reach" campaign-level
daily or lifetime budget for the ad, allowing Facebook to spend the objective, and “Facebook will aim to spend your entire budget and get the most link
advertiser’s money between and within auctions according to an clicks using the lowest cost bid strategy" for the “Traffic" campaign-level objective. An
optional “Bid Control" (maximum bid in each auction) and “Cost control" (the average
cost per link click) are also available for the “Reach" and “Traffic" campaign-level
4 Recallthat Facebook receives information from a variety of sources beyond the objectives, respectively.
Facebook website and app, including Facebook Pixel tracking [20], app data sharing [9], 6 Most platforms have a review process (consisting of a combination of automated and
third-party data brokers [53], and location data [22]. manual review) to prevent abuse or violations of their advertising policies [2, 55].
 In short, Facebook plays a significant role in determining which Facebook does, indeed, further refine even precisely targeted audi-
users see which ads, based on its own judgment about which ads are ences, introducing demographic and political biases in the reached
likely to be "relevant" to particular users, its own judgement of how audiences, beyond those intended by the advertiser.
to bid on an advertiser’s behalf and distribute the specified budget
among auctions, and possibly other considerations connected to its Filter bubbles The extent, or even the existence, of the filter
business interests. This role—in the context of political ads—is the bubble effect has been a point of contention both in academia and in
key phenomenon that this paper seeks to explore in its experiments. popular media. After the initial reports by Eli Pariser [39] scholars
have attempted to measure the phenomenon in services includ-
ing Google Search [24, 28], Google News [35], and Facebook [13].
2.7 Reporting However, the observed differences could often be explained by user
During the ad delivery phase, Facebook provides semi-live [4] de- location differences (in the case of Google Search) or attributed
tailed statistics to advertisers about how their ad is being delivered. to an individual’s choice of friends to follow (in the case of Face-
In particular, Facebook reports the impressions (the number of times book), rather than stemming from algorithmic personalization of an
the ad was shown), the reach (the number of unique users who saw individual’s experience. Furthermore, Bail et al. warn against over-
the ad), the clicks (the number of times users clicked on the ad), and exposing users to messaging from politicians they do not support
the spend (how much money was spent). Facebook also allows ad- as it appears to increase, rather than decrease their partisanship [6].
vertisers to obtain breakdowns of these performance metrics along Other previous work indicated that the effect might be more
a few axes, most notably gender (broken into “Male”, “Female”, and pronounced in ads than in organic content. Datta et al. showed
“Other”), age (broken down into brackets of 10 year increments), and that personal attributes are used in ad selection, specifically that
location (broken down into Designated Market Area [38], or DMA). changing one’s self-reported gender influences the job ads one
In other words, Facebook will tell advertisers how much they have sees [10]. More recently, Ali et al. showed that the demographic
spent on users in different regions (or of different genders, etc.), and distribution of the audience that receives the ad changes depending
how many clicks/impressions/reach those users represent. Notably on the ad content, even if the same audience was targeted [4].
for this work, Facebook does not provide breakdowns along axes Our work provides further evidence that the filter bubble effect
such as Facebook’s estimated political leaning. is pronounced in the ads users are exposed to; we show that at-
tempting to “burst” the political advertising filter bubble can prove
expensive, especially for smaller advertisers. As far as we are aware,
3 RELATED WORK we are the first to study the filter bubble effect due to ad deliv-
We now provide a brief overview of related work on skew in ad ery aspects of political messaging; prior work on filter bubbles for
delivery, filter bubbles, and political advertising. political content focused on possibilities of disparate treatment
of organic rather than sponsored content or disparate treatment
Skew in ad delivery Recently, concerns have been raised about during other parts of the process, such as during the ad review
how the platforms’ desire to show “relevant” ads to users may raise stage [5, 33].
issues of lack of fairness and lack of transparency. Recent work [4,
34] demonstrated that on Facebook, ads can be showed to skewed
subsets of the target audience, sometimes with dramatic effects on 4 METHODOLOGY
delivery (e.g., ads targeting the same audience but with different
content can be shown to over 95% women or less than 15% women, In this work we aim to answer two related, but separate questions,
depending only on the content of the ad and not on the advertiser’s and design our experiments accordingly. First, we want to verify
targeting choices or competition from other advertisers). In some whether the skew in delivery reported in previous work [4] exists
cases, skews in ad delivery may be unsurprising or desirable. In along the lines of political affiliation for political ads. To this end, we
others, such as civil rights areas and political ads, they can raise replicate the study setup from [4] as closely as possible, including
serious issues that demand research and, potentially, regulation. setting the campaign objective to “Traffic”. Second, we ask whether
a political campaign determined to reach users who may not be
Online political advertising Prior work, conducted mostly aligned with its views—and explicitly requesting such audience
in the form of laboratory experiments, indicated high efficiency from Facebook—can achieve their goal. To be able to better answer
of written persuasion personalized to the psychological profile this question, we set the campaign objective to “Reach”.
and motivation of the recipient [29, 56]. More recently, Matz et al. For the sake of clarity, we run ads for only one Democratic pres-
conducted a large-scale experiment in which they showed that idential candidate (Bernie Sanders) and compare their performance
Facebook ads tailored to individual’s psychological characteristics to that of the ads for only one Republican candidate (Donald Trump).
yielded higher click-through and conversion rates compared to non- We choose these two candidates because at the time of experiment
personalized ads and mismatching ads [36]. Matz et al. relied on design (early July 2019), they had spent most on Facebook adver-
the mechanism first documented by Kosinski et al.: that personality tising among the major candidates of each party [14]. Therefore,
traits of an individual can be accurately inferred from the content their election performance is least likely to be influenced by ads
that they “Like” on Facebook [32]. Other researchers, however, run on our limited budget.
pointed out that the unknown optimization mechanisms employed Next, we provide more details on the audiences and ad campaigns
by Facebook might obfuscate the measurement of effectiveness in our experiments, how we measured their performance over time,
of these personalized ads [11]. Our prior results [4] indicate that and the statistical apparatus necessary to interpret the results.
 CAA CAB CAC CAD
DMA(s) [38]
Dem Rep Dem Rep Dem Rep Dem Rep

Greensboro, Charlotte 70,000 0 0 70,000 70,000 0 0 70,000

Wilmington, Raleigh-Durham,
0 63,137 70,000 0 0 54,000 64,166 0
Greenville-(New Bern and Spartanburg)

Table 1: Number of uploaded records for Custom Audiences created using publicly available voter records. We divide the DMAs
in the state into two sets, and create two audiences, each with voters registered with one party per DMA set (CAA and CAB ).
We repeated this process with separate voter records (creating CAC and CAD ), allowing us to run experiments on separate
audiences. The number of uploaded records does not match, as we uploaded records so that the Estimated Daily Reach was
the same.

4.1 Creating audiences CAE CAF

DMAs [38]
We use two mechanisms for targeting audiences on Facebook: Cus- Trump Sanders Trump Sanders
tom Audiences and detailed targeting. donors donors donors donors
Recall that we are interested in studying the skew in delivery
along political lines. Since Facebook does not provide ad delivery DMA Set 1 40,973 0 0 32,000
breakdowns by political leaning, but does provide breakdowns by DMA Set 2 0 32,000 41,458 0
location (Section 2.7), we craft our Custom Audiences in such a Table 2: Overview of Custom Audiences built from public
way that the statistics about the political leanings of the actual FEC donor records and ActBlue. The number of uploaded
recipients of an ad can be inferred from the statistics about their records does not match, as we uploaded records so that the
location. Specifically, we follow the method introduced by prior Estimated Daily Reach was the same.
work [4, 46].
estimates have been shown to leak private information about indi-
Custom Audiences from voter records We obtained publicly viduals [51, 52]. Instead, we use the Estimated Daily Reach provided
available voter records from North Carolina, which, in addition to by Facebook with a budget set to a very high number (e.g., $1M/day),
PII, include each voter’s political party registration (if one exists). thereby obtaining an estimate of the total daily active users in the
We then create Custom Audiences as follows: uploaded audience.
(1) Divide the DMAs in North Carolina into two sets of roughly
equal population sizes. Custom Audiences from donor records We build separate
(2) Create a Custom Audience that contains PII of registered Custom Audiences from publicly available donor records for politi-
Democrats from the first set of DMAs and another Custom cal campaigns. The U.S. Federal Election Commission (FEC), in par-
Audience with PII of registered Republicans from the second. ticular, makes publicly available the PII of all contributors who have
(3) Upload the lists to Facebook, and compare their “Estimated donated a total of $200 or more towards a political campaign [23].
Daily Reach” statistics provided by Facebook. We obtain data from the FEC for individuals who have donated to
(4) If one audience has a higher Estimated Daily Reach, subsam- the “Bernie 2020" or the “Donald J. Trump for President" committees
ple it and re-upload until the estimates match. as of July 1, 2019 to craft Custom Audiences of users who actively
(5) Repeat steps (2)–(4) with the opposite assignment: regis- engage with these campaigns. Because there are fewer donors for
tered Republicans from the first set of DMAs and registered Bernie Sanders than for Donald Trump in FEC data, we also use
Democrats from the second set of DMAs. mid-year FEC filing by ActBlue, a popular Democratic fundraising
platform [7, 41] to obtain a list of Democratic donors who donated
Table 1 shows the summary statistics of the audiences created from less than $200. Since the FEC data isn’t limited to a particular state
voter records. or region, we randomly split all 210 U.S. DMA regions [38] into two
There are two distinctions between our method and previous sets, and then create Custom Audiences of approximately equal
work [4]. First, we introduce an additional step in an attempt to numbers of Democratic and Republican donors in each, by relying
create audiences consisting of roughly equal numbers of Democrats on the estimated daily reaches (as in Steps (2)–(4)) above. Table 2
and Republicans. We do so for ease and clarity of subsequent anal- shows the size and configuration of our audiences created from
ysis; it is not strictly necessary to do so, as we will compare the donor records.
delivery of two ads targeting the same audience and run at the same
time to observe differences due to ad delivery (thus, any differences Detailed targeting audiences Although the ability to create
in population, usage, or time of day will affect both campaigns Custom Audiences is only granted to advertisers with some history
equally). Second, Facebook no longer provides advertisers with the of running and paying for ads (the exact eligibility criteria are not
number of uploaded records that match Facebook users, as these publicly disclosed), all advertisers can specify their audience using
Figure 1: Ads used in our experiments concerning political issues and promoting candidates’ merchandise. The ads were copies
of real ads run on Facebook by the official campaigns, with the exception for Bernie Sanders related merchandise (as his store
has no official Facebook advertising).

detailed targeting (Section 2.2). We create a number of audiences running the ads as the campaign itself). Examples of the ad copies
this way, selecting a geographic region centered around a town and of types (1) and (2) that we ran are presented in Figure 1.
Facebook’s inferred characterization such as “Likely engagement
with US political content (Conservative)” and “Likely engagement 4.3 Isolating role of content
with US political content (Liberal)”. For some of the audiences we Most of our ads link directly to either a candidate’s official website
further narrowed the targeting by specifying additional required or generic voting information websites. In one of the experiments,
characteristics such as those who are, according to Facebook’s however, we wanted to isolate the effect that the content of the
characterization, “interested in” topics such as “Donald Trump advertised website has on the delivery skew, while keeping the
for President”, “Make America Great Again”, “Bernie Sanders”, or users’ reactions to the ad (such as possible Likes, comments, or
“Elizabeth Warren”. We aimed to approximately match the sizes of reactions) constant. We found that during ad creation, Facebook
liberal and conservative audiences for each geographic region by would automatically crawl the destination link as part of the ad
adjusting the targeting radius around a chosen location until the review and classification process. We develop a methodology that
Estimated Daily Reach matches. The Appendix presents the details would use this feature to create ads that look like they have the
and size statistics for these audiences. same content to users, but different content to Facebook.
To this end, we created a generic ad with a call to register to vote,
4.2 Creating ad copies a picture of the American flag, and a link to a nondescript domain:
We ran three types of ads throughout our campaigns: (1) merchan- psdigital.info (see Figure 2). We created three copies of this ad,
dise ads for candidates that link to the candidates’ online campaign with each copy having a destination link to a different page under
stores, (2) “issues ads” that have detailed content and that link to that domain. We configured our web server to deliver a different
the candidates’ websites, and (3) “neutral” political ads that simply response for requests for these pages based on the IP address of
encourage users to vote and link to generic voting information the requestor. If the requestor was a Facebook-owned8 IP address,
websites.7 The majority of ads we ran were replicated from real we served a copy of the HTML9 from the official Trump campaign
ads run by official political campaigns obtained from the Facebook website, the official Sanders campaign website, or a generic voting
Ad Library [14]. Ads for Bernie Sanders’ merchandise store were information website,10 depending on the particular page under
the only exception, as—unlike the other campaigns in question— our domain requested. Otherwise, if the requestor was from any
the Bernie Sanders campaign had not advertised merchandise on other IP address, the user would be immediately redirected to the
Facebook; we created the ad creative for this ad. Whenever the generic voting information website. In this way, all three ads would
replicated ad was written in the first person, we changed it to be a 8 We determined Facebook IP addresses by using the IP address blocks advertised by

third person reference to the name of the candidate (as we were not Autonomous Systems numbers owned by Facebook.
9 Only the HTML code was served from our server; we modified the HTML so that
all images, JavaScript, and stylesheets would be downloaded from the corresponding
official websites.
7 https://www.usa.gov/election and https://www.usa.gov/register-to-vote 10 https://www.usa.gov/register-to-vote
 where L.L. is the lower confidence limit, U .L. is the upper confi-
dence limit, p̂ is the observed fraction of Democrats in the audience,
n is the total size of the audience exposed to the ad. To obtain the
99% interval we set z α /2 = 2.576.
Second, we compare whether the fractions in two scenarios are
statistically significantly different. If their confidence intervals do
not overlap (easily judged visually from the subsequent figures),
the difference is statistically significant. If the intervals do overlap,
we need to perform a difference of proportion test as shown in
Equation (2):
(pˆ1 − pˆ2 ) − 0
Z= q (2)
p̂(1 − p̂)( n11 + n12 )
where pˆ1 and pˆ2 are the fractions of Democrats in the two audi-
ences, n 1 and n 2 are the total sizes of these audiences, and p̂ is
the fraction of Democrats in the two audiences combined. If the
resulting Z -score is above 2.576 (corresponding to 99% confidence)
the difference in proportion is statistically significant.

5 EXPERIMENTS
We now present the detailed set-ups and results of our experiments.
Recall that our aim is to study both (a) whether the content of a
Figure 2: Ads that have a destination link to our webserver political campaign’s ad could lead to skew in delivery along political
(psdigital.info), which serves HTML from the candidate lines, and, if so, (b) whether a political campaign can successfully
webpages to requests from Facebook’s IP addresses, but redi- reach users who Facebook believes are not aligned with the cam-
rects all other traffic to a generic voting information site. paign’s views. In the two subsections below, we address each of
The ads look identical to users, but different to Facebook. these questions in turn, before discussing the implications and
limitations of our study.
appear identical to users (and those users would all be brought
to the same voting information site if they clicked on the ad), but 5.1 Ad content and skew
Facebook’s algorithm believed they linked to pages with different
political content. We begin by examining whether the content of an ad can lead to
skew in delivery along political lines.
4.4 Collecting performance statistics Voter records Similar to methodology of prior work [4] for
As mentioned in Section 2.7, Facebook provides semi-live statistics studying skews along race and gender, we use the Custom Au-
on how the ad is delivering. Once an ad starts running, we query diences CAA , CAB , CAC , and CAD described in Table 1 that are
Facebook every five minutes in order to get these statistics over the based on voter records. These audiences are designed so that asking
lifetime of the ad. For ads where we use Custom Audiences with Facebook to report delivery statistics by DMA serves as a proxy for
DMAs as a proxy for political leaning, we request these delivery obtaining delivery statistics by political affiliation.
statistics be broken down by DMA. We create three ad creatives: one taken from the official Donald
Trump campaign, another from the Bernie Sanders campaign (both
4.5 Statistical analysis found in Facebook’s Ad Library [14], shown in Figure 1 and linking
The core questions in this work revolve around comparisons of the to the respective campaign’s web site), and a “neutral” political ad
fractions of Democrats (or Republicans) among the users exposed that simply encourages users to vote and links to a generic election
to two ads that differ in their content. The comparison process website.11 We then run one copy of each ad targeting each of the
consists of two steps and is based on previous work [4]. four Custom Audiences, for a total of 12 individual ads. Our ads
First, we estimate the fraction of Democrats in each ad and the are run with a daily budget of $20 per ad set and use the objective
99% confidence interval around that estimate as shown in Equa- “Traffic" and optimization “Link Clicks" (Section 2.3) as in prior
tion (1): work [4].
r Figure 3 (top row) presents the overall delivery statistics for these
z2 p̂(1−p̂) z2
α /2
p̂ + 2n − z α /2 α /2
+ 4n three ads, with the delivery statistics of all four instances of each ad
n 2
L.L. = , aggregated together. We can immediately observe significant differ-
1 + z α2 /2 /n ences in delivery: The neutral ad delivers to 47% Democrats, while
(1)
z α2 /2
r
z α2 /2
the Trump ad delivers to less than 40% Democrats. The Sanders
p̂(1−p̂)
p̂ + 2n + z α /2 n + 4n 2 ad, on the other hand, delivers to almost 70% Democrats. Note that
U .L. = ,
1 + z α2 /2 /n 11 https://www.usa.gov/election
 Sanders Neutral Trump Trump Sanders C L
Voter records Issues, Broad, 6
Issues, Narrow, 1
Donor records Issues, Narrow, 7
0.2 0.3 0.4 0.5 0.6 0.7 0.8 Merch, Broad, 2
Estimated fraction of Democrats in the audience Merch, Broad, 3
Merch, Broad, 6
Figure 3: The estimated fraction of Democrats who were Merch, Narrow, 1
shown our ads, targeting both registered voters in North Car- Merch, Narrow, 4
olina and political donor records. In the case of voter records, Merch, Narrow, 8
the ad delivery to Democrats ranges from approximately 0 0.5 10 10000 20000
69% for Sanders’ ad to only 39% for the Trump’s ad. In the Estimated fraction Estimated
of Democrats in the audience audience size
case of donor records, we do not see statistically significant
differences in ad delivery.
Figure 4: We ran merchandise and issue ads with two levels
of targeting specificity (Broad: users with “Likely engage-
this difference in delivery is despite the fact that all ads are run ment with US political content (Conservative)” or “... (Lib-
from the same ad account, at the same time, targeting the same eral)"; Narrow: additional detailed targeting for inferred in-
audiences, and using the same goal, bidding strategy, and budget; terest in Donald Trump or Bernie Sanders), and targeting
the only difference between them is the content and destination link different regions (1: Celina, OH; 2: Dutchess, NY; 3: Lorain,
of the ad. OH; 4: Macclenny, FL; 5: McCormick, SC; 6: Richlands, VA;
7: Saginaw, MI; 8: Slinger, WI). In all cases, Sanders’ ads de-
Donor records Having observed that delivery skew along polit- liver to a larger fraction of Democrats than Trump ads even
ical lines can occur due to the content of the ad, we next turn to though they are targeting the same audiences at the same
examine whether that skew is amplified if we choose users who time using the same budgets. The effect is more pronounced
recently engaged with politics. In particular, we examine whether for smaller audiences (compare, for example Merch, Broad,
recent donors to political campaigns are estimated by Facebook to 3 and Merch, Broad, 6).
have greater relevance for our ads, when compared to users who
are simply registered as Democratic or Republican voters. Thus,
we use our Custom Audiences of donor records (CAE and CAF ,
described in Table 2); however, due to the limited size of the donor conservative users. For this, and all further experiments, we op-
record databases, we are only able to run our experiment on two, timize for “Reach”, not “Traffic”. To calculate the delivery skew
and not four, audiences. Thus, we run six ads in the same manner of a politician’s ads, we sum reach across the two audiences, and
as the experiment we just described. calculate the fraction of deliveries to the liberal audience.
The results of this experiment are presented in Figure 3 (bottom The result of our first experiment is shown in the top row of
row). Surprisingly, we do not find statistically significant differences Figure 4. We can immediately observe similar skews in delivery
in the ad delivery between the three ads targeting political donors. to the ones observed for voter records, with the content of the ad
While we can only speculate as to why we observe a skew with causing delivery skew along political lines. This indirectly suggests
voter records but not with donor records, the absence of a skew that our hypothesis for the reasons behind differences for voter vs
for the donor record audiences might suggest that Facebook does donor records could have some merit.
not have sufficient information about these users to do accurate Next, we explore this finding in depth, varying three aspects of
relevance estimation for political ads. our experiment:
(1) The size of the audience, as reported by Facebook’s Estimated
Detailed targeting To further explore the role of Facebook’s use Daily Reach,
of inferences about its users in delivery and its impact on political
(2) The “specificity” of the audience (narrowing the detailed tar-
ad skew, we next use audiences where we know that Facebook has
geting further by attributes such as users’ inferred interest in
inferred the political affiliation of its users. We do so using detailed
“Donald Trump for President” or “Bernie Sanders” according
targeting (Section 2.2), selecting attributes “Likely engagement with
to Facebook), and
US political content (Conservative)” for one audience and “Likely
engagement with US political content (Liberal)” for another. As (3) The specific topic of the ad (adding ads that advertise small
discussed in Section 4.1, we then geographically limit our targeting campaign-branded merchandise that users can purchase, as
to regions where we can ensure an approximately equal number of shown in Figure 1).
users in each audience (as shown by Facebook’s audience size esti- The results for these experiments are shown in Figure 4 (remaining
mates). Then, over a course of six hours we concurrently ran two rows), with each row representing a separate experiment. Experi-
ad copies, each to two audiences (a total of four ads): one Sanders ments described as “issues” are run with the first two ad creatives
ad targeting liberal users and another targeting conservative users, from Figure 1 and “merch” ads correspond to the third and fourth
and one Trump ad targeting liberal users and another targeting creative in Figure 1.
 Sanders Neutral Trump delivery mechanism to ‘exhaust’ the audience rather than showing
A B the ad to the same subset of users.
Narrow targeting
Generic ads We begin by applying the method described in
Broad targeting
Section 4.3, setting up ads that look identical to Facebook users
0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 (Figure 2) and, when clicked, redirect the user to a governmental
Estimated fraction Cost penalty website with instructions to register to vote. However, when visited
of Democrats in the audience for non-alignment
by Facebook’s web crawler, each ad’s landing website shows differ-
ent HTML: one serves Trump’s campaign HTML, another Sanders’,
Figure 5: Delivery statistics for ads that look identical to and a third a generic voting information site. Since all ads look
users, but appear partisan to the Facebook classification identical to users, any skew can only be attributed to Facebook’s
mechanism. (A) The skew in delivery is consistent with optimization based on the content of the linked website.
that observed in visibly distinct ads. (B) There is a financial Each ad copy is targeted at four audiences: two that are Broad
penalty for trying to show an ad that Facebook deems non- and two that are Narrow. The Broad audiences target “Likely en-
aligned; reaching the same number of people in the same gagement with political content (Liberal)” and “...Conservative”,
audience is up to 1.5 times more expensive. and the Narrow audiences additionally target users with interest in
Bernie Sanders and Donald Trump, respectively. Table 3 entries for
Oxford, NC and Scranton, PA provide detailed audience parameters
We make a number of observations from this experiment. First, and size statistics for the Broad and Narrow audiences respectively.
we observe statistically significant skews in ad delivery along polit- Since we are attempting to reach everyone in the targeting set here,
ical lines for all of our ad configurations. This suggests that such we set a higher budget $40 per day for each ad. These ads were run
skew is a pervasive property of Facebook’s ad delivery system. Sec- for two days and did not fully exhaust the audiences. The results
ond, we observe that the skews tend to be less pronounced when of this experiment are presented in Figure 5A. Even though the
the ads are targeting larger audiences (more than 10,000 daily ac- users see the same ad in all three cases, and therefore their explicit
tive users). While we do not know the underlying cause of this or implicit reactions to them are not more different than chance,
phenomenon, we hypothesize that the larger audiences provide the delivery is still skewed according to what Facebook’s crawler
the platform with a big enough pool of users to afford “relevant” sees. We also present the price differentiation in Figure 5B. For a
users regardless of their inferred political leaning. On the other given audience, we measure how much it cost for the aligned ad
hand, we suspect that when running our ads with smaller audiences, to reach the same number of users as the non-aligned ad did. For
Facebook “exhausts” the (small) subset of users in the non-aligned example, it cost 1.5× more for the ad linking to Sanders’ campaign
audience (e.g., Sanders advertising to a conservative audience) for page (as perceived by Facebook) to reach the same number of users
whom Facebook believes the ad is, in fact, relevant, and thus pauses in the Broad conservative audience than the ad linking to Trump’s
or raises the price for delivery, but continues the delivery among campaign page. Conversely, it cost 1.2× more for the ad linking to
the aligned audience. We explore this hypothesis in more detail in Trump’s campaign page to reach the same number of people in the
the next experiment. Broad liberal audience than the ad linking to Sanders’ campaign
Overall, our findings strongly support our hypothesis that the page.
content of an ad could lead to skew in its delivery along political These results show that the contents of the destination link—
lines whenever the platform has enough information (or thinks it and not users’ reaction or engagement with the ad—play a role in
has enough information) about the political leanings of the users Facebook’s decision for skewed delivery and differential pricing.
being targeted, and that the skew is due to ad delivery optimiza- An implication of this finding is that two campaigns running an ad
tion algorithms run by Facebook, rather than to other factors. As about the same issue to the same target audience might reach differ-
discussed in the introduction, this has profound implications for ent fractions of that audience and at different prices, only because
political advertisers, users, and society. the destination links are different. This differential delivery and
pricing may be particularly damaging for local political campaigns,
5.2 Longitudinal delivery where candidates may agree on some issues but not others.
We now explore what happens if a political campaign aims to reach Real ads We now turn to explore how this effect plays out for
users who Facebook believes are not aligned with the campaign’s real-world ads that differ in content and destination link. In this
views. Specifically, we “force” the Facebook ad platform to consider experiment, we run three ads (Trump, Sanders, and neutral issue
showing ads to all users in the political advertiser’s targeting set, ads as before), each to two Narrow audiences over a period of seven
including users for whom Facebook may believe the ad is not rele- days and with a daily budget of $100 for each ad and audience
vant. We do so in two steps: First, on a small audience we measure combination.12 The ad copies are the first two presented in Figure 1,
whether skew appears even if the ads look the same to users but and details about the target audiences are provided in Table 3 (the
differently to Facebook. Second, we run near-copies of real ads of audiences from Michigan and Wisconsin). The conservative and
political campaigns on a larger audience to measure the total ef- liberal audiences were selected such that they had approximately
fect. In both cases, we configure the campaigns using the objective
“Reach” and optimization “Reach”, which enables us to tell Facebook 12 Our
total spend over the week ended up being $4,228.19 distributed roughly equally
to only show the ad once to each user each week, thus forcing the among Sanders, Trump, and neutral ads.
 Sanders; liberal audience Trump; conservative audience Neutral; liberal audience
Sanders; conservative audience Trump; liberal audience Neutral; conservative audience
A C E G

Cumulative cost penalty

40 40
3
Cumulative reach

20000

Cummalitve cost

Cumulative cost
per 1000 users

per 1000 users

30
10000 20 20 2

10 Equal cost
0 0 1
5000 B 200 D F 5 H

Running cost penalty

per 1000 users

per 1000 users

102
Running cost

Running cost
150 4
reach

2500 100 3
50 2
101
0 0 1 Equal cost
0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 10000 20000 5k 10k 15k
Campaign duration [days] Campaign duration [days] Users reached Users reached

Figure 6: Ads for a political campaign deliver to more users and for a lower cost if the targeted users have the same partisanship.
A and B - the delivery rates are the highest in the beginning of the ad runtime and for aligned audiences. C and D - the cost
of reaching non-aligned audiences is higher, especially in the beginning of the experiment. E and F - the more people have
already seen the ad, the more expensive it becomes to show it to even more people; that growth is log-linear (see F). G and H
show the ratio between the cost of a political campaign advertising to a non-aligned audience and their competitor advertising
to the same audience.

the same daily active reach, and such that we expected our ads to per thousand users, a difference noticeable from the first day of the
have reached almost everybody in the audience by the end of the experiment. By the end of the experiments, when the liberal ad is
seven days. shown to the liberal audience, it is charged $21 per thousand users;
The results of this experiment are presented in Figure 6. We first when the conservative ad is delivered to the same audience, it is
focus on panel A, which shows the cumulative number of users charged over $40 per thousand users.
reached over seven days (along with its derivative in panel B). We Because the delivery rates slow down after the first day plots C
can observe that the delivery increases rapidly for all ads during the and D make the growth of cost per 1,000 also appear to slow down.
first day and then slows down quickly. However, we can observe Therefore, we turn to plots E and F, which show this growth as
two notable outliers in panel A: the Trump ad targeting the liberal a function of the size of reached audience, rather than time. We
audience and the Sanders ad targeting the conservative audience. observe that the growth is rapid, and the running cost per 1,000
Both of these non-aligned ads end up delivering to over 25% fewer is growing exponentially as a function of the number of users
users than their aligned counterparts. In other words, when the reached. Finally, in plots G and H we show that the ratio between
Trump ad is advertised to the conservative audience, it delivers to the cost a political campaign pays to show their ad to the non-
a total of 21,792 users; when the Sanders ad is run at the same time aligned audience and the cost of their competitor showing to the
and targeted to the same conservative audience, it delivers to only same audience is relatively stable, between 2:1 and 4:1.
17,964 users. Note that this difference cannot be attributed to some Overall, Figure 6 emphasizes three findings: First, the penalty
unknown underlying difference between Facebook use between for reaching a non-aligned audience remains at a relatively stable
the users categorized as liberal and conservative by Facebook be- ratio between 2:1 and 4:1 as a function of audience already reached
cause the neutral ad delivers equally to liberals and conservatives, (see Figure 6H). Second, that while the cost per thousand view-
reaching approximately 23,000 users. ers grows linearly with time (Figure 6D), it grows super-linearly
We turn to panel C, which shows the cumulative cost per thou- with the number of users already reached (Figure 6F). For example,
sand unique users to help explain why this is occurring. We can panel F shows that the cost of showing the Sanders ad to the first
immediately notice an increasing cost trend for all ads: as the ads 1,000 liberal users (solid blue line) is approximately $5, and the
run longer, their cost increases substantially. Presumably, this is cost of reaching the first 1,000 conservative users with this ad is
because Facebook first delivers the ad to the “cheaper” users in the approximately $10. However, once 10,000 users in each of these
target audience before deciding to spend our budget on the more audiences are already reached, reaching another thousand of liberal
“expensive" users. However, we can observe that the non-aligned users costs approximately $15 (a three-fold increase) and reaching
ads are again outliers here: both show a substantially higher cost another thousand of conservative users costs approximately $37
 Oxford, NC Richlands, VA advertising are not mere hypotheticals and warrant further scrutiny
40 40 (Section 6).
Cumulative cost
per 1000 users

30 30 Role of advertiser’s identity We have repeated a subset of

20 20 our experiments using another advertising account registered as
an advertiser in the area of “Social Issues, Elections, and Politics”
10 10 and linked to a Facebook page unrelated to the first. Our results
were quantitatively and qualitatively similar. This suggests that
0 0
1 2 3 4 5 10 15 the effects we observed were not tied to our particular advertising
account. Nevertheless, we do not make any statements about the
2.0 2.0 extent to which the observed effects hold when run by real political
Cumulative

campaigns with a more established history than ours.

cost ratio

1.5 1.5 Role of budget We also re-ran a subset of our experiments with
varying lifetime budgets ranging from $10–$100 per campaign, and
1.0 1.0 with a generous bid cap of $10 in each auction. Our ads ran on
1 2 3 4 5 10 15 consecutive weekdays at similar times; we observed qualitatively
Users reached [thousands] Users reached [thousands] similar skews regardless of the budget. Although $100 per ad may
seem small compared with the total political ad spending, such
ads are representative of practice: recent work [12] that analyzes
Figure 7: The price penalty for showing to non-aligned au-
data from Facebook’s political ad archive has found that 82% of all
dience is not just an effect of a particular location or tar-
political ads spend less than $100.
geting more “extreme” (Narrow) audiences. Findings from
Figure 6 hold also in other cities and with weaker targeting Role of competition We ran each pair of campaigns targeting a
(here: only political alignment, without additional interests). particular audience representing two different political campaigns
We note that in some cities Trump’s ads are cheaper for his at the same time and with the same budget. Such a set-up is de-
aligned audience than Sanders’ ads are for his. signed to ensure that both campaigns have the same users available
for delivery (i.e., if run at different times, the skews could be attrib-
(nearly a four-fold increase compared to the first thousand conser- uted to different Facebook use patterns by liberals or conservatives)
vatives). Third, at the end of the experiment, both neutral ads and and both are experiencing the same competition from other ad-
the two aligned partisan ads reached over 20,000 users, while the vertisers (i.e., that it would not be the case that one campaign is
non-aligned ads reached significantly fewer. This demonstrates the under-performing because it happened to run at the same time that
core phenomenon: it is cheaper and more effective for a political another large and wealthy advertiser was targeting those users,
campaign to reach audiences that are politically aligned (as inferred whereas another campaign avoided such a collision). Thus, run-
by Facebook) with their agenda, and as the campaign progresses it ning campaigns simultaneously is an effective strategy to isolate
becomes more expensive to reach additional viewers. the effects of delivery optimization from other extraneous factors.
Finally, we run a similar experiment targeting Broad audiences However, to verify that the skews are not merely the effect of our
(i.e., audiences without specific interest in candidates). The results ads competing with each other, we also re-ran a subset of campaigns
of this experiment are presented in Figure 7. We find that the core separately. The qualitative and quantitative skew effects for those
phenomenon holds there as well, and note that in some cases, the campaigns we similar.
conservative audience is cheaper for Trump than the liberal audi-
ence is for Sanders. Audience sizes We aimed to match our constructed liberal and
conservative audiences in size as closely as possible, but the matches
5.3 Limitations are inevitably imprecise as Facebook only provides estimates of daily
reach13 rather than audience sizes.
We now discuss limitations of our study and briefly mention the
steps we took to mitigate them when possible. Controlling for all User engagement with our ads There are a number of ways
possible variables that may affect political ad delivery is beyond users can engage with the ads we present, each of which poten-
the scope and financial capabilities of our work, and is better suited tially influences future delivery and pricing: reactions (‘like’, ‘love’,
to be performed by Facebook itself or by an independent third- ‘haha’, ‘wow’, ‘sad’, ‘angry’), commenting, and sharing. Facebook
party auditor that would be granted broader data and algorithms’ advertising interface reports all such engagements. Additionally,
access than what is available through the ad interface. Similarly, it is Facebook might be collecting and using telemetric information; for
important to note that we can only report on delivery skew that we example, how long each user spent looking at the ad. This tele-
observed for our own ads; we cannot draw any conclusions about metric information is not available to the advertisers (and thus,
how political ads in general (or all ads run by a particular campaign) neither to us), but might still play a role in ad delivery optimization
are delivered. Nonetheless, the fact that we observe strong and algorithms.
statistically significant effects in our small set of ads suggests that
the potential negative outcomes for individuals, political campaigns,
and society in the context of ad delivery optimization of political 13 https://www.facebook.com/business/help/1691983057707189?helpref=faq_content
 Some of our ads received reactions, comments, and re-shares to be aligned with Facebook’s business interests, but not necessarily
from the users they were delivered to. We note four important, with important societal goals.
related observations, that emphasize that our findings about skew Third, today, researchers, regulators, and campaigns lack access
in delivery and differential pricing are not merely a function of the to algorithms and data required for a more thorough study of ad
ad delivery algorithm’s use of user engagement. First, we observe delivery skews and their likely impacts. In particular, although
consistent skew and price differences in ads that look identical to much has already been said about the inadequacy of current ad
users, yet trick Facebook into classifying them as partisan (Fig- transparency tools provided by ad platforms [19, 37, 50], our work
ure 5). Users do not react differently to ads that appear identical, draws attention to the need to expand these efforts to account for
and, therefore, the entire observed difference can be attributed to ad delivery algorithms as well.
Facebook’s pre-delivery classification (and some random effects).
Second, we observe consistent skew in delivery of ads that had Policy analysis Today, U.S. law cannot do much, if anything,
virtually no engagement since they were run on small budgets and to directly change how ad platforms deliver political ads. For the
only for a few hours, as shown in Figure 4. Third, longitudinal ads foreseeable future, it is likely that the primary regulator of digital
with neutral content proved less engaging than either aligned or political advertising will not be the government, but rather ad
not-aligned ads, yet they eventually reached larger audiences and platforms themselves.
at lower prices (Figure 6). Specifically, non-aligned ads were shared The U.S. Congress has addressed conceptually similar "ad deliv-
at higher rates than neutral ads (0.34% vs 0.03% for conservative ery issues" in the past, albeit in a different domain. For example, the
audience, 0.19% vs 0.05% for the liberal audience). This leads us to Federal Communications Commission (FCC) enforces the so-called
believe that the relatively lower costs of aligned ads compared to Equal-Time Rule [1], which originated in 1927 in response to wor-
non-aligned ads do not stem from “free” exposures originating from ries that broadcast licensees could unduly influence the outcome of
re-shares. Finally, we do find a negative correlation between the elections. The rule requires that licensees make air time available
fraction of positive reactions (“like” and “love”) among all reactions to all candidates for the same office on equivalent terms. However,
and the price in the longitudinal ads with ρ = −0.91, pval = 0.01. the rule only applies to broadcast licensees, and has only narrowly
Taken together, our work demonstrates although the skew in deliv- survived constitutional scrutiny in part because it implicates gov-
ery as well as differential pricing can be further amplified during ernment interests in managing limited broadcast spectrum [8].
the course of delivery by users’ reactions, the primary reason stems Prevailing interpretations of the First Amendment are likely to
from Facebook’s ad delivery optimization’s use of classification of block efforts to extend the logic of the Equal-Time Rule to digital ad-
an ad and its landing page content. vertising platforms, which are not regulated like broadcast licensees.
We leave a more precise quantification of the influence of users’ As an initial matter, the First Amendment strongly protects political
interactions with the ads on Facebook’s ad delivery and pricing speech, and generally tolerates only narrowly-tailored government
algorithms to future work. regulations [30]. This protection is so strong that legal scholars
cannot even be confident that lighter-touch kinds of regulations—
for example, a requirement that social media users be entitled to
6 DISCUSSION
opt-in to micro-targeted political advertising—would survive con-
Our findings suggest that Facebook is wielding significant power stitutional scrutiny. Moreover, the Supreme Court recently declared
over political discourse through its ad delivery algorithms without that “the creation and dissemination of information" constitutes
public accountability or scrutiny. speech under the First Amendment [45]. This reasoning, which
Implications First, Facebook limits political advertisers’ abil- might expand the “commercial free speech” rights of companies,
ity to reach audiences that do not share those advertisers’ politi- creates some uncertainty about the government’s ability to restrict
cal views in ways that are significantly different from traditional corporations’ use of data in the context of digital advertising.
broadcast media. The existence and extent of this skew may not be Looking ahead, it is clear that government regulation of digital
apparent to advertisers and varies based on their ad’s message and political advertising is on firmest legal footing when it requires dis-
the destination link used by the campaign. For example, a campaign closure about who is speaking to whom, when, and about what [30].
targeting a certain geographic region might reasonably expect to Accordingly, Congress and the FEC can consider transparency re-
reach an audience whose political views are representative of users quirements that will enable detailed auditing and research about
in the region. To discover otherwise would require careful research, ad targeting and the delivery of political ads.
as we have demonstrated in this study. Furthermore, the strength Mitigations As an initial data, the public and the campaign
of delivery skews vary for campaigns of different political lean- managers need more information about the operation of ad delivery
ings and targeting different populations, making digital advertising algorithms and their real-world effects. Ad platforms could increase
inequitable for political campaigns with identical budgets. transparency around political ads (including key metrics such as
Second, recent moves to restrict political advertisers’ targeting targeting criteria, detailed ad metadata, ad budgets, and campaign
options [16, 26, 48], although valuable from a user privacy per- objectives) to enable further study of the effects of ad targeting
spective [22, 31, 46], might be undermined by the operation of ad and delivery. And they could provide access to and insight into
delivery algorithms, and even give companies like Facebook more the ad delivery algorithms themselves (including those involved
control over selecting which users see which political messages. in running the auction, relevance measurement and estimation,
This selection occurs without the users’ or political advertisers’ and bid and budget allocation on advertisers’ behalf), allowing
knowledge or control. Moreover, these selection choices are likely
third parties greater ability to study and audit their performance [12] Edelson, L., Sakhuja, S., Dey, R., and McCoy, D. An Analysis of United States
and effect on political discourse. Without these and similar steps, Online Political Advertising Transparency. https://arxiv.org/abs/1902.04385.
[13] Eytan, B., Solomon, M., and A, A. L. Exposure to ideologically diverse news
policymakers and the public will be unable to formulate appropriate and opinion on Facebook. Science 348, 6239 (2015), American Association for the
responses. Advancement of Science.
[14] Facebook Ad Library. https://www.facebook.com/ads/library/.
Ad platforms could also disable delivery optimization for po- [15] Facebook Lets Advertisers Exclude Users by Race. https://www.propublica.org/
litical content, or a least allow advertisers to do so. They could article/facebook-lets-advertisers-exclude-users-by-race/.
also introduce more nuanced user-facing controls for political con- [16] Facebook Weighs Steps to Curb Narrowly Targeted Political Ads.
https://www.wsj.com/articles/facebook-discussing-potential-changes-to-
tent delivery and expand public ad archives to make them more political-ad-policy-11574352887.
accessible and usable by everyone. [17] Facebook for Government, Politics & Advocacy. https://www.facebook.com/gpa.
Finally, we call on ad platforms to acknowledge the central role [18] Facebook for Government, Politics & Advocacy — Getting Started with Facebook
Ads. https://www.facebook.com/gms_hub/share/getting-started-with-facebook-
they play in the delivery of political ads, and to collaborate with ads_english-us-.pdf.
other key stakeholders—including researchers, political campaigns, [19] Facebook’s Ad Archive API is Inadequate. https://blog.mozilla.org/blog/2019/04/
29/facebooks-ad-archive-api-is-inadequate/.
journalists, law, policy and political philosophy scholars—to address [20] Facebook: About Facebook Pixel. https://www.facebook.com/business/help/
that role when it is not aligned with public interests. 742478679120153.
[21] Facebook: About the delivery system: Ad auctions. https://www.facebook.com/
business/help/430291176997542.
ACKNOWLEDGEMENTS [22] Faizullabhoy, I. and Korolova, A. Facebook’s Advertising Platform: New At-
We would like to thank Dean Eckles and David Lazer for their in- tack Vectors and the Need for Interventions. Computing Research Repository (Mar.
2018), https://arxiv.org/abs/1803.10099, Workshop on Technology and Consumer
valuable insights. We are also extremely grateful to the participants Protection (ConPro).
and organizers of the REAL ML workshop for their encouragement [23] Federal Election Commission — Individual Contributions. https://www.fec.gov/
and constructive feedback. This work was done, in part, while Alek- data/receipts/individual-contributions.
[24] Flaxman, S. R., Goel, S., and Rao, J. M. Filter bubbles, echo chambers, and
sandra Korolova was visiting the Simons Institute for the Theory online news consumption. Public Opinion Quarterly 80 (2015).
of Computing, where she benefited from supportive feedback from [25] Fowler, E. F., Franz, M. M., Martin, G. J., Peskowitz, Z., and Ridout, T. N.
Political Advertising Online and Offline. https://web.stanford.edu/~gjmartin/
participants of the Privacy and Fairness programs, and particularly, papers/Ads_Online_and_Offline_Working.pdf.
from the suggestions of Amos Beimel and Kobbi Nissim. This work [26] Google: An update on our political ads policy. https://www.blog.google/
was funded in part by a grant from the Data Transparency Lab, NSF technology/ads/update-our-political-ads-policy.
[27] Green, J. and Issenberg, S. Inside the Trump Bunker, with Days to Go. In
grants CNS-1616234, CNS-1916020, and CNS-1916153, and Mozilla Bloomberg (2016). https://www.bloomberg.com/news/articles/2016-10-27/inside-
Research Grant 2019H1. the-trump-bunker-with-12-days-to-go.
[28] Hannak, A., Sapiezyński, P., Molavi Kakhki, A., Krishnamurthy, B., Lazer,
D., Mislove, A., and Wilson, C. Measuring Personalization of Web Search. In
ERRATA International World Wide Web Conference (Rio de Janeiro, Brazil, May 2013).
v3: Clarified the discussion on efficacy of personality-based target- [29] Hirsh, J. B., Kang, S. K., and Bodenhausen, G. V. Personalized persuasion:
Tailoring persuasive appeals to recipients’ personality traits. Psychological science
ing. 23, 6 (2012), Sage Publications Sage CA: Los Angeles, CA.
v2: We clarified the optimization goal used in the experiments: [30] K, W. A. and M, R. A. Fool Me Once: Regulating Fake News and Other Online
Advertising. Southern California Law Review 91 (2017), HeinOnline.
Traffic in experiments with donors and registered voters and Reach [31] Korolova, A. Privacy Violations Using Microtargeted Ads: A Case Study. Journal
in all other experiments. of Privacy and Confidentiality 3, 1 (2011).
[32] Kosinski, M., Stillwell, D., and Graepel, T. Private traits and attributes are
predictable from digital records of human behavior. PNAS: Proceedings of the
REFERENCES National Academy of Sciences of the United States of America 110, 15 (2013).
[1] 47 USC §315 – Candidates for public office. https://www.law.cornell.edu/uscode/ [33] Kreiss, D. and Mcgregor, S. C. The “Arbiters of What Our Voters See": Facebook
text/47/315. and Google’s Struggle with Policy, Process, and Enforcement around Political
[2] About Twitter Ads approval. https://business.twitter.com/en/help/ads-policies/ Advertising. Political Communication 0, 0 (2019), Routledge.
introduction-to-twitter-ads/about-twitter-ads-approval.html. [34] Lambrecht, A. and Tucker, C. E. Algorithmic bias? An empirical study into
[3] Ads About Social Issues, Elections or Politics. https://www.facebook.com/ apparent gender-based discrimination in the display of STEM career ads. (2018).
business/help/208949576550051. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2852260.
[4] Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., and Rieke, A. [35] Mario, H., Andreas, G., and Hans-Bernd, B. Burst of the filter bubble? Effects
Discrimination through Optimization: How Facebook’s Ad Delivery Can Lead of personalization on the diversity of Google News. Digital journalism 6, 3 (2018),
to Biased Outcomes. Proceedings of the ACM on Human-Computer Interaction 3, Taylor & Francis.
CSCW (2019), ACM. [36] Matz, S. C., Kosinski, M., Nave, G., and Stillwell, D. J. Psychological targeting
[5] An Update on Senator Kyl’s Review of Potential Anti-Conservative Bias. https:// as an effective approach to digital mass persuasion. PNAS: Proceedings of the
newsroom.fb.com/news/2019/08/update-on-potential-anti-conservative-bias/. National Academy of Sciences of the United States of America 114, 48 (2017).
[6] Bail, C. A., Argyle, L. P., Brown, T. W., Bumpus, J. P., Chen, H., Hunzaker, [37] More than 200 Researchers Sign Letter Supporting Knight Institute’s
M. F., Lee, J., Mann, M., Merhout, F., and Volfovsky, A. Exposure to opposing Proposal to Allow Independent Research of Facebook’s Platform.
views on social media can increase political polarization. PNAS: Proceedings of https://knightcolumbia.org/news/more-200-researchers-sign-letter-
the National Academy of Sciences of the United States of America 115, 37 (2018). supporting-knight-institutes-proposal-allow-independent/.
[7] BuzzFeedNews. Analysis of ActBlue contributions to presidential candi- [38] Neilson DMA® Regions. https://www.nielsen.com/intl-campaigns/us/dma-maps.
dates, January-June 2020. https://github.com/BuzzFeedNews/2019-08-actblue- html.
donations. [39] Pariser, E. The Filter Bubble: What the Internet is Hiding from You. (London,
[8] CBS, Inc. v. FCC. https://supreme.justia.com/cases/federal/us/453/367/. UK, 2011), Penguin Press.
[9] Creating Customer Lists with Mobile Advertiser IDs. https://developers.facebook. [40] Political Ad Spending Will Approach $10 Billion in 2020, New Forecast Pre-
com/docs/app-ads/targeting/mobile-advertiser-ids/. dicts. https://www.wsj.com/articles/political-ad-spending-will-approach-10-
[10] Datta, A., Tschantz, M. C., and Datta, A. Automated Experiments on Ad billion-in-2020-new-forecast-predicts-11559642400.
Privacy Settings: A Tale of Opacity, Choice, and Discrimination. In Privacy [41] Report for Receipts and Disbursements, ActBlue Mid-Year Report, Filing FEC-
Enhancing Technologies Symposium (Philadelphia, Pennsylvania, USA, June 2015). 1344765. https://docquery.fec.gov/cgi-bin/forms/C00401224/1344765/.
[11] Eckles, D., Gordon, B. R., and Johnson, G. A. Field studies of psychologically [42] Ribeiro, F. N., Saha, K., Babaei, M., Henriqe, L., Messias, J., Benevenuto,
targeted ads face threats to internal validity. PNAS: Proceedings of the National O. G. F., Gummadi, K. P., and Redmiles, E. M. On Microtargeting Socially Divisive
Academy of Sciences of the United States of America 115, 23 (2018).
 Ads: A Case Study of Russia-Linked Ad Campaigns on Facebook. In Conference [51] Venkatadri, G., Liu, Y., Andreou, A., Goga, O., Loiseau, P., Mislove, A., and
on Fairness, Accountability, and Transparency (Atlanta, Georgia, USA, Jan. 2019). Gummadi, K. P. Privacy Risks with Facebook’s PII-based Targeting: Auditing a
[43] Sheryl Sandberg delivered a passionate, defiant defense of Facebook’s busi- Data Broker’s Advertising Interface. In IEEE Symposium on Security and Privacy
ness. https://www.cnbc.com/2018/04/26/facebooks-sheryl-sandbergs-brilliant- (San Francisco, California, USA, May 2018).
defense-of-the-ad-business.html. [52] Venkatadri, G., Lucherini, E., Sapiezyński, P., and Mislove, A. Investigating
[44] Showing Relevance Scores for Ads on Facebook. https://www.facebook.com/ sources of PII used in Facebook’s targeted advertising. In Privacy Enhancing
business/news/relevance-score. Technologies Symposium (Stockholm, Sweden, July 2019).
[45] Sorrell v. IMS Health Inc. https://supreme.justia.com/cases/federal/us/564/552/. [53] Venkatadri, G., Sapiezyński, P., Redmiles, E. M., Mislove, A., Goga, O.,
[46] Speicher, T., Ali, M., Venkatadri, G., Ribeiro, F. N., Arvanitakis, G., Ben- Mazurek, M., and Gummadi, K. P. Auditing Offline Data Brokers via Face-
evenuto, F., Gummadi, K. P., Loiseau, P., and Mislove, A. On the Potential book’s Advertising Platform. In International World Wide Web Conference (San
for Discrimination in Online Targeted Advertising. In Conference on Fairness, Francisco, California, USA, May 2019).
Accountability, and Transparency (New York, New York, USA, Feb. 2018). [54] Weintraub, E. L. Don’t abolish political ads on social media. Stop microtargeting.
[47] Trump campaign drops more than $1 million on Facebook ads in counter impeach- The Washington Post (Nov. 2019). https://www.washingtonpost.com/opinions/
ment push. https://abcnews.go.com/Politics/trump-campaign-drops-million- 2019/11/01/dont-abolish-political-ads-social-media-stop-microtargeting/.
facebook-ads-counter-impeachment/story?id=65962584. [55] What it means when your ad is pending review. https://www.facebook.com/
[48] Twitter Will Ban All Political Ads, C.E.O. Jack Dorsey Says. https://www.nytimes. business/help/204798856225114.
com/2019/10/30/technology/twitter-political-ads-ban.html. [56] Youngme, M. Personalization and personality: Some effects of customizing
[49] United States v. Internet Research Agency. https://www.justice.gov/file/1035477/ message style based on consumer personality. Journal of Consumer Psychology
download. 12, 4 (2002), Elsevier.
[50] Upturn. Leveling the Platform: Real Transparency for Paid Messages on Face-
book. (2018). https://www.upturn.org/reports/2018/facebook-ads/.
 Location Targeting Size APPENDIX
Celina, OH Engagement: liberal
1,500
(+25 mi) Interests: Bernie Sanders Audiences from Facebook’s targeting attributes. Table 3
Celina, OH Engagement: conservative shows the geographical location, targeting parameters, and esti-
1,400
(+21 mi) Interests: Donald Trump for President mated potential reach given by Facebook. In each case, the geo-
Dutchess
Engagement: liberal 15,000
graphical diameter and targeting specificity is tweaked to have
County, NY
roughly equal sized liberal and conservative audiences.
Dutchess
Engagement: conservative 15,000
County, NY

Loraine, OH
Engagement: liberal 20,000
(+13 mi)

Loraine, OH
Engagement: conservative 22,000
(+10 mi)

Macclenny, FL Engagement: liberal

8,500
(+24 mi) Interests: Bernie Sanders

Macclenny, FL Engagement: conservative

8,300
(+30 mi) Interests: Donald Trump for President

McCormick, SC
Engagement: liberal 3,000
(+20 mi)

McCormick, SC
Engagement: conservative 3,400
(+17 mi)

Richlands, VA
Engagement: liberal 5,000
(+34 mi), VA

Richlands, VA
Engagement: conservative 5,000
(+10 mi), VA

Saginaw, MI
Engagement: liberal 13,000
(+10 mi)

Saginaw, MI
Engagement: conservative 13,000
(+10 mi)

Engagement: liberal
Slinger, WI
Interests: Bernie Sanders, U.S. Senator Bernie 2,900
(+21 mi)
Sanders

Slinger, WI Engagement: conservative

3,100
(+24 mi) Interests: Donald Trump for President

Engagement: liberal
Interests: Democratic Party (United States),
Michigan 34,000
Bernie Sanders, U.S. Senator Bernie Sanders,
Joe Biden, Barack Obama

Engagement: conservative
Wisconsin Interests: Donald Trump for President, Re-
38,000
and Michigan publican Party (United States), Make America
Great Again or Mike Pence

Oxford, NC
Engagement: liberal 3,000
(+12 mi)

Oxford, NC
Engagement: conservative 3,600
(+10 mi)

Engagement: liberal
Scranton, PA Interests: Bernie Sanders, Democratic Party
3,000
(+45 mi) (United States), U.S. Senator Bernie Sanders,
Barack Obama, Joe Biden

Engagement: conservative
Scranton, PA
Interests: Donald Trump for President, Make 3,200
(+50 mi)
America Great Again

Table 3: Overview of the audiences created using Facebook’s

inferred interests. The targeting parameters are chosen so
that the number of conservative and liberal users in au-
diences corresponding to a particular location are roughly
equal. Interests are combined together to narrow audiences
down (i.e. with a logical AND), except for the Wisconsin and
Michigan audience, where one OR clause is used.