2016 Bad Bot Report: Quantifying the Risk and Economic Impact of Bad Bots
Download as PPTX, PDF1 like1,331 views
The 2016 Bad Bot Report by Distil Networks reveals that bad bots constitute 18.6% of web traffic and pose significant risks to various industries, particularly in e-commerce and digital publishing, by enabling tasks like price scraping and content theft. The report highlights that the U.S. and China are the primary origins of bad bot traffic, with advanced persistent bots becoming increasingly sophisticated in evading detection. It emphasizes the necessity for businesses to understand bot activity and implement effective strategies to mitigate these threats, with insights into quantifying risk and economic impact.
![31
Two Questions Modern Information Security Professionals Must Answer
What is the risk of [x]? How does an investment in [y]
quantifiably reduce that risk?](https://image.slidesharecdn.com/2016badbotlandscapereport1-160406144818/85/2016-Bad-Bot-Report-Quantifying-the-Risk-and-Economic-Impact-of-Bad-Bots-31-320.jpg)
![32
Three Challenges Modern Information Security Professionals Must Overcome
What is the risk of [x]?
• A language challenge
• A measurement challenge
How does an investment in [y]
quantifiably reduce that risk?
• A communications challenge](https://image.slidesharecdn.com/2016badbotlandscapereport1-160406144818/85/2016-Bad-Bot-Report-Quantifying-the-Risk-and-Economic-Impact-of-Bad-Bots-32-320.jpg)
2016 Bad Bot Report: Quantifying the Risk and Economic Impact of Bad Bots
- 1. Quantifying the Risk and Economic Impact of Bad Bots Distil Networks 2016 Bad Bot Report
- 2. Our Speakers Rami Essaid CEO & Co-founder Distil Networks Derek Brink VP & Research Fellow Aberdeen Group
- 3. 2015 Bad Bot Landscape Report Methodology Study is based on anonymized data from: 74 billion bot requests Real web traffic from hundreds of customers 17 global datacenters
- 5. Bad Bot, Good Bot and Human Traffic, 2015 Good Bots Humans Bad Bots 19% of Web Traffic Causes The Following Problems
- 6. Humans take back the Web with 54.35% of all web traffic But why? 2013 vs. 2014 vs. 2015
- 7. Humans internet users grew 8% in 2105 Especially in countries such as China, India, Indonesia, etc. 2015 Saw Tremendous Growth in Human Users Source: http://www.statista.com/statistics/273018/number-of-internet-users-worldwide/ Number of internet users worldwide from 2000 to 2015 (in millions)
- 8. Meanwhile, Bot Operators Were Updating their Software Bot software used in 2015 was vastly more advanced than in previous years This was a shift in focus from quantity of bots to quality
- 10. Traffic Distribution by Size of Site, 2014 and 2015
- 11. Traffic by Type of Site, 2014 vs 2015 In 2015 the most targeted verticals were digital publishing and real estate
- 12. Traffic by Size and Type of Site, 2014 vs 2015 More specifically, small digital publishers and large real estate sites were hardest hit in 2015
- 13. Defense Tactics - Know your Industry Understand how great of a risk bots pose to your industry Learn how bots attack sites similar to yours Industry Most Common Bot Problem Ecommerce Price scraping Digital Publishing Content theft Travel Aggregation and loss of up-sell / cross-sell opportunities Finance Brute force attacks Real Estate Scraping Listing Information
- 14. Bad Bot Origins
- 15. China and US Home to the Worst Bad Bot Originators Companies from China and the US dominate the list of organizations with the most bad bot traffic The US is always on top of this list, China is new Chin a Chin aChin a Chin a Chin a Chin a US US US US US US US
- 16. Worst Bad Bot Originators 2013 to 2015 Amazon makes the Top 5 for three years in a row Verizon Business and residential ISPs Comcast, Time Warner Cable clean up their acts
- 17. Mobile Carriers with the Most Bad Bots Dutch carriers emerge as a new hotbed for mobile client based bots The four largest mobile carriers in the US are all present on this year’s list ● Verizon Wireless ● AT&T ● T-Mobile ● Sprint PCS
- 18. Countries Originating the Most Bad Bots, 2014 vs 2015 The US still tops the list of countries with the most bad bots Israel, India, and the UK make the biggest gains Germany, Canada, Russia, and the Netherlands move down the list
- 19. Countries Most Often Blocked by Geofencing Rules 2014 saw customers blocking developing countries and stereotypical “bad guys” 2015 saw customers blocking more industrialized countries
- 20. Top “Bad Bot GDP’s” of 2014 and 2015 Maldives rules the roost with 526 bad bots per human online user The average number of bots per human user on this list increased from 26.1 bots/user to 99.2 bots/user
- 21. Defense Tactics - Know Their Origins Does your business model support all regions? Is it normal that your customer is originating from a commercial data center or cloud provider? Are there any reasons visitors to your site should go through a TOR network? Analyze your business. Then trim the fat.
- 22. Bad Bot Capabilities and Behavior
- 23. The Majority of Bots are Now APBs Advanced Persistent Bots (APBs) are becoming more commonplace APBs are defined as having one or more of the following abilities: ● Mimicking human behavior ● Loading JavaScript and external assets ● Cookie support ● Browser automation ● IP spoofing and rotation ● User agent spoofing and rotation ● Distributed attacks (using many IP addresses at once)
- 24. Loading Assets & Bots Mimicking Humans % of bots able to load external Assets (e.g. JavaScript) % of bots able to mimic human behavior These bots will skew marketing tools such as (Google Analytics, A/B testing, conversion tracking, etc.) These bots will fly under the radar of most security tools
- 25. That Majority of Bad Bots Now Use Multiple IP Addresses Bots which dynamically rotate IP addresses, or distribute attacks are significantly harder to detect and mitigate
- 26. Bad Bots Obtain New User Agents to Persistently Attack Websites Over 36% of bots use multiple user agents to evade detection and overcome blacklisting and custom blocking rules
- 27. Chrome Takes the Lead as Most Assumed User Agent
- 28. Defense Tactics - Defeat APBs with Fingerprinting Real-analysis and device fingerprinting allows security solutions to track bots even if they ● Assume new identities ● Mimic human behavior ● Rotate IP Addresses ● Distribute their attack over Many IP addresses
- 29. 29 Quantifying the Risk of Bad Bots Derek E. Brink, CISSP Vice President and Research Fellow, Information Security and IT GRC [email protected] www.linkedin.com/in/derekbrink April 2016 Derek E. Brink, CISSP Vice President and Research Fellow, Information Security and IT GRC [email protected] www.linkedin.com/in/derekbrink April 2016 Quantifying the Risk of Bad Bots
- 30. 30 Context: The Dual Roles of Modern Information Security Professionals Subject Matter Experts Trusted Advisors
- 31. 31 Two Questions Modern Information Security Professionals Must Answer What is the risk of [x]? How does an investment in [y] quantifiably reduce that risk?
- 32. 32 Three Challenges Modern Information Security Professionals Must Overcome What is the risk of [x]? • A language challenge • A measurement challenge How does an investment in [y] quantifiably reduce that risk? • A communications challenge
- 33. 33 The Threat of Bad Bots: A Material Percentage of Web Site Traffic Bad Bots Good Bots Humans 18.6% 27.0% 54.4% Source: Distil Networks, 2016 Bad Bot Landscape Report
- 34. 34 Web Site Vulnerabilities and Exploits Related to Bad Bots Bad Bot Vulnerabilities and Exploits (illustrative) Web Security Brute force login; account takeover; fraudulent account creation Man-in-the-browser attacks Reconnaissance attacks; application coding exploits Application denial of service Spam Web Scraping Content theft Price scraping API scraping Competitive data mining Waste and Abuse Web site performance Negative SEO Skewed web site analytics Fraud Fraudulent transactions Digital ad fraud Source: adapted from Distil Networks, 2016 Bad Bot Landscape Report; Aberdeen Group, April 2016
- 35. 35 The Risk of Bad Bots: How Likely? What Business Impact? Bad Bot Vulnerabilities and Exploits (illustrative) Likelihood Impact Web Security Brute force login; account takeover; fraudulent account creation How Likely is it that these Vulnerabilities are Successfully Exploited? What is the Business Impact, when Successful Exploits Do Occur? Man-in-the-browser attacks Reconnaissance attacks; application coding exploits Application denial of service Spam Web Scraping Content theft Price scraping API scraping Competitive data mining Waste and Abuse Web site performance Negative SEO Skewed web site analytics Fraud Fraudulent transactions Digital ad fraud Source: adapted from Distil Networks, 2016 Bad Bot Landscape Report; Aberdeen Group, April 2016
- 36. 36 Qualitatively, Four Categories for the Business Impact of Bad Bots • Additional cost • Data breaches • Loss of current revenue • Loss of future revenue
- 37. 37 At a Qualitative Level, the Business Impact of Bad Bots Bad Bot Vulnerabilities and Exploits (illustrative) Likelihood Incr. Cost Data Loss Curr .Rev . Fut. Rev. Web Security Brute force login; account takeover; fraudulent account creation How Likely is it that these Vulnerabilities are Successfully Exploited? X X X X Man-in-the-browser attacks X X X X Reconnaissance attacks; application coding exploits X X X X Application denial of service X X X Spam X X Web Scraping Content theft X X X X Price scraping X X X X API scraping X X X X Competitive data mining X X X X Waste and Abuse Web site performance X X X Negative SEO X X X Skewed web site analytics X X X Fraud Fraudulent transactions X X X Digital ad fraud X X Source: adapted from Distil Networks, 2016 Bad Bot Landscape Report; Aberdeen Group, April 2016
- 38. 38 There are Many Approaches to Measuring and Communicating Risk that We’re All Familiar With … But These Don’t Really Work! Techno-babble about threats, vulnerabilities, and exploits Headlines of recent breach disclosures ALE-style calculations Averages, based on surveys Crackpot rigor Qualitative “heat maps” “$201 / record”
- 39. 39 With These Approaches, Most Decisions About Security-Related Risks are Still Made by the Intuition and Gut Instinct of the HiPPO … (The Highest-Paid Person in the Organization)
- 40. 40 Let’s Try to Raise the Bar for Making Important Decisions About Security-Related Risks, Beyond Mere Intuition and Gut Instinct! Source: http://dilbert.com/strip/2016-03-24
- 41. 41 Modeling the Risk of Bad Bots • Let’s estimate the risk (both likelihood, and impact) of bad bots, using these four high-level categories: • Additional cost • Data breaches • Loss of current revenue • Loss of future revenue • Remember that risk is inherently about making decisions in the face of uncertainties • Models are not about precision … • … they are about making better-informed decisions about risk … • … most of which are based primarily on intuition
- 42. 42 Monte Carlo Modeling is a Proven, Widely Used Solution for our Measurement Problem • In a nutshell: we can carry out the same familiar estimates and computations we have traditionally made • Except that we do this for many (say, ten thousand) scenarios, each of which uses a random value from our estimated ranges and distributions • The results of these computations are likewise not a single, static number – which says nothing about risk • The output is also a range and distribution, from which we can readily describe both probabilities and business impact • I.e., the results can be expressed in terms of risk – which is exactly what we are looking for!
- 43. 43 We’re All Familiar with This Approach, Too – Note the Inclusion of Both Likelihood and Impact in This Illustrative Example!
- 44. 44 Just So Long As We Don’t Do This … Remember, All Models Are Wrong – But Some Can Be Useful! Source: http://dilbert.com/strip/2016-04-01
- 45. 45 Risk of Bad Bots Additional Cost Overprovisioning of web site infrastructure Web site contribution to annual revenue Data breaches Loss of Current Revenue Loss of Future Revenue Factoring the Risk of Bad Bots – Conceptual $ $ $ $ Source: Aberdeen Group, April 2016 % of annual revenue spent on web site infrastructure % of web traffic represented by bad bots Web site contribution to annual revenue % of annual revenue spent of website marketing % of web traffic represented by bad bots # of “incidents” represented by bad bots (i.e., an attempt) Likelihood of a “breach” (i.e., a success) Business impact of a breach Web site contribution to annual revenue Web site contribution to annual revenue Time that web site is negatively affected (e.g., downtime or slowdown) % of revenue lost during the period of downtime or slowdown % of web traffic represented by bad bots % of website revenue lost as a result of fraud Wasted web site marketing Cost of data breaching Downtime and slowdown Fraudulent transactions $
- 46. 46 Factoring the Risk of Bad Bots – Computational Source: Aberdeen Group, April 2016
- 47. 47 Run the Numbers – The Results Provide Invaluable Insights into the Risk of Bad Bots Histogra m Probability Curve Source: Aberdeen Group, April 2016
- 48. 48 Quantifying the Risk of Bad Bots Source: Aberdeen Group, April 2016
- 49. 49 Quantifying the Risk of Bad Bots … and Addressing the Two Fundamental Questions • For a web site contributing $100M / year in revenue (% of web site annual revenue) • Median annual reduction in risk: about 18 times • Median annual return on investment: about 22 times • Note: the risk owner still needs to decide … Source: Aberdeen Group, April 2016
- 51. Distil Networks 2016 Bad Bot Report: Quantifying the Risk and Economic Impact of Bad Bots Distil Networks has produced their third annual Bad Bot Report. It's the IT Security Industry's most in-depth analysis on the sources, types, and sophistication levels of last year's bot attacks -- and there are serious implications for anyone responsible for securing websites and APIs. Join Derek Brink, Vice President of Research at Aberdeen Group and Rami Essaid, CEO of Distil Networks as they dive into the data to reveal: ● 6 high-risk lessons every IT security pro must know ● How to quantify the risk and economic impact of bad bots for your organization ● How bot activity varies across websites based on industry and popularity ● The worst offending countries, ISPs, mobile operators, and hosting providers Bad bots are the key culprits behind web scraping, brute force attacks, competitive data mining, online fraud, account hijacking, unauthorized vulnerability scans, spam, man-in-the-middle attacks, digital ad fraud, and downtime. Register today to gain actionable insights on how to defend your websites and APIs for the coming year of threats. Abstract
- 52. 52 Modeling the Risk of Bad Bots: Additional Cost (1) 1. Web site contribution to annual revenue ($ / year) • For the purposes of this analysis, let’s model based on $100,000,000 2. % of annual revenue spent on web site infrastructure • “Infrastructure” = all related people, process, technologies • Model as 4% - 6%; uniform distribution (analyst estimates) 3. % of web traffic represented by bad bots • Model as 0% - 50%; most likely 18.6%; beta distribution (Distil Networks) 4. Annual cost of overprovisioning web site infrastructure • (1) x (2) x (3) Source: Aberdeen Group, April 2016
- 53. 53 Modeling the Risk of Bad Bots: Additional Cost (2) 1. Web site contribution to annual revenue ($ / year) • For the purposes of this analysis, let’s model based on $100,000,000 2. % of annual revenue spent on web site marketing • “Marketing” = all costs related to driving web traffic • Model as 5% - 15%; normal distribution (analyst estimates) 3. % of web traffic represented by bad bots • Model as 0% - 50%; most likely 18.6%; beta distribution (Distil Networks) 4. Annual cost of wasted web site marketing (e.g., negative SEO, skewed web site analytics, etc.) resulting from bad bots • (1) x (2) x (3) Source: Aberdeen Group, April 2016
- 54. 54 Modeling the Risk of Bad Bots: Data Breaches 1. # of “incidents” represented by bad bots (i.e., an attempt) • One extreme: all bad bots = 1 incident • The other extreme: every bad bot = 1 incident • My modeling choice: 1 (one incident per year) to 12 (one incident per month); beta distribution 2. Likelihood of a “breach” (i.e., a success) • 0% - 100%; mostly likely 30%; beta distribution (Verizon DBIR) 3. Business impact of a data breach • Expressed as a function of the number of records (Verizon DBIR) • Use 100,000 – 1,000,000 records as the range (Privacy Rights Clearinghouse) 4. Annual cost of data breaches resulting from bad bots • (1) x (2) x (3) Source: Aberdeen Group, April 2016
- 55. 55 Modeling the Risk of Bad Bots: Loss of Current Revenue (1) • Bad bots → negative impact on web site availability and performance • Combination of downtime and slowdown results in web site customers abandoning what they were trying to do … which leads to lost revenue during this time of disruption 1. Web site contribution to annual revenue ($ / year) • For the purposes of this analysis, let’s model based on $100,000,000 2. Time that web site is negatively affected (e.g., downtime or slowdown) (hours / year) • For simplicity, assume 24x7x365 operation • Model as 0 – 720 hours; most likely 200 hours; beta distribution (Arbor Networks) 3. % of revenue lost during the period of downtime or slowdown • Model as 1% to 30%; most likely 3%; beta distribution (analyst estimates) 4. Loss of current revenue as a result of bad bots • (1) x (2) x (3) Source: Aberdeen Group, April 2016
- 56. 56 Modeling the Risk of Bad Bots: Loss of Current Revenue (2) • Bad bots → fraudulent transactions 1. Web site contribution to annual revenue ($ / year) • For the purposes of this analysis, let’s model based on $100,000,000 2. % of web site traffic represented by bad bots • 0% - 50%; most likely 18.6%; beta distribution (Distil Networks) 3. % of web site revenue lost as a result of fraud from bad bot traffic • Model as 0% – 10%; most likely 1.4%; beta distribution (Kroll, Global Fraud Survey) 4. Loss of current revenue as a result of bad bots • (1) x (2) x (3) Source: Aberdeen Group, April 2016
- 57. 57 Final Important Detail: Effectiveness of Countermeasures for Bad Bots • Status quo = manual blocking • 0% - 50%; most likely 12%; beta distribution • Assume that the annual cost of manual blocking is already baked in to the cost of overprovisioned web site infrastructure • Future state = use the Distil Networks solution • 90% - 100%; mostly likely 99.9%; beta distribution • The model for the future state must also incorporate the annual cost of the Distil Networks solution Source: adapted from Distil Networks, 2016 Bad Bot Landscape Report; Aberdeen Group, April 2016