Distil Networks’ Third Annual Bad Bot Landscape Report Finds Advanced Persistent Bot Activity on the Rise, Despite Overall Decrease in Bad Bot Activity
Google Chrome Most Popular Bad Bot Disguise; Rapid Bad Bot Growth from Chinese ISPs; Bad Bots Skewing Google Analytics; Real Estate, Digital Publishing and Medium-Sized Sites Face Greatest Threats
San Francisco, CA – Distil Networks, Inc., the global leader in bot detection and mitigation, published, “The 2016 Bad Bot Landscape Report: The Rise of Advanced Persistent Bots,” an annual report that identifies statistically significant data on global bot traffic.
Bad bots are used by competitors, hackers and fraudsters and are the key culprits behind web scraping, brute force attacks, competitive data mining, online fraud, account hijacking, data theft, unauthorized vulnerability scans, spam, man-in-the-middle attacks, digital ad fraud, and downtime.
“When we dug into the bot activity in 2015, we identified an influx of Advanced Persistent Bots (APBs),” said Rami Essaid, co-founder and CEO of Distil Networks. “ABPs can mimic human behavior, load JavaScript and external assets, tamper with cookies, perform browser automation, and spoof IP addresses and user agents. The persistency aspect is that they evade detection with tactics like dynamic IP rotation from huge pools of IP addresses, use Tor networks and peer to peer proxies to obfuscate their origins, and distribute attacks over hundreds of thousands of IP addresses. A whopping 88 percent of 2015 bad bot traffic were APBs. This shows that bot architects have already taken note of traditional bot detection techniques and are finding new sophisticated ways to invade websites and APIs, in an effort to take advantage of critical assets and impact a business’s bottom line.”
Key Findings
Bot traffic
- 46 percent of all web traffic originates from bots, with over 18 percent from bad bots
- For the first time since 2013, humans outnumbered bots for website traffic
- Medium-sized websites (10,001 to 50,000 Alexa ranking) are at a greater risk, as bad bot traffic made up 26 percent of all web traffic for this group
- Chrome edged out Firefox as the browser of choice for bad bot creators with over 26 percent of all user agents utilizing the Google browser
The rise of Advanced Persistent Bots (APBs)
- 88 percent of all bad bot traffic has one or more characteristics of an Advanced Persistent Bot
- 53 percent of bad bots are now able to load external resources like JavaScript meaning these bots will end up falsely attributed as humans in Google analytics and other tools
- 39 percent of bad bots are able to mimic human behavior so tools such as WAFs, web log analysis, or Firewalls, which perform less detailed analysis of clients and their behavior, will likely result in huge amounts of false negatives
- 36 percent of bad bots disguise themselves using two or more user agents, and the worst APBs change their identities over 100 times
- 73 percent of bad bots rotate or distribute their attacks over multiple IP addresses and of those, a whopping 20 percent surpassed 100 IP addresses
Amazon earns a hat-trick, China has the worst offending ISPs, while residential ISPs Comcast and Time Warner clean up their act
- Amazon has appeared in the Top 5 Bad Bot Originators three years in a row
- Despite their repeated appearance in the top Bad Bot Originators list in 2013 and 2014, residential ISPs Comcast and Time Warner fell off the Top 20 bad bot originators for 2015
- Six out of the top 20 ISPs with the highest percentage of bad bot traffic originated from China
- US and Netherlands had the most mobile carriers, 5 and 3 respectively, on the top 20 list of bad bot mobile carriers
Digital publishing and real estate industry websites are bot targets
- Real estate websites saw a 300 percent increase in bad bot activity, with large real estate sites experiencing the most pain
- As an industry, digital publishers were hit hardest by bad bots, which make up over 31 percent of all their traffic
- For small digital publishers (Alexa 50,001 – 150,000) 56 percent of traffic originates from bad bots
Huge increase in bad bot traffic from China, but the United States still has biggest bot problem
- Maldives, Israel and Kyrgyzstan had the highest bad bot GDP (number of bad bots per online user) at 526, 168, and 94 respectively
- China, Norway, Germany, and the Netherlands are the most blocked countries for web traffic
- The United States boasts the largest originator of bots again, with over 39 percent of bot traffic, while India and Israel moved up to number two and three, respectively
The 2016 Bad Bot Landscape Report is based on aggregate data gathered from Distil Networks’ bot detection and mitigation solution that identifies and tracks bots in real time, the world’s largest Known Violators Database of bad bot fingerprints, as well as Distil’s global network of 17 data centers.
To download a copy of the report, visit; http://resources.
To learn more, register for “Distil Networks 2016 Bad Bot Report: Quantifying the Risk and Economic Impact of Bad Bots,” a webinar with SecurityWeek and Derek Brink, vice president of research at Aberdeen Group on Tuesday, April 5, 2015 at 1:00 pm Eastern, at https://www.brighttalk.com/
About Distil Networks
Distil Networks, the global leader in bot detection and mitigation, is the first easy and accurate way to identify and police malicious website traffic, blocking 99.9% of bad bots without impacting legitimate users. Distil protects against web scraping, brute force attacks, competitive data mining, online fraud, account hijacking, unauthorized vulnerability scans, spam, man-in-the-middle attacks, digital ad fraud, and downtime. Slash the high tax that bots place on your internal teams and web infrastructure and make your online applications more secure with API security, real-time threat intelligence, a 24/7 security operations center, and complete visibility and control over human, good bot, and bad bot traffic. For more information on Distil Networks, visit us at www.distilnetworks.com or follow @DISTIL on Twitter.
