Study finds web scrapers can be employed for as little as $3.33 per hour
SAN FRANCISCO – Distil Networks, Inc., the global leader in bot detection and mitigation, released a new study titled, “The 2016 Economics of Web Scraping” which illustrates web scraping’s prevalence, sophistication, and industry use cases. Through analysis of top web scraping platforms and services, the report also outlines how the democratization of web scraping allows users to effortlessly steal sensitive information on the web.
Web scraping is a computer software technique for extracting information from websites, and often includes transforming unstructured website data into a database for analysis or repurposing content into the web scraper’s own website and business operations. In most cases, bots, which make up 46 percent of web traffic, are implemented by individuals to perform web scraping at a much faster rate than humans alone. Diverse actors leverage web scraping bots, including nefarious competitors, internet upstarts, hedge funds, fraudsters, hackers, and spammers, to effortlessly steal whatever pieces of content they are programmed to find, and often mimic regular user behavior, making them hard to detect and even harder to block. Roughly two percent of online revenue is lost as a result of web scraping.
“If your content can be viewed on the web, it can be scraped,” said Rami Essaid, CEO and co-founder of Distil Networks. “Not only does web scraping pose a critical challenge to a website’s brand, it can threaten sales and conversions, lower SEO rankings, or undermine the integrity of content that took considerable time and resources to produce. Understanding the pervasive nature of today’s web scraping economy not only raises awareness about this growing challenge, it also allows website owners to take action in the protection of their proprietary information.”
Key Findings
- 38 percent of companies who engage in web scraping do so to obtain content
Web scraping is also used for research, contact scraping, price comparison, weather data monitoring, and website change detection.
- Real estate sites are the number one victim of web scraping
Other industries plagued by web scraping include digital publishing, travel, online directories, e-commerce, and marketplace and classifieds.
- Web scraping services cost as little as $3.33 per hour
The average web scraping project costs roughly $135.
- The average web scraper makes $58,000 annually
When working for a large company specializing in web scraping, individuals can earn up to $128,000 per year for their work.
To download a copy of the The 2016 Economics of Web Scraping study, visit http://resources.
About Distil Networks
Distil Networks, the global leader in bot detection and mitigation, is the only easy and accurate way to protect web applications from bad bots, API abuse, and fraud. With Distil, you automatically block 99.9% of malicious traffic without impacting legitimate users. Distil Web Security defends websites against web scraping, competitive data mining, account takeovers, transaction fraud, unauthorized vulnerability scans, spam, man-in-the-middle attacks, digital ad fraud, and downtime. Distil API Security protects public and partner-facing APIs against developer errors, integration bugs, automated scraping, and web and mobile hijacking. For more information on Distil Networks, visit us at http://www.distilnetworks.com or follow @DISTIL on Twitter.
