Why Traditional Techniques Fail To Block Bots

We are living in the era of digital revolution with the internet at the epicentre, changing the way we read, view, communicate, shop, travel and experience the web. With the rise in internet businesses around the world, the web has witnessed a whopping rise in bot traffic that mimic human behaviour, typically 40-60% of the total web traffic.

To understand the bot detection problem better, let's address the common website security measure taken by companies to prevent bot attacks and why they aren't effective against blocking bots.

Code Level Security

It's is a good practice to implement code level security in the initial development rather than worrying about it later. Code level security is effective when it comes to basic website security threats. But as online bots evolve to exploit new website vulnerabilities, it becomes impossible for websites to detect and block sophicated bot attacks.

Why code level security isn't effective against online bot detection?

Advanced bot attacks that can almost perfectly mimic a human user make it difficult for code level security measures and in-house bot detection tools to detect sophisticated bot patterns. Such bots that mimic human behavior are programmed to interact directly with web pages, for example, to spam forms or throw password dictionaries at user login fields. Code level security also lacks the power of collective fraud intelligence against evolving bot patterns.

Traditional IP blocking

IP filtering is effective and appropriate for some situations. For instance, a company might want to block a specific subset of web traffic to their sites, maybe due to geographic location or by only allowing a certain whitelist of IP addresses. With advancing fraud patterns, a blanket solution of IP blocking doesn't suit all business use-cases.

Why IP blocking isn't effective against online bot detection?
First, it is tedious and difficult to track and block individual IPs manually when half of the web traffic is from online bots, making a lot of IP addresses to keep track of. Second, many IPs have multiple users behind them, blocking an IP would block not just the scraper, but also all the genuine users using the IP. Additionally anyone can rent space on cloud or a TOR network and use the IP addresses that come with that space, which allows hackers to quickly obtain new IP addresses, and use them for short periods of time. It gets even more complicated if you decide to block IP addresses based on geolocation, especially if you do business on an international scale.

Web Application Firewall (WAF)

A web application firewall (WAF) is an appliance, server plugin, or filter that applies a set of rules to an HTTP conversation. Generally, these rules cover common attacks such as cross-site scripting (XSS) and SQL injection. By customizing the rules to your application, many attacks can be identified and blocked. It's possible to build some degree of website protection using in-house resources around web application firewall.

Why Web Application Firewall isn't effective against online bot detection?
Web application firewalls look at packets, and can only block specific IP addresses and are largely ineffective against detecting bots. WAF will likely block legitimate customers as well. Also WAF isn't an updated SaaS offering making it lack collective fraud intelligence that a robust bot detection platform provides.

To deploy an effective website protection and bot protection platform, most organizations will need expert assistance and a machine learning algorithm that evolves with the evolving bots.