Online Bots - Dominating More Than Half Of The Internet Traffic

More than half of global web traffic comes from automated programs - many of them malicious

Most website visitors aren’t humans, but are instead bots - or, programs built to do automated tasks. Bots are software programs that run automated tasks over the internet. Bots typically perform simple and repetitive tasks much faster than humans can. There are personal assistant bots, chat bots, command line bots, web scraping bots and even app stores for bots.

Today, we are in a bot age where even all large companies are in the bot game. Facebook launched a bunch of new bots, Google has a home device with a bot built in, and Microsoft is in the bot game, too.

Bot frauds has been in the news lately from Linkedin filing law suits against web scrapers to Google hiring 100 employees to hunt down and remove fraudulent ads to fight against bot fraud. In 2016 Google took down 1.7m suspect adverts.

What Do You Need to Know About Online Bots

Some bots help to improve google rank in search results; other bots impersonate humans and carry out fraudulent activities like web scraping, price scraping, cart abandonment and booking fraud to name a few. Our research from InfiSecure’s bot protection platform shows that as good bots are growing in popularity and sophistication to assist in productivity tools, bad bots are growing in complexity to evade current tools and imitate humans. These advanced bots mimic humans, they know your personal preferences, and act on your behalf, prompting conversation, answering questions, and making reservations or purchases. But are these bots safe to use, and can they be trusted?

Good bots come from reputable sources, perform useful tasks, and help their human counterparts be more productive. In fact, many good bots have been around for years like Googlebot (Google’s search engine bot) that crawls the web to index and optimize a website’s SEO.

However, the bigger threats to your business may be cart abandonment, price scraping, content theft, and product or service aggregation, cutting into your revenue and margins. We have recently seen bad bots purchasing large amounts of entertainment tickets for resale at higher prices. Bots are also actively working to get your pricing information for competitors or discounters, stealing your content for reuse, damaging your search engine optimization (SEO), influencing your advertising, and skewing your analytics.

Your customers and employees are also being targeted, as bots try to steal personal information and credentials for aggregation and resale. Additionally, cybercriminals could remotely control your computer and perform illegal activities such as stealing your intellectual property, spreading spam, and distributing malware via bots.

How To Block Bad Bots

Advanced bad bots mimic human behavior and try to bypass most in-house bot prevention solutions. However, the improvements in machine learning, and intelligent bot detection techniques relying on global bot behavior has made bot detection extremely accurate. Advanced bot blocking techniques ensures that legitimate users aren’t classified as bots. For example, a user might be on a TOR network for privacy reasons to browse a website. Some mobile browsers like Opera route their traffic through proxy servers for serving mobile-optimized pages to mobile clients. Proxy servers could have a different country of origin from the end user, or could be hosted in data centers. In that case, "looking at the country your traffic is coming from, and studying traffic from cloud data centers" need to be considered carefully so as not to block legitimate web traffic and users.

Here are some tips for blocking bad bots:

1. Look at the country your traffic is coming from and verify that it is a legitimate place your company does business. This is a basic check to block bot traffic.
   This way, you will be able to block bad bots but advanced bots will use dynamic IPs and spoof browser and locations to bypass this check.


2. Study traffic that is originating from cloud data centers, which are often used by bots, both good and bad. Bad bots that originate from data centers can be scraping bots, spam bots or even a bot attack to exhaust the server capacity.
   This is an effective way to block bots if the economics of building and maintaining a big list of all data center IPs makes business sense.


3. Consider "prove that you are not a robot" tests for critical information such as account logins and pricing or purchasing details.
   This helps to block simple bots but advanced bots will bypass the CAPTCHA.


4. Evaluate the benefits of two-stage login and form submission, using email, mobile phone, or other two-factor authentication techniques.


5. Frequently mine your traffic logs for domain names that are from bot traffic and block them in your firewall. This is only work against blocking already configured bots in the firewall.


6. Have a robust bot protection platform in place that can detect and block bad bots in real-time. You will not have to worry about bot traffic on your website by implementing a bot protection platform because you get the most accurate bot detection and other advance features that help block bad bots. A SaaS bot protection platform will cover all aspects of website security, including Bot Protection, Click Fraud Protection and Fake Traffic Prevention.

Good and bad bots are probably here to stay, and they will continue to grow in capability as we learn more about natural language processing and expand machine-learning capacity. Security systems and defensive techniques will develop along with bot capabilities.