Skip to main content

Towards a framework for detecting advanced Web bots.

Iliou, C., Kostoulas, T., Tsikrika, T., Katos, V., Vrochidis, S. and Kompatsiaris, Y., 2019. Towards a framework for detecting advanced Web bots. In: ARES 2019: 14th International Conference on Availability, Reliability and Security, 26--29 August 2019, Canterbury, UK.

Full text available as:

paper_82.pdf - Accepted Version
Available under License Creative Commons Attribution Non-commercial.


Official URL:


Automated programs (bots) are responsible for a large percentage of website traffic. These bots can either be used for benign purposes, such as Web indexing, Website monitoring (validation of hyperlinks and HTML code), feed fetching Web content and data extraction for commercial use or for malicious ones, including, but not limited to, content scraping, vulnerability scanning, account takeover, distributed denial of service attacks, marketing fraud, carding and spam. To ensure their security, Web servers try to identify bot sessions and apply special rules to them, such as throttling their requests or delivering different content. The methods currently used for the identification of bots are based either purely on rule-based bot detection techniques or a combination of rulebased and machine learning techniques. While current research has developed highly adequate methods for Web bot detection, these methods’ adequacy when faced with Web bots that try to remain undetected hasn’t been studied. For this reason, we created and evaluated a Web bot detection framework on its ability to detect conspicuous bots separately from its ability to detect advanced Web bots. We assessed the proposed framework performance using real HTTP traffic from a public Web server. Our experimental results show that the proposed framework has significant ability to detect Web bots that do not try to hide their bot identity using HTTP Web logs (balanced accuracy in a false-positive intolerant server > 95%). However, detecting advanced Web bots that present a browser fingerprint and may present a humanlike behaviour as well is considerably more difficult.

Item Type:Conference or Workshop Item (Paper)
Uncontrolled Keywords:Web bot detection; Evasive Web bots; Advanced Web bots; humanlike behaviour
Group:Faculty of Science & Technology
ID Code:32523
Deposited By: Symplectic RT2
Deposited On:12 Jul 2019 08:59
Last Modified:14 Mar 2022 14:16


Downloads per month over past year

More statistics for this item...
Repository Staff Only -