Headless browser detection

selenium webdriver phantomjs puppeteer headless-browser
BT101 picture BT101 · Mar 26, 2019 · Viewed 8.1k times · Source

Do you know any webapps/online tests/online firewalls that are trying to detect if user is using selenium/puppeteer/phantomJS or any other headless browser?

I've created my puppeteer online crawler. I've changed many different stuff like window.navigator object (user-agent, ~.webdriver etc.).

Now I want to make sure that it is undetectable.

Answer

Thomas Dondorf picture Thomas Dondorf · Mar 26, 2019

There is a headless browser detection test which tests for the following:

  • Does the User-Agent contain the string "HeadlessChrome"?
  • Is navigator.webdriver set?
  • Is window.chrome unset?
  • Does the browser skip asking for permissions (like notifications)?
  • Are browser plugins unavailable?
  • Is navigator.languages unset?

If your browser answers any of these questions with yes, then you fail the test. For more information on the test, check out this post, which is a reply to a post called "Detecting Chrome headless, new techniques".

The author of the latter post also published another test test (code), which claims to be able to detect bots and crawlers. It performs various tests on browser attributes and generates a fingerprint of your browser.

Other "soft" tests done by websites, might include the mouse movement, scrolling behavior, IP address, etc. I doubt you will find many tests regarding these information as this is basically a cat-and-mouse game.

Related questions

Open web in new tab Selenium + Python

So I am trying to open websites on new tabs inside my WebDriver. I want to do this, because opening a new WebDriver for each website takes about 3.5secs using PhantomJS, I want more speed... I'm using a multiprocess python …

Read More
Is it possible to use Selenium WebDriver to drive PhantomJS?

I’m going through the documentation for the Selenium WebDriver, and it can drive Chrome for example. I was thinking, wouldn't it be far more efficient to ‘drive’ PhantomJS? Is there a way to use Selenium with PhantomJS? My intended …

Read More
selenium webdriver takes too long to load a page

I use PhantomJS as my webdriver. Sometimes it takes too long to load a webpage but I don't know why import time from selenium import webdriver from selenium.webdriver.common.desired_capabilities import DesiredCapabilities dcap = dict(DesiredCapabilities.PHANTOMJS) dcap["phantomjs.…

Read More