How can I scrape a website without getting detected and bypassing reCAPTCHA using selenium webdriver through Python?

python selenium-webdriver web-scraping recaptcha webdriver-w3c-spec
Imran Rafiq picture Imran Rafiq · Mar 13, 2019 · Viewed 7k times · Source

I know the webscraping and I have taken the data from different website and I am using python language and selenium webdriver chrome. But I call a website it is open front page and then I click or go any other page then website restrict me and website know that I am using automated chrome.

Answer

Carlos picture Carlos · Mar 13, 2019

This may be because the website uses reCAPTCHA v3, which "allows you to verify if an interaction is legitimate without any user interaction". This means that they can identify if you are not a human without asking you to check the famous "I'm not a robot" box. That box is used in the former version of reCAPTCHA, v2.

Read more about reCAPTCHA here: https://developers.google.com/recaptcha/docs/versions

I don't think it's possible to work around this with Selenium. And, as was already mentioned, web scraping is often illegal.

Related questions

selenium with scrapy for dynamic page

I'm trying to scrape product information from a webpage, using scrapy. My to-be-scraped webpage looks like this: starts with a product_list page with 10 products a click on "next" button loads the next 10 products (url doesn't change between the two …

Read More
How can I download a file on a click event using selenium?

I am working on python and selenium. I want to download file from clicking event using selenium. I wrote following code. from selenium import webdriver from selenium.common.exceptions import NoSuchElementException from selenium.webdriver.common.keys import Keys browser = webdriver.…

Read More
Clicking on a link via selenium

I am trying to do some webscraping via Selenium. My question is very simple: How do you find a link and then how do you click on it? For instance: The following is the HTML that I am trying to …

Read More