How to move to the next page on Python Selenium?

Question

I am trying to build a proxy scraper for a specific site, but I'm failing on move to next page.

This is the code that I'm using.

If you answer my question, please, explain me a bit about what you used and if you can, please, if there are any good tutorials for over this kind of code, provide me some:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time

options = Options()
#options.headless = True     #for headless
#options.add_argument('--disable-gpu') #for headless and os win

driver = webdriver.Chrome(options=options)

driver.get("https://hidemyna.me/en/proxy-list/")
time.sleep(10) #bypass cloudflare


tbody = driver.find_element_by_tag_name("tbody")
cell = tbody.find_elements_by_tag_name("tr")

for column in cell:
    column = column.text.split(" ")
    print (column[0]+":"+ column[1]) #ip and port

nxt = driver.find_element_by_class_name('arrow_right')
nxt.click()

Answer 1

To move to the next page you can try the following solution:

• Code Block:

  from selenium import webdriver
  from selenium.webdriver.chrome.options import Options
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.common.exceptions import TimeoutException, WebDriverException
  
  options = Options()
  options.add_argument("start-maximized")
  options.add_argument("disable-infobars")
  options.add_argument("--disable-extensions")
  driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
  driver.get('https://hidemyna.me/en/proxy-list/')
  while True:
      try:
          driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//li[@class='arrow__right']/a"))))
          driver.find_element_by_xpath("//li[@class='arrow__right']/a").click()
          print("Navigating to Next Page")
      except (TimeoutException, WebDriverException) as e:
          print("Last page reached")
          break
  driver.quit()
  

• Console Output:

  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  .
  .
  .
  Navigating to Next Page
  Last page reached