no such element: Unable to locate element using chromedriver and Selenium in production environment

Question

I have a problem with selenium chromedriver which I cannot figure out what's causing it. Some weeks ago everything was working OK, and suddenly this error started to show up. The problem is coming from the following function.

 def login_(browser):
    try:
        browser.get("some_url")
        # user credentials
        user = browser.find_element_by_xpath('//*[@id="username"]')
        user.send_keys(config('user'))
        password = browser.find_element_by_xpath('//*[@id="password"]')
        password.send_keys(config('pass'))
        login = browser.find_element_by_xpath('/html/body/div[1]/div/button')
        login.send_keys("\n")
        time.sleep(1)
        sidebar = browser.find_element_by_xpath('//*[@id="sidebar"]/ul/li[1]/a')
        sidebar.send_keys("\n")
        app_submit = browser.find_element_by_xpath('//*[@id="sidebar"]/ul/li[1]/ul/li[1]/a')
        app_submit.send_keys("\n")
    except TimeoutException or NoSuchElementException:
        raise LoginException

This function works with no problem in the development environment (macOS 10.11), but throws the following error in the production environment:

Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="sidebar"]/ul/li[1]/a"}
(Session info: headless chrome=67.0.3396.79)
(Driver info: chromedriver=2.40.565383 (76257d1ab79276b2d53ee97XXX),platform=Linux 4.4.0-116-generic x86_64)

I already updated both Chrome and chromedriver (v67 & 2.40, respectively) in each environment. I also gave it more time.sleep(15). But the problem persists. My latest guess is that maybe the initialization of the webdriver is not working properly:

def initiate_webdriver():
   option = webdriver.ChromeOptions()
   option.binary_location = config('GOOGLE_CHROME_BIN')
   option.add_argument('--disable-gpu')
   option.add_argument('window-size=1600,900')
   option.add_argument('--no-sandbox')
   if not config('DEBUG', cast=bool):
       display = Display(visible=0, size=(1600, 900))
       display.start()
       option.add_argument("--headless")
   else:
       option.add_argument("--incognito")
   return webdriver.Chrome(executable_path=config('CHROMEDRIVER_PATH'), chrome_options=option)

Because, if the Display is not working, then there may not be the mentioned sidebar but some other button.

So my questions are: does anybody have had a similar issue? Is there a way to know what is the page showing at the time the driver is looking for such an element?

Answer 1

It's report that the element not found error after you supplying the login , so I think the login failed and the page redirected to somewhere. You can use screenshot option to take a screenshot of the page and then see which page the driver load.

driver.save_screenshot("path to save screen.jpeg")

Also you can save the raw html code and inspect the same page.

Webdriver Screenshot

Using Selenium in Python to save a webpage on Firefox