Parsing HTML5 data-* attribute values with Selenium in Python

meepl picture meepl · Feb 4, 2015 · Viewed 15.1k times · Source

I am parsing a JS generated webpage like so:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


driver = webdriver.Firefox()
driver.get('https://www.consumerbarometer.com/en/graph-builder/?question=M1&filter=country:singapore,canada,mexico,brazil,argentina,united_states,bulgaria,austria,belgium,croatia,czech_republic,denmark,estonia,finland,france,germany,greece,hungary,italy,ireland,latvia,lithuania,norway,netherlands,poland,portugal,russia,romania,serbia,slovakia,spain,slovenia,sweden,switzerland,ukraine,united_kingdom,australia,china,israel,hong_kong_sar,japan,korea,new_zealand,malaysia,taiwan,turkey,vietnam')

// wait for svg to appear
WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.TAG_NAME, 'svg')))

for text in driver.find_elements_by_class_name('bar-text-label'):
    print(text.text)

driver.close()

Besides getting the text from the class bar-text-label I would also like to get values from an HTML5 data-attribute. For example,<rect rx="3" ry="3" width="76%" height="40" transform="translate(0,40)" data-value="76" class="bar"></rect> and I would like to be able to parse 76 from this.

Is this possible to do in Selenium?

I tried both of the below, with no sucess:

for text in driver.find_elements_by_class_name('bar'): 
    print(data_value.text)

for data in driver.find_elements_by_xpath('//*[contains(@data-value)]/@data-value'): 
    print(data.text)

Answer

Jessamyn Smith picture Jessamyn Smith · Feb 4, 2015

If you have elements like the following:

<rect rx="3" ry="3" width="76%" height="40" transform="translate(0,40)" data-value="75" class="bar">bar1</rect>
<rect rx="3" ry="3" width="76%" height="40" transform="translate(0,40)" data-value="76" class="bar">bar2</rect>

You can get the text value and the attribute value as follows:

elements = driver.find_elements_by_class_name('bar')
for element in elements:
    print element.text
    print element.get_attribute('data-value')

This prints out:

bar1
75
bar2
76