How to capture network traffic using selenium webdriver and browsermob proxy on Python?

Question

I would like to capture network traffic by using Selenium Webdriver on Python. Therefore, I must use a proxy (like BrowserMobProxy)

When I use webdriver.Chrome:

from browsermobproxy import Server

server = Server("~/browsermob-proxy")

server.start()
proxy = server.create_proxy()

from selenium import webdriver
co = webdriver.ChromeOptions()
co.add_argument('--proxy-server={host}:{port}'.format(host='localhost', port=proxy.port))

driver = webdriver.Chrome(executable_path = "~/chromedriver", chrome_options=co)

proxy.new_har
driver.get(url)
proxy.har # returns a HAR 

for ent in proxy.har['log']['entries']:
    print ent['request']['url']

the webpage is loaded properly and all requests are available and accessible in the HAR file. But when I use webdriver.Firefox:

# The same as above
# ...
from selenium import webdriver
profile  = webdriver.FirefoxProfile()
driver = webdriver.Firefox(firefox_profile=profile, proxy = proxy.selenium_proxy())

proxy.new_har
driver.get(url)
proxy.har # returns a HAR

for ent in proxy.har['log']['entries']:
    print ent['request']['url']

The webpage cannot be loaded properly and the number of requests in the HAR file is smaller than the number of requests that should be.

Do you have any idea what the problem of proxy settings in the second code? How should I fix it to use webdriver.Firefox properly for my purpose?

Answer 1

Just stumbled across this project https://github.com/derekargueta/selenium-profiler. Spits out all network data for a URL. Shouldn't be hard to hack and integrate into whatever tests you're running.

Original source: https://www.openhub.net/p/selenium-profiler