Explaining how the real world works!     RSS Feed The Automated Tester on Twitter The Automated Tester on LinkedIn The AutomatedTester on github

Automating Web Performance data collection with BrowserMob Proxy and Selenium

Thu 26 Jan 2012

Back in 2008, David Henderson and I wanted to try automate collecting client side performance data for the web application that we were working on at the time. We were getting a large number of complaints from users about load time and we need to try solve this. The way that we decided on doing this was to use Selenium, which was already running our tests, and hack the Selenium server to give us the information we needed. What we came up with was showcased at GTAC 2009. It was really good for the time and we were happy.

About a year later just as the new movement for HTTP Archive or HAR as its commonly known was taking off I found that Jan "Honza" Odvarko had created a Firebug add on to export the network tab to HAR. This, when used with Selenium, could mean that we could get the same data that the browser wanted with little to no effort. Run your tests as you were with the new WebDriver API and get it to programmatically install the necessary addons as well as set all the preferences needed. I wrote this for Python and for .NET. This works really well but now limits this type of data to only being collected within Firefox. Some modern web applications can send back totally different javascript and sprites dependant on which browser hits.

What if you could run your Selenium tests and collect the same info as the if you were using the Firebug Net tab but using any browser. Browsermob Proxy is a good way to collect this information and it has a programmatic interface that allows us to set it before our tests start. It can also return a HTTP Archive of the traffic that it is routing. I have released the Python Library for BrowserMob Proxy that can be injected into tests quite easily. I have put an example below.

from browsermobproxy import Server
server = Server("path/to/browsermob-proxy")
server.start()
proxy = server.create_proxy

from selenium import webdriver
profile  = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)

proxy.new_har("google")
driver.get("http://www.google.co.uk")
proxy.har # returns a HAR JSON blob

proxy.stop()
driver.quit()

If we now feed the data from proxy.har into HAR Viewer we can get a nice waterfall of the traffic as below! Jari Bakken has a useful commandline version called Har.

waterfall from HAR of this site

I recently showed this off at London Web Performance. Unfortunately it wasn't recorded but you can view the slides here

    Area: blog

blog comments powered by Disqus