Explaining how the real world works!     RSS Feed The Automated Tester on Twitter The Automated Tester on LinkedIn The AutomatedTester on github

The right tool for the job!

Thu 08 Nov 2012

One of the biggest things that always shocks me is the way people approach problems. Software developers are essentially pattern matchers using experience to figure out how to solve problems that are placed in front of them.

This can unfortunately lead them to a problem where developers use their current knowledge to solve problems. Imagine that you are in QA and your only experience of automation is using Selenium. You are told that you need to test a REST API, or as I have seen in forums to test stored procedures in your database, you go for the tool that you know and feel comfortable with. The downside to this is that you could possibly pick the wrong tool!

If you are thinking that people don't use Selenium to test Databases/SOAP/REST think again and go have a read of the Selenium User Forum!

Selenium is a wonderful tool! I am not just saying that because I have written two books or because part of my time is to try support the project, I say it because I love watching the tests run! Unfortunately it uses a browser to drive a web application. This means that you have the entire stack to worry about when testing. Actually you have more than your stack to worry about when testing. A Browser is a very complex bit of software. Both Google and Mozilla have shown that it can be its own operating system with Chrome OS and Firefox OS respectively.

That is a lot of lines of code that are run just to make sure that your tests do what they are expected. When you need a browser you use it and then make sure everything is working. The browser start up time annoys you but for the coverage you are getting it is acceptable.

But when you are testing something that isn't rendering there is a lot of wasted parts of the browser stack that are called and aren't really used.

So when running your tests thing about how much is being invoked and how much really needs to be invoked. Try not to be wasteful with the resources that are consumed. When little resources are consumed in a test then the tests will run run quicker making feed back loops take less time.

    Area: blog

Where we are with W3C Browser Automation Specification

Mon 05 Nov 2012

Last week there was a very large gathering for the W3C in Lyon, France. The TPAC conference gets together all working groups and allows them to share their specifications with everyone else and demo implementations if there are some complete.

The Browser Automation Working group was there too to discuss moving the current specification further. We had 2 days of meetings and below is a break down of the topics we discussed.

Day One - Minutes

On the first day we discussed the following

  • Interoperability (the spec is very loose on this, but interoperability is vital)
  • Internationalised input (do we need the IME methods from OSS WebDriver? Our current solution is very desktop-centric)
  • Possibly adding an ARIA locator
  • Handling the shadow DOM
  • Do we use strings or numbers for status codes (there are arguments either way)
  • Identifying areas of the WebDriver spec that better belong elsewhere

Day Two - Minutes

Discussion of open issues in the spec:

  • Screenshots
  • Non-HTML content (SVG, XML, XHTML, canvas, text/plain)
  • Publishing a new working draft
  • Start work on the conformance tests

The main items that came from the work this week are:

  • Browser Vendors are to provide a shim that understands the HTTP JSON Wire Protocol. This allows browser vendors to use the best transport mechanism they want but gives implementors of the local end library to use the same approach for all browsers ensuring interoperability between browsers.
  • We are going to be changing error codes to error strings. Using strings allows us to pass back a meaningful error without it needing to be translated from an integer to allow easier development of libraries for local ends.
  • We are going to be adding the ability to get a screenshot of a specific element on the page.
  • We have started work on the conformance tests for the specification. The first tests actually found a bug in the Selenium WebDriver implementation which is great.

We will be cleaning up a few sections and hopefully be pushing the next version of the specification in the next few weeks. If you have thoughts or you spot any errors please let us know!

    Area: blog

What is it like to be a Mozillian; My first two years.

Thu 25 Oct 2012

It has been two years to the day that I joined Mozilla. I have mentioned in the past that I came in having missed the contributor to employee route and, as I said then, feel like I have missed out.

So what have the last two years been like? If I was only allowed one word it would be "Rollercoaster". Now I know that you are probably thinking that I chose it for the cliche of ups and downs. I chose the word because it is full on, and at a tremendous speed, and with out a doubt, fun!

So what have I done in the last 2 years at Mozilla? I have worked in 3 different teams. I started out in the WebQA team helping standardise the way they write their Selenium tests. They have gone on to do much much more than I expected and I love watching what this team does. It is after all my first home in Mozilla.

The next team that I was in was Automation Services. This was a team that I co-lead with Henrik Skupin and we did some really good things. This team was then merged with the Automation and Tools team, the A-Team as they are known internally, which is my current team.

I have worked on everything from web applications to Firefox Mobile to Firefox itself. I am co-editing a W3C standard with Simon Stewart to make sure that web developers and software testers can increase the quality of the products that they deliver to the web. I have worked on Frameworks and production code and I have written a lot of tests.

All the while knowing that everything I do is making a difference to the way the web is being used and worked on. It is one of the main things that drives to me to want to work at Mozilla. One of the other main things that I have found interesting, and to be honest still shocks me, is the warmth that people exhibit when they find that you work for Mozilla.

So after two years at Mozilla I can honestly say that I am working with a large number of brilliant people, though I have always been lucky with my previous employers, who continue to teach me, guide me and to give me opportunities to show how brilliant our craft and industry is!

P.s. Want to help Mozilla? Have a look at What can I do for Mozilla?

    Area: blog

Selenium 2 Testing Tools is out!

Tue 09 Oct 2012

So it is official, the 2nd version of my book is officially out! It has been a lot of work but I am happy with the current result. Selenium 2 Testing tools is a beginners guide that walks the user from creating easy throw away tests with Selenium IDE to being able to use Selenium WebDriver on desktop Operating Systems to being able to run them on mobile devices.

It is a beginners guide so if you know how to code you should be able to get started and by the end of the book be pretty confident at using Selenium WebDriver on a number of different platforms and devices.

You can buy the book from Packt Publishing or from Amazon.com or from Amazon.co.uk.

I hope that you enjoy it!

    Area: blog

Marionette - The Future of FirefoxDriver in Selenium

Wed 25 Jul 2012

We all know that Selenium WebDriver is on track to becoming a W3C standard. First we saw this happen with Opera who have integrated this with their browser when they created OperaDriver. Google Chrome followed with the release of their ChromeDriver executable. This is a separate download to the browser and client bindings. So who is next? Mozilla.

Why do we need a new FirefoxDriver for Selenium?

Mozilla has been working on a project called Marionette which will become a future version of FirefoxDriver. Marionette was created as a way to automate Firefox OS, since Firefox OS is just a kernel then starts up Gecko, the core engine of Firefox. I appreciate that is overly simplified but for now it all that we need to know. Firefox OS did not have a mechanism that allows Firefox extensions, when the work started on Marionette, to be installed which means that the current way that FirefoxDriver works by installing an extension and updating preferences on start-up will not work.

I have been working on Mozilla Open Web Apps project as one of my many things. This is a project that allows you to install web applications natively on to your OS and then use them like native applications. This relies on AppCache and other offline tricks but when you start the installed version of a web application you will not have access to extensions either. So both Firefox OS and Open Web Apps both block the current approach that FirefoxDriver has.

So in step the Automation and Tools Team, affectionately known as the A-Team internally at Mozilla, and more specifically Malini Das and Jonathan Griffin. They started working on a way to get an automation framework and build system going. Malini and Jonathan started working from the Selenium JSON Wire Protocol to create Marionette so that it can fit in with our plans for making Selenium a W3C Standard. Malini and Jonathan have stabilised Marionette for the current need in the Firefox OS project. I have been working to try to stabilise the desktop version, purely for my own needs on the Open Web Applications project at the moment but we be trying get this as complete as possible in the next couple of quarters.

How does it work?

Marionette works by using the socket transport of the JavaScript remote debugger as the JavaScript Remote debugger that is coming to Firefox. This just opens a socket in the browser that we then connect to and pass through commands that we want the browser to do. Does that mean that the JSON Wire Protocol isn't followed exactly? Yes, we have created our own take on the JSON Wire Protocol mainly just to fit in with the transportation mechanism that we are using. Since this is just a socket one could open a telnet connection to the browser and just send the commands. It is an extremely lightweight approach when we come to mobile and therefore benefits desktop implementations.

Does it only work on web content?

No! Marionette can also work against the browser chrome. Marionette is ultimately going to be the central item when it comes to automation frameworks within Mozilla. All of which are homegrown since we have unique requirements. We need to make sure that what we can do at least most of their functionality via Marionette if not all of it.


With Marionette being maintained in-house by the people that know the browser best, and being part of Mozilla build process, if Marionette gets broken it will get backed out very quickly. That means FirefoxDriver is going to be part of the Mozilla build process!

So when can I play with it? It is available now as part of Firefox debug builds. Why only debug builds? It still has to go through our security review since just having an open socket can drive the browser is quite scary and an opportunistic hacker could just run a selenium script and steal details. It will probably be preferenced off to start but is still a potentially scary problem.

    Area: blog

Automation Services Week 19 - 23 March

Thu 29 Mar 2012

This is 2nd to last week of the quarter and we have been busy closing off projects!

Henrik has implemented the dynamic job execution per node group for running daily and L10N Firefox builds. This means that we can consume Pulse messages and trigger tests for that specific platform. We are also looking at how we can find a solution for running scripts cross platform without having to duplicate certain parts of the code. This goal should be finished by the end of the quarter. He is also wanting to use a new tool called Pulse build monitor from the Automation and Tools Team to make processing of Pulse messages easier.

I have pretty much finished the validation code for Mozmill Dashboard. This goal should be finished too by the end of the quarter and then we just need a quick security review we can finally close it off!

Memchaser now has support for the new GC/CC API by listening for Observer notifications. We have also made updates for what info is being logged after a request from the JavaScript team. We have also been working on a few different features and still on track for a 3 April release!

We have also successfully published a new version of Nightly Tester Tools Addon and have passed review so make sure to get the update!

We have a number of projects that could do with some help so if you have free time have a look and see how you can help us!

Look out for future updates and if you want to join our team meetings you can get all of the details at here and you can also get notes from previous meetings.

    Area: blog

Automation Services Week - 12 - 17 March

Thu 22 Mar 2012

The team continues to close out projects for the quarter.

Henrik has continued working on the MozMill CI automation. He has added nodes for items to be run on windows and is now moving on to having items setup in Linux. He will then be working on code so that we distribute all the testing across all of the machines. This is really coming together and I look forward to the end result.

I have been working on getting things ready for working with Simon Stewart on the W3C Browser Automation Spec which will be doing next week! Hopefully not long after you read this you will be able to see it nearly complete!!

I have also been working on the Mozmill Dashboard validation code. This will hopefully be a trivial amount of work to finish off and then I will be submitting it for a security review with the Mozilla InfraSec team.

Henrik has released MemChaser 0.2.1 last week to be able to use the new console parser code that was added. Have a look at all the new details in his blog post.

Dave has been investigating a regression in memory usage within Firefox. All the details can be found in Bug 724553

We have a number of projects that could do with some help so if you have free time have a look and see how you can help us!

Look out for future updates and if you want to join our team meetings you can get all of the details at here and you can also get notes from previous meetings.

    Area: blog

Automation Services week 05 Mar - 09 Mar

Thu 15 Mar 2012

This week the team has been busy starting to wrap up projects as we near the end of the quarter.

We are releasing a new version of Nightly Tester Tools after have made a few minor changes. Henrik has been working with a couple contributors to get it done. You can download the latest version from AMO!

Geo, Henrik and Dave have been working to finish off our Mozmill CI system. Geo has been trying to solve and issue with Windows 2000 machines executing tests from Jenkins. Henrik has managed to get specific tests to be executed for different branches when Mozilla Pulse notifies about finished Firefox builds. This is all starting to look very good!

I have started work on data validation for the Mozmill Dashboard, the web application we use to view results. This is a minor task that is needed for the dashboard to become visible to the outside world!

Memchaser has had a couple pull requests opened recently. We are looking to add a few more features by the end of the month and release another version out on AMO!

We have a number of projects that could do with some help so if you have free time have a look and see how you can help us!

Look out for future updates and if you want to join our team meetings you can get all of the details at here and you can also get notes from previous meetings.

    Area: blog

Automation Services 27 February - 03 March

Thu 08 Mar 2012

It's amazing how quickly a week can go by!

The team has been hard at work on all of their projects.

Cameron has managed to get a stage and dev server up and running for Case Conductor so that people can help test it.

Dave has also done a proof of concept of a Proposal he blogged about a while back. With this, and being on track for a 1.0 release of Case Conductor at the end of the month this is looking great!

Our CI projects for Mozill are coming along nicely. Geo has been testing on getting this multiple Windows versions. Multi-platform products can create an interesting matrix of what needs testing.Henrik added support for Firefox ESR 10.0 to the CI system and will finishing that off soon! I am really excited by this project.

If you missed it last week, Memchaser 0.2 has been released and blogged about it! Thanks to the entire team for contributing to this!

Finally, David Guo, our intern, has been working on moving our Mozmill tests over to work with Mozmill 2.0. Currently we are working with 1.5.x and have been waiting for 2.0 to be stablised. The A*team have done a great job of stabising the code for us to now update!

We have a number of projects that could do with some help so if you have free time have a look and see how you can help us!

Look out for future updates and if you want to join our team meetings you can get all of the details at here and you can also get notes from previous meetings.

    Area: blog

Automation Services Week 20 - 24 February

Fri 02 Mar 2012

The team have been really busy over the last week. Dave Hunt joined the WebQA work week. They had their first ever live test day. I suggest that you go have a read of what happened on Dave's blog. I for one am really glad that it was success and hope that it can be emulated again in the new London office soon!

The team has also been really hard at work getting Memchaser 0.2 finished. Henrik Skupin has blogged about what we improvements have been made. This has been approved on AMO so you should be told about the update soon!!.

Cameron has been working on Case Conductor and has release 0.8. Major kudos to him on getting it all the way to this point and still be on track for a 1.0 release at the end of the month!

We have a number of projects that could do with some help so if you have free time have a look and see how you can help us!

Look out for future updates and if you want to join our team meetings you can get all of the details at here and you can also get notes from previous meetings.

    Area: blog

Using Travis CI for building and testing Firefox addons

Tue 28 Feb 2012

Let me start by explaining what Travis CI is. Travis CI is an open source cloud CI solution that integrates with GitHub so you can build and test your applications. It is free to use for OSS projects. They started only supporting Ruby and have grown really quickly to support other languages.

The other week I explained what my team is up to and I mentioned that we are working on a Firefox Addon to visualise what is happening with Garbage collection in the browser. This is very important since garbage collection taking a lot of time can have a negative impact on the user experience.

This week I wanted to make sure that we have a CI for the work that we are doing so we can spot if we have any regressions. We all know that manual testing is long, tedious and can miss things. Below are the steps that you need to take to get a JetPack addon built and tested. This is for Addons not built on the Builder site. In theory this will work on older addons too.

  • Add a .travis.yml to the root of the repository
  • Add the following lines to it. Make sure that you run it on a Node Worker so you don't swamp the ruby machines

         language: node_js
                - sh -e /etc/init.d/xvfb start
                - git submodule update --init
                - cd addon-sdk
                - . bin/activate
                - cd ../extension
               - DISPLAY=':99.0'
             script: cfx test     

  • Turn on Travis CI for that project. You can do this by logging in to Travis CI and flicking the switch on your profile page or you can set it up manually.
  • Commit the .travis.yml file to your remote repo and watch it run.

You can see an example of this here. Thanks to Travis CI guys for making this possible and if you find it useful show them some Love!

    Area: blog

Automation Services Week 13 - 17 Feburary

Fri 24 Feb 2012

This week has been relatively quiet for the team in terms of their usual projects mainly because there was a chemsplill last week. A chemsplill is where a major exploit has been found and we need to get this closed as quickly as possible so that Firefox users are safe from malicious code execution. Unfortunately details of this have not been released yet but you will have noticed that 10.0.2 was released last week.

The team has also been working hard on getting more tasks for contributors. We have also added more mentored projects, below is links to projects that have tasks community members can get involved in

Cameron has been working hard on Case Conductor, the a test case management tool. 0.8 is now done and ready for deployment. Cameron is also working with WebQA to get more test coverage on it and get it out the door. Cameron has been discussing with the Case Conductor team about how we can have a 1.0 release by the end of March. Look out for more on this in weeks to come.

Look out for future updates and if you want to join our team meetings you can get all of the details at here and you can also get notes from previous meetings.

    Area: blog

Automation Services Week 6 February - 10 February

Wed 15 Feb 2012

Automation Services, the team I co-lead with Henrik Skupin, are going to start sending out weekly updates so that everyone can be aware of what we have been upto in the last week on our main projects.

Nightly Tester Tools

Our team is now taking ownership of the addon from Heather Arthur. This extension has over 100000 active users at the moment and we want to grow that community. We need to still do a few logistical things with Heather to get it moved over and we hope to have this sorted in the next few weeks. Once that is done we will do some house keeping on the project like changing the repository structure and getting build scripts in place. Next quarter we hope to do some work on the project and will create some issues out that will be mentored by someone in the team for anyone to work on!

Actually we will be creating mentored issues for a lot of our projects at the moment and will have a central place for these soon! In the mean time you can have a look at:


One of the major projects on the go at the moment is a project to see where we have large GC/CC pauses in the browser. Long pauses lead to the browser feeling jerky which isn't a great user experience. Last week Henrik blogged about the MemChaser 0.1.1 release and got a couple of good comments! It has helped identify a few bugs. A really good example is where we identified memory leak with Add-ons SDK bug, which is already partly fixed for SDK v1.6 and even for the upcoming v1.5 release. Currently we have over 1000 downloads and nearing 400 active daily users (statistics). We need to get docs in place for further planning on the project but this is a great start!

Look out for future updates and if you want to join our team meetings you can get all of the details at here and you can also get notes from previous meetings.

    Area: blog

Automating Web Performance data collection with BrowserMob Proxy and Selenium

Thu 26 Jan 2012

Back in 2008, David Henderson and I wanted to try automate collecting client side performance data for the web application that we were working on at the time. We were getting a large number of complaints from users about load time and we need to try solve this. The way that we decided on doing this was to use Selenium, which was already running our tests, and hack the Selenium server to give us the information we needed. What we came up with was showcased at GTAC 2009. It was really good for the time and we were happy.

About a year later just as the new movement for HTTP Archive or HAR as its commonly known was taking off I found that Jan "Honza" Odvarko had created a Firebug add on to export the network tab to HAR. This, when used with Selenium, could mean that we could get the same data that the browser wanted with little to no effort. Run your tests as you were with the new WebDriver API and get it to programmatically install the necessary addons as well as set all the preferences needed. I wrote this for Python and for .NET. This works really well but now limits this type of data to only being collected within Firefox. Some modern web applications can send back totally different javascript and sprites dependant on which browser hits.

What if you could run your Selenium tests and collect the same info as the if you were using the Firebug Net tab but using any browser. Browsermob Proxy is a good way to collect this information and it has a programmatic interface that allows us to set it before our tests start. It can also return a HTTP Archive of the traffic that it is routing. I have released the Python Library for BrowserMob Proxy that can be injected into tests quite easily. I have put an example below.

from browsermobproxy import Server
server = Server("path/to/browsermob-proxy")
proxy = server.create_proxy

from selenium import webdriver
profile  = webdriver.FirefoxProfile()
driver = webdriver.Firefox(firefox_profile=profile)

proxy.har # returns a HAR JSON blob


If we now feed the data from proxy.har into HAR Viewer we can get a nice waterfall of the traffic as below! Jari Bakken has a useful commandline version called Har.

waterfall from HAR of this site

I recently showed this off at London Web Performance. Unfortunately it wasn't recorded but you can view the slides here

    Area: blog

The beginning of a standard for browser automation

Thu 12 Jan 2012

WebDriver, the browser automation framework we all love, is on its way to becoming a browser standard. WebDriver as an OSS project is 5 years old and started being merged into the Selenium project about 3 years ago. From there we have seen WebDriver grow. It has had a lot updates in that time. The team is currently averaging ~100 commits a week.

In the short time, from GTAC 2009 when they were created, the Ruby bindings created by Jari Bakken have had over a million downloads! The Python bindings have around 3000 downloads a week and growing. Jim Evans work on the .NET bindings is great, I don't have figures because of Nuget. Same with the Java bindings. Maven makes it difficult to know how many downloads.

We have also seen that Selenium Jobs have overtook the commercial counterparts. All of these things point to us needing a standard to make sure if we can do the same thing on multiple devices and multiple OSes.

Yesterday we held a W3 meeting in the Google UK office. It was great to be able to get in a room and discuss what needed. A plan was hatched on what needs to be done next, and it is going to be a lot of work.You can see the minutes of the meeting here to see what we discussed.

Mozilla, Opera and Google are happy with the direction we are going and long may it last! It will be great to get Microsoft and Apple involved soon.

Remember that all of this work is happening out in the open and you can watch the Hg repository for the editors draft updates.

    Area: blog