Explaining how the real world works!     RSS Feed The Automated Tester on Twitter The Automated Tester on LinkedIn The AutomatedTester on github

Who wants to be an alpha tester for Marionette?

Wed 01 Jul 2015

Are you an early adopter type? Are you an avid user of WebDriver and want to use the latest and great technology? Then you are most definitely in luck.

Marionette, the Mozilla implementation of the FirefoxDriver, is ready for a very limited outing. There is a lot of things that have not been implemented or, since we are implementing things agains the WebDriver Specification they might not have enough prose to implement (This has been a great way to iron out spec bugs).

Getting Started

At the moment, since things are still being developed and we are trying to do things with new technologies (like writing part this project using Rust) we are starting out with supporting Linux and OS X first. Windows support will be coming in the future!

Getting the driver

We have binaries that you can download. For Linux and for OS X . The only bindings currently updated to work are the python bindings that are available in a branch on my fork of the Selenium project. Do the following to get it into a virtualenv:
  1. Create a virtualenv
  2. activate your virtualenv
  3. cd to where you have cloned my repository
  4. In a terminal type the following: ./go py_install

Running tests

Running tests against marionette requires that you do the following changes (which hopefully remains small)

Update the desired capability to have marionette:true and add binary:/path/to/Firefox/DeveloperEdition/or/Nightly. We are only supporting those two versions of at the moment because we have had a couple in compatible issues that we have fixed which means speaking to Marionette in the browser in the beta or release versions quite difficult.

Since you are awesome early adopters it would be great if we could raise bugs. I am not expecting everything to work but below is a quick list that I know doesn't work.

  • No support for self-signed certificates
  • No support for actions
  • No support for Proxy (but will be there soon)
  • No support logging endpoint
  • I am sure there are other things we don't remember

Thanks for being an early adopter and thanks for raising bugs as you find them!

    Area: blog

WebDriver Specification - Have you read it lately?

Tue 09 Jun 2015

A lot of work has gone into the WebDriver Specification this year. The methods in there have had a major make over to make them more specific in the steps that are required as well as having the relevant links. Go have a read of it and feel free to raise bugs against it, we will be updating it quite regularly. You can see all the work that is happening on Github. We do everything via pull requests so you can read things before they land.

My team have also been working hard at making sure that our implementation is following the specification and are making some great leaps with it. I will be releasing a development version of the python bindings soon that use the httpd, like InternetExplorerDriver and ChromeDriver, to drive the browser. Currently our httpd only works against Nightly but there is a merge to Aurora happening soon when we will be sending out links for you to start playing with it all. I am actually looking forward to some of the feedback that we get about it.

    Area: blog

Marionette - Where we are

Mon 09 Mar 2015

I thought that I spend some time describing where we are with the Marionette project. For those that don't know, Marionette is the project to implement WebDriver in Firefox. We are implementing the WebDriver based on the W3C WebDriver specification.

We have working quite hard to get as much of implementation done as per specification. One thing to note is there is a few places where the specification and the open source project have diverged but hopefully a Selenium 3 release can align them again.

So... what is left to do for the Marionette project to be able to ship it's 1.0 release?

and a few other things. Feel free to look at our current roadmap!

That means we have some of the big ticket items, like modal dialog support, landed! We have some of the actions landed and most importantly we have large parts of a driver executable (written in Rust!), like chromedriver or internetexplorerdriver, completed.

Somethings are going slower than anticipated and other sections are shooting along so all in all I am really pleased with the current progress!

If you want to help out, we have a number of good first bugs that you can all join in!

    Area: blog

Bugsy 0.4.0 - More search!

Sat 06 Dec 2014

I have just released the latest version of Bugsy. This allows you to search bugs via change history fields and within a certain timeframe. This allows you to find things like bugs created within the last week, like below.

I have updated thedocumentation to get you started.

>>> bugs = bugzilla.search_for\
                    .change_history_fields(["[Bug creation]"])\
                    .timeframe("2014-12-01", "2014-12-05")\

You can see the Changelog for more details..

Please raise issues on GitHub

    Area: blog

WebDriver Face To Face - TPAC 2014

Tue 04 Nov 2014

Last week was the 2014 W3C TPAC. For those that don't know, TPAC is a conference where a number of W3C working groups get together in the same venue. This allows for a great amount of discussions between groups and also allows people to see what is coming in the future of the web.

The WebDriver Working Group was part of TPAC this year, like previous years, and there was some really great discussions.

The main topics that were discussed were:

  • We are going to be moving more discussions to the mailing list. This is to prevent people waiting for a Face to face to discuss things
  • The Data model of how data is sent over the wire between the remote and local ends
  • Attributes versus properties -This old chestnut
  • An approach to moving some of the manual tests that are used for W3C specs to automated ones with WebDriver - This is exciting

The meeting minutes for Thursday and Friday

    Area: blog

WebDriver F2F - London 2014

Mon 14 Jul 2014

Last week saw the latest face to face of the WebDriver Working Group held at Facebook. This meeting was important as this is hopefully the last face to face before we go to Last call allowing us to concentrate on issues that come up during last call.

This meeting was really useful as we were a number of discussions around the prose of the spec when it comes to conformance and usability of the spec, especially when given to implementors who have never worked on WebDriver.

The Agenda from the meeting can be found here

The notable items that were discussed are:

  • Merge getLocation and getSize to single call called getElementRect. This has been implemented in FirefoxDriver already
  • Describe restrictions around localhost in security section
  • How the conformance test will look (Microsoft have a huge raft tests they are cleaning up and getting ready to upstream!)
  • Actions has been tweaked from the original straw man delivered by Mozilla, hopefully see the new version in the next few weeks.

To read what was discussed you can see the notes for Monday and Tuesday.

    Area: blog

Bugsy 0.3.0 - Comments!

Mon 14 Jul 2014

I have just released the latest version of Bugsy. This allows you to get comments from Bugzilla and add new comments too. This API is still experimental so please send back some feedback since I may change it to real world usage.

I have updated the documentation to get you started.

>>> comments = bug.get_comments()
>>> comment[0].text
"I <3 Cheese"
>>> bug.add_comment("And I love bacon")

You can see the Changelog for more details.

Please raise issues on GitHub

    Area: blog

Bugsy 0.2 - Now with 100% more search!

Thu 26 Jun 2014

I have updated Bugsy to now have the ability to search Bugzilla in a meaningful way. I have updated the documentation to get you started.

For example to search Bugzilla you would do

>>> bugs = bugzilla.search_for\
...                .keywords("checkin-needed")\
...                .include_fields("flags")\
...                .search()

You can see the Changelog for more details..

Please raise issues on GitHub

    Area: blog

Introducing Bugsy - Client Library for interacting with Bugzilla

Mon 16 Jun 2014

I have created a library for interacting with Bugzilla using the native REST API. Bugsy allows you to get bugs from Bugzilla, change what you need to and then post it back to Bugzilla. I have created documentation to get you started.

For example to get a bug you would do

import bugsy
bugzilla = bugsy.Bugsy()
bug = bugzilla.get(123456)

and then to put it back, or if there is no bug ID (like if you were creating it) then you would do

import bugsy
bug = bugsy.Bug()
bug.summary = "I really realy love cheese"
bug.add_comment("and I really want sausages with it!")
bugzilla = bugsy.Bugsy("username", "password")
bug.id #returns the bug id from Bugzilla

Searching Bugzilla is not currently supported but will definitely be there for the next version.

Please raise issues on GitHub

    Area: blog

My Ideal build, test and land world

Thu 05 Jun 2014

The other week I tweeted I was noticing for that day we had 1 revert push to Mozilla Inbound in every 10 pushes. For those that don't know, Mozilla Inbound is the most active integration repository that Firefox code lands in. A push can contain a number of commits depending on the bug or if a sheriff is handling checkin-needed bugs.

This tweet got replies like, and I am paraphrasing, "That's not too bad", "I expected it to be worse". Personally I think this is awful rate. Why? On that day, only 80% of pushes were code changes to the tree. The bad push and the revert leads to no changes to the tree but still uses our build and test infrastructure. This mean that at best we can(on that day) only be 80% efficient. So how can we fix this?

Note: A lot of this work is already in hand but I want to document where I wish them to go. A lot of the issues are really paper cuts but it can be death by 1000 paper cuts.


Mach, the current CLI dispatch tool at Mozilla, passes the build detail to the build scripts. It is a great tool if you haven't been using it yet. The work the build peers have done with this is pretty amazing. However I do wish the build targets passed to Mach and then executed were aligned with the way that Chromium, Facebook, Twitter and Google build targets worked.

For example, working on Marionette, if I want run marionette tests I would do |./mach //testing/marionette:test| instead of the current |./mach marionette-test| . By passing in the directory we are declaring what we want explicitly to be built and test.. The moz.build file should have a dependency saying that we need a firefox binary (or apk or b2g). The test task in the moz.build in testing/marionette folder would pick runtests.py and then pass in the necessary arguments ideally based on items in the MozConfig. Knowing the relevant arguments based on the build is hard work involving looking at your history or at a wiki.

Working on something where it has unit tests and mochitests or xpcshell tests? It's simple to just change the task. E.g. ./mach //testing/marionette:test changes quickly to //testing/marionette:xpcshell. Again, not worrying about arguments when we can create sane defaults based on what we just built. I have used testing in my examples because it is simple to show different build targets on the same call.

The other reason declaring the path (and mentioning the dependencies in the same manner) is that if you call |./mach //testing/marionette:test| after updating your repo it will do an incremental build (or a clobber if needed) without you knowing you needed it. Manually clearing things or running builds just to run tests is just busy work again.

Reviews and Precommit builds/tests

Want a review? You currently either have to use bzexport or manually create a diff and upload it to the bug and set the reviewer. The Bugzilla team are working to stand up review board that would allow us to upload patches and has a gives us a better review tool.

The missing pieces for me are: 1)We have to manually pick a reviewer and 2) that we don't have a pre commit build and test step.

1) I have been using Opera's Critic for reviewing Web Platform Tests. Having the ability to assign people to review changes for a directory means that reviewing is everyones responsibility. Currently Bugzilla allows you to pick a reviewer based on the component that the bug is on. Sometimes a patch may span other areas and you then need to figure out who a reviewer should be. I think that we can do better here.

As for 2) don't necessarily need to do everything but the equivalent of a T-Style run would suffice I'm my opinion. We could even work to pair this down more to be literally a handful of tests that regularly catch bugs or make it run tests based on where the patch was landing.

Why does this matter, we have try that people can use and "my code works" and "it was reviewed, it will compile". Mozilla Inbound was closed for a total of 2 days (48+ hours) in April and 1 day (24hrs) in May. At the moment I only have the data on why the tree was closed, not the individual bugs that caused the failure, but a pre commit step would definitely limit the damage. The pre commit step might also catch some of the test failures (if we had test suites we could agree on for being the smoke test suite) which had Mozilla Inbound closed for over 2 1/2 (61+ Hours) days in April and over 3 days (72+ hours) in May.

Landing code

Once the review has passed we still have to manually push the code or set a keyword in the bug (checkin-needed) so that the sheriffs can land it. This manual step is just busy work really that isn't really needed. If something has a r+ then ideally we should be queueing this up to be landed. This is minor compared to manual step required to update the bug with the SHA when it has landed, when it is landed we should be updating the bug accordingly. It's not really that hard to do.

Unneeded manual steps have an impact on engineering productivity which have a financial cost that could be avoided.

I think the main reason why a lot of these issues have never been surfaced is there is not enough data to show the issues. I have created a dashboard, only has the items I care about currently but could easily be expanded if people wanted to see other bits of information. The way we can solve the issues above is being able to show the issues.

    Area: blog