Explaining how the real world works!     RSS Feed The Automated Tester on Twitter The Automated Tester on LinkedIn The AutomatedTester on github

Treeclosure stats

Fri 21 Mar 2014

As the manager of the sheriffs I am always interested in how ofter the sheriffs, and anyone else, closes the tree. For those who don't know who the Mozilla Sheriffs are, they are the team that manage the code landing in a number of Mozilla trees. If a bad patch lands they are the people who typically back it out. There has been some recent changes in the way the infrastructure does things which has led to a few extra closures. Not having the data for this I went and got it (you can see the last year's worth of data for Mozilla-Inbound below)

2013-03
         infra: 14:59:38
         no reason: 5 days, 12:13:31
         total: 6 days, 3:13:09
2013-04
         infra: 22:21:18
         no reason: 3 days, 19:30:21
         total: 4 days, 17:51:39
2013-05
         infra: 1 day, 2:03:08
         no reason: 4 days, 11:30:41
         total: 5 days, 13:33:49
2013-06
         checkin-compilation: 10:04:17
         checkin-test: 1 day, 5:48:15
         infra: 18:44:06
         no reason: 5:05:59
         total: 2 days, 15:42:37
2013-07
         backlog: 22:38:39
         checkin-compilation: 1 day, 13:05:52
         checkin-test: 2 days, 16:43:53
         infra: 1 day, 2:16:02
         no reason: 0:30:13
         other: 1:32:23
         planned: 4:59:09
         total: 6 days, 13:46:11
2013-08
         backlog: 4:13:49
         checkin-compilation: 1 day, 23:49:34
         checkin-test: 1 day, 12:32:35
         infra: 13:06:19
         total: 4 days, 5:42:17
2013-09
         backlog: 0:21:39
         checkin-compilation: 1 day, 8:27:27
         checkin-test: 2 days, 15:17:50
         infra: 15:34:16
         other: 2:02:07
         planned: 3:16:22
         total: 4 days, 20:59:41
2013-10
         checkin-compilation: 15:29:45
         checkin-test: 3 days, 10:41:33
         infra: 16:31:41
         no reason: 0:00:05
         other: 0:09:01
         planned: 2:30:35
         total: 4 days, 21:22:40
2013-11
         checkin-compilation: 1 day, 9:40:25
         checkin-test: 4 days, 18:41:35
         infra: 1 day, 19:11:36
         no reason: 0:05:54
         other: 3:28:40
         planned: 1:50:20
         total: 8 days, 4:58:30
2013-12
         backlog: 5:07:06
         checkin-compilation: 18:49:29
         checkin-test: 1 day, 16:29:16
         infra: 6:30:03
         total: 2 days, 22:55:54
2014-01
         backlog: 1:54:43
         checkin-compilation: 20:52:34
         checkin-test: 1 day, 12:22:01
         infra: 1 day, 5:37:14
         no reason: 1:20:46
         other: 4:53:42
         planned: 3:48:16
         total: 4 days, 2:49:16
2014-02
         backlog: 3:08:18
         checkin-compilation: 1 day, 12:26:35
         checkin-test: 15:30:42
         infra: 19:40:38
         no reason: 0:00:16
         other: 0:47:38
         total: 3 days, 3:34:07
2014-03
         backlog: 8:52:34
         checkin-compilation: 19:27:21
         checkin-test: 1 day, 0:37:55
         infra: 19:47:13
         other: 2:53:21
         total: 3 days, 3:38:24

I created a graph of the data showing Mozilla Inbound since we started using it in August 2012 till now.

Closures by reason on Mozilla Inbound

The first part of the graph there wasn't any data for specific things but the sheriffs changed that in the middle of last year. I am hoping that we can get information like this, and other interesting back out info into a "Tree Health Report" in Treeherder (The TBPL replacement the Automation and Tools Team is developing).

    Area: blog

Management is hard

Thu 13 Mar 2014

I have been a manager within the A*Team for 6 months now and I wanted to share what I have learnt in that time. The main thing that I have learnt is being a manager is hard work.

Why has it been hard?

Well, being a manager is requires a certain amount of personal skills. So being able to speak to people and check they are doing the tasks they are supposed to is trivial. You can be a "good" manager on this but being a great manager is knowing how to fix problems when members of your team aren't doing the things that you expect.

It's all about the people

As an engineer you learn how to break down hardware and software problems and solve them. Unfortunately breaking down problems with people in your team are nowhere near the same. Engineering skills can be taught, even if the person struggles at first, but people skills can't be taught in the same way.

Working at Mozilla I get to work with a very diverse group of people literally from all different parts of the world. This means that when speaking to people, what you say and how you say things can be taken in the wrong way. It can be the simplest of things that can go wrong.

Careers

Being a manager means that you are there to help shape peoples careers. Do a great job of it and the people that you manage will go far, do a bad job of it and you will stifle their careers and possibly make them leave the company. The team that I am in is highly skilled and highly sought after in the tech world so losing them isn't really an option.

Feedback

Part of growing people's careers is asking for feedback and then acting upon that feedback. At Mozilla we have a process of setting goals and then measuring the impact of those goals on others. The others part is quite broad, from team mates to fellow paid contributors to unpaid contributors and the wider community. As a manager I need to give feedback on how I think they are doing. Personally I reach out to people who might be interacting with people I manage and get their thoughts.

But I don't stop at asking for feedback for the people I manage, I also ask for feedback about how I am doing from the people I manage. If you are a manager and never done this I highly recommend doing it. What you can learn about yourself in a 360 review is quite eye opening. You need to set ground rules like the feedback is private and confidential AND MUST NOT influence how you interact with that person in the future. Criticism is hard to receive but a manager is expected to be the adult in the room and if you don't act that way you head to the realm of a bad manager, a place you don't want to be.

So... do I still want to be a manager?

Definitely! Being a manager is hard work, let's not kid ourselves but seeing your team succeed and the joy on people's faces when they succeed is amazing.

    Area: blog

Don't write "Five Hidden Costs of X" but when you do I will reply

Wed 12 Mar 2014

Recently I was shown that Telerik did a "Five Hidden Costs of Selenium". I knew straight away from the title that this was purely a marketing document targeting teams with little to no automation skills to do automation. For what it is worth, if you want to do automation you should really hire the right engineers for the job.

My offence with the article is not that its wrong, there are a few items I disagree with which are documented below, but with it trying to sell snake oil or silver bullets. So let's even up the argument a bit. Note I am only comparing the WebDriver parts since if it were purely Selenium IDE vs Teleriks tool then I think it would be fair comments.

No Build/Scheduling Server

Telerik say we don't have those items and we don't. We don't want to be working on them since there are some awesome open source products with many years worth of engineering effort in them that you can use. These are free and allow a great amount freedom of customisation. They also work really well if you have hybrid systems as part of your test. Have you seen that ThoughtWorks has open sourced Go which is a great product from people who have been doing continuous integration for nearly, if not more, than a decade. Don't want to host it yourself, because managing servers is a hidden cost in all worlds, then look at the huge amount of Continuous Integration as a Service companies out there.

Execution Agents/Parallel Running

It says this is a 3rd party plugin which is not true. The Selenium server has a remote server system built in and if the correct arguments are passed in it can become a grid with a central hub managing which nodes are being used. This is called Selenium Grid.

The one thing, from the documentation, is that you have to host all these nodes yourself. Does it create hermetic environments to run against when scheduling? Hermetic environments is something that each core developer would want and if we can't give it then its not worth releasing. There are Infrastructure as a Service companies that WebDriver tests can be hooked up to so you don't need to maintain all the infrastructure yourself. The infrastructure costs can be quite expensive if you're a smallish team, using a service here can help. Unfortunately Telerik don't offer execute nodes as a service so you'll have to manage them yourself.

Also, its fine that nUnit doesn't support parallel execution, get your scheduling server to run a task for each browser and these tasks could be run in parallel.

Reporting

This is best done by the continuous integration server as part of the build and test. These take the output from the tasks they have told them to execute and then report. Having this as a selling point in marketing documentation feels like it is just targeting the untrained staff.

Multi-Browser Support

This is where you would think we would be even but Telerik is stuck to desktop browsers. WebDriver, due to its flexible transport mechanism (it's like we thought about it or something, means anyone can implement the server side and control a browser and all languages just work. We recently saw that with Blackberry creating an implementation for their devices. We have Selendroid for Android and iOS-Driver for iOS. Mobile is the future so only supporting the major desktop browsers is going to limit your future greatly.

Jim also mentioned that you would need to build a factory and teach it to get things running against multiple projects. You do have to but here is a link to the Selenium projects' way of handling it. We need to run our tests in multiple environments and we do it pretty well. This is just a one time sunk cost.

Maintenance of tests

I might be pulling a Telerik here but having tests look like the following
lolwat? What is SENDSELECTEDDiv even?.

Being able to code as an automation engineer is crucial. Being able to write good tests is useful too! I am biased but Mozilla has some really good examples that show good maintainable tests. Tests are trivial to write and update since they invested in a good pattern for tests. Record and playback tools have never had the ability to write maintainable tests and for them to be using meaningful API names. Also, it hampers (as in no one will take you seriously) your career calling yourself an automation engineer and only using record and playback tools.

Now beware of the snake oil that is being offered by vendors and for all that is holy... if you want to do automation don't do record and replay. I, and my peers, will not even let you past a phone screen if you don't show enough knowledge about coding and automation.

Also if Telerik was thinking straight they would be wrapping webdriver and then they would get everything that is happening in the webdriver world. Knowing your tests will always work in the browser no matter what platform (including mobile) is a huge selling point. And its standards based, feels like a no-brainer but i am obviously biased.

    Area: blog

WebDriver Face To Face - February 2014

Fri 28 Feb 2014

This week saw the latest WebDriver F2F to work on the specification. We held the meeting at the Mozilla San Francisco office.

The agenda for the meeting was placed, as usual, on the W3 Wiki. We had quite a lot to discuss and, as always, was a very productive meeting.

The meeting notes are available for Tuesday and Wednesday. The most notable items are;

  • Changing switchToFrame to only accept a WebElement or Index
  • Adding switchToParentFrame to the API
  • Potential changes to the way we do clicks on elements larger
  • Numerous bugs in the spec
  • Removing findElement(By.className and By.Id)
  • Landed the User Interactions spec and created an endpoint for batched actions

The other amazing things that happened was we had Blackberry join the working group, especially after their announcement saying they have created an implementation.

And... how can I forget about this...

The specification is getting a lot of attention from the people that we need and want which makes me really excited!

    Area: blog

Updated BrowserMob Proxy Python Documentation

Mon 03 Feb 2014

After much neglect I have finally put some effort into writing some documentation for the BrowserMob Proxy Python bindings however I am sure that they can be a lot better!

To do this I am going to need your help! If there are specific pieces of information, or examples, that you would like in the documentation please either raise an issue or even better create a pull request with the information, or example, that you would like to see in the documentation.

    Area: blog

TPAC 2013 - WebDriver Face To Face and more

Wed 20 Nov 2013

Last week I was at W3 TPAC for week of face to face meeting to discuss WebDriver and other W3C specifications that other working groups are working on.

Our initial agenda went up just before the meeting and we were lucky enough to get through all the items. If you would like to read the notes for the meeting (Monday and Tuesday.

Highlights from the meeting are

  • We are rescoping the specification to have the relevant things we need - we have been suffering from scope creep so we are removing things that aren't complete or won't be completed soon. We will be creating WebDriver "Level 2" specification which those features will go into.
  • We have roughly 36 weeks worth of work to get the spec and tests done. - I will be writing details on how anyone can help make this time line realistic so we hit all of our milestones!
  • We are going to be adding an API for when an element is intractable via mouse/touch, we haven't decided on a name yet but will be in the spec soon
  • Simon and I were speaking to other editors/working groups to see if we could offload definitions of things to them. It seemed to be positive, let's see where things head.

There are other actions from the meeting that need to be done but I think the items above cover the main points, at least for me, that came out of the meeting.

I found the rest of the week actually really useful from a networking perspective and from a learning perspective. I have a lot of changes that I need to put into the WebDriver spec and have been getting feed back about when i am doing it wrong which is great!

    Area: blog

Test The WebForward - Shenzhen, China

Tue 19 Nov 2013

Just over a week ago I was in Shenzhen, China helping with Test The Web Forward. A great initiative to get anyone and everyone to write tests for browser companies to use. The tests are conformance tests for W3C.

In the next couple of weeks I am going to document what is needed to help out and how you, yes you, can help! The WebDriver open source test suite needs to be checked against the WebDriver Specification and then ported over or bugs raised against the spec or implementations. There will be bugs which is a great thing!

P.S. We had 4 patches with ~10 tests from people who had never used WebDriver so I consider that a great success. We also noticed a number of pain points we need to fix before we get more people involved

    Area: blog

HackBMTH last weekend

Wed 17 Jul 2013

Last weekend there was another HackBMTH, this time hosted by Adido. This was my first time going to a hackathon in the Bournemouth area and it was a great. Jonathan Ginn and Adam Howard are the organisers and to be honest, they did a brilliant job of organising the event.

There were people from all over Bournemouth and as far away as Reading who came to work on their little projects or create new hacks. There were ops people helping people setup Apache and a few other things. There were people creating games, one of which was a tank game that took a picture from the webcam and turned that into the level. Finally a use for QR codes!

I worked on my GitHub+Travis Firefox Add-on. I worked to start pulling in the build history of Travis CI Projects to the project page on Github. Below is a screenshot of what it looks like at the moment. It's still rough around the edges.

You can install a pre-release from Github if you want play with it. Feel free to raise bugs about it on the Github page.

The only thing that I was disappointed with was that there was only 1 female. This I should point out I don't think is the fault of the organisers. When I worked in Southampton there weren't many females applying for positions so I know the area suffers from a lack of them. I know that there are some as students, having met them at B & W Meet, so I hope that more will come. After all, one of the main reasons there are not enough woman in tech is because there are not enough woman in tech, to paraphrase Sheryl Sandberg.

    Area: blog

W3C WebDriver Face To Face - June 2013

Thu 27 Jun 2013

A couple weeks ago was the latest face to face of the W3C Browser Automation (WebDriver) Working Group. We as a group don't meet anywhere near enough in my opinion but when we do the conversations we have are quite amazing.

Below is the Agenda that we used for discussions. If you want to read the minutes you can read them here and here.

If you have any questions please leave a comment!

    Area: blog

The tale of Selenium bug 141

Wed 26 Jun 2013

The selenium project has for a while now wanted to create a library that allows user emulation in the browser. The project has done a reasonable job at this so far with respect to this. We check if items are in the DOM, we check the visibility of items and we normalise text from the browser, amongst our other amazing talents!

Our default position is we enforce the idea of emulation. "How would a user do this action?" is the main question we ask ourselves! So when a bug that doesn't meet that we tend not to implement it. One of these times is bug 141 - WebDriver lacks HTTP response header and status code methods. This is one of the most contentious bugs in the Selenium repository. People see Selenium as more than browser automation library, they see it as a web testing framework. Yes, Selenium RC was like this but then again Selenium RC also had ~140 methods hung off one object. We made a mistake in the past, let's try not to repeat it! Anyway, I digress, in a web testing framework world there may be times where there are use cases where you need to find out data from the response header and status codes.

So is this a feature we want to add to WebDriver? No. "But David, we do!?!?!?" I hear you cry! The use case that comes up regularly is I want to know if a server is returning a 404 or 500 when I access a page. Why is there a need to start a browser to find this out? When I asked this question on twitter, my friend and fellow committer Kevin Menard said that it could be useful to to see the page response on failure and some pages can only be accessed with a real browser. The use cases that Kevin had is you want to check that the endpoint that you originally hit might might change based on the user agent passed in or there might be a site that follows the single page idea and breaks push state and therefore most of the web semantics. I admit Kevin is right with those use cases. However its only looking at a small part of the issue and there are a lot of corner cases. The major one for me is watching XHRs. What if one of your XHR returns a 500 and you get an error? Should WebDriver show you that information as well? I am sure you can see how quickly this can escalate. If you start worrying about all requests, and you should if you are worried about the initial request to a page, then you want the browser to capture all of this information and return it in a standard format. There just happens to be a format called Http Archive (HAR).

So does the project take the task of creating a mechanism for extracting a HAR from the browser? That could be near impossible on browsers that don't readily share network usage information. So we go from WebDriver being a browser automation framework for user emulation to browser automation framework for user emulation and networking information gathering. That might be a little obtuse but its meant to be.

While Selenium's main use case is for test automation, is not solely designed for that. We want to be the best at browser automation. Proxies are really good at managing network information and reporting on it in standard formats. Examples are fiddler and browsermob-proxy. And you can even hook them into Selenium WebDriver! Like the following:

from browsermobproxy import Server
server = Server("path/to/browsermob-proxy")
server.start()
proxy = server.create_proxy()

from selenium import webdriver
profile  = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)


proxy.new_har("google")
driver.get("http://www.google.co.uk")
proxy.har # returns a HAR JSON blob

server.stop()
driver.quit()

Yes it might be a little verbose to use them but if you really want to see that type of information then you should do it properly otherwise the real errors, that you originally wanted, will go missing.

Putting in half a solution into Selenium is just going to create scope creep and a large number of bugs because its not doing the "right" thing. The "right" thing is far too subjective for it to be done correctly.

    Area: blog