Explaining how the real world works!     RSS Feed The Automated Tester on Twitter The Automated Tester on LinkedIn The AutomatedTester on github

Treeclosure stats

Fri 21 Mar 2014

As the manager of the sheriffs I am always interested in how ofter the sheriffs, and anyone else, closes the tree. For those who don't know who the Mozilla Sheriffs are, they are the team that manage the code landing in a number of Mozilla trees. If a bad patch lands they are the people who typically back it out. There has been some recent changes in the way the infrastructure does things which has led to a few extra closures. Not having the data for this I went and got it (you can see the last year's worth of data for Mozilla-Inbound below)

2013-03
         infra: 14:59:38
         no reason: 5 days, 12:13:31
         total: 6 days, 3:13:09
2013-04
         infra: 22:21:18
         no reason: 3 days, 19:30:21
         total: 4 days, 17:51:39
2013-05
         infra: 1 day, 2:03:08
         no reason: 4 days, 11:30:41
         total: 5 days, 13:33:49
2013-06
         checkin-compilation: 10:04:17
         checkin-test: 1 day, 5:48:15
         infra: 18:44:06
         no reason: 5:05:59
         total: 2 days, 15:42:37
2013-07
         backlog: 22:38:39
         checkin-compilation: 1 day, 13:05:52
         checkin-test: 2 days, 16:43:53
         infra: 1 day, 2:16:02
         no reason: 0:30:13
         other: 1:32:23
         planned: 4:59:09
         total: 6 days, 13:46:11
2013-08
         backlog: 4:13:49
         checkin-compilation: 1 day, 23:49:34
         checkin-test: 1 day, 12:32:35
         infra: 13:06:19
         total: 4 days, 5:42:17
2013-09
         backlog: 0:21:39
         checkin-compilation: 1 day, 8:27:27
         checkin-test: 2 days, 15:17:50
         infra: 15:34:16
         other: 2:02:07
         planned: 3:16:22
         total: 4 days, 20:59:41
2013-10
         checkin-compilation: 15:29:45
         checkin-test: 3 days, 10:41:33
         infra: 16:31:41
         no reason: 0:00:05
         other: 0:09:01
         planned: 2:30:35
         total: 4 days, 21:22:40
2013-11
         checkin-compilation: 1 day, 9:40:25
         checkin-test: 4 days, 18:41:35
         infra: 1 day, 19:11:36
         no reason: 0:05:54
         other: 3:28:40
         planned: 1:50:20
         total: 8 days, 4:58:30
2013-12
         backlog: 5:07:06
         checkin-compilation: 18:49:29
         checkin-test: 1 day, 16:29:16
         infra: 6:30:03
         total: 2 days, 22:55:54
2014-01
         backlog: 1:54:43
         checkin-compilation: 20:52:34
         checkin-test: 1 day, 12:22:01
         infra: 1 day, 5:37:14
         no reason: 1:20:46
         other: 4:53:42
         planned: 3:48:16
         total: 4 days, 2:49:16
2014-02
         backlog: 3:08:18
         checkin-compilation: 1 day, 12:26:35
         checkin-test: 15:30:42
         infra: 19:40:38
         no reason: 0:00:16
         other: 0:47:38
         total: 3 days, 3:34:07
2014-03
         backlog: 8:52:34
         checkin-compilation: 19:27:21
         checkin-test: 1 day, 0:37:55
         infra: 19:47:13
         other: 2:53:21
         total: 3 days, 3:38:24

I created a graph of the data showing Mozilla Inbound since we started using it in August 2012 till now.

Closures by reason on Mozilla Inbound

The first part of the graph there wasn't any data for specific things but the sheriffs changed that in the middle of last year. I am hoping that we can get information like this, and other interesting back out info into a "Tree Health Report" in Treeherder (The TBPL replacement the Automation and Tools Team is developing).

    Area: blog

Management is hard

Thu 13 Mar 2014

I have been a manager within the A*Team for 6 months now and I wanted to share what I have learnt in that time. The main thing that I have learnt is being a manager is hard work.

Why has it been hard?

Well, being a manager is requires a certain amount of personal skills. So being able to speak to people and check they are doing the tasks they are supposed to is trivial. You can be a "good" manager on this but being a great manager is knowing how to fix problems when members of your team aren't doing the things that you expect.

It's all about the people

As an engineer you learn how to break down hardware and software problems and solve them. Unfortunately breaking down problems with people in your team are nowhere near the same. Engineering skills can be taught, even if the person struggles at first, but people skills can't be taught in the same way.

Working at Mozilla I get to work with a very diverse group of people literally from all different parts of the world. This means that when speaking to people, what you say and how you say things can be taken in the wrong way. It can be the simplest of things that can go wrong.

Careers

Being a manager means that you are there to help shape peoples careers. Do a great job of it and the people that you manage will go far, do a bad job of it and you will stifle their careers and possibly make them leave the company. The team that I am in is highly skilled and highly sought after in the tech world so losing them isn't really an option.

Feedback

Part of growing people's careers is asking for feedback and then acting upon that feedback. At Mozilla we have a process of setting goals and then measuring the impact of those goals on others. The others part is quite broad, from team mates to fellow paid contributors to unpaid contributors and the wider community. As a manager I need to give feedback on how I think they are doing. Personally I reach out to people who might be interacting with people I manage and get their thoughts.

But I don't stop at asking for feedback for the people I manage, I also ask for feedback about how I am doing from the people I manage. If you are a manager and never done this I highly recommend doing it. What you can learn about yourself in a 360 review is quite eye opening. You need to set ground rules like the feedback is private and confidential AND MUST NOT influence how you interact with that person in the future. Criticism is hard to receive but a manager is expected to be the adult in the room and if you don't act that way you head to the realm of a bad manager, a place you don't want to be.

So... do I still want to be a manager?

Definitely! Being a manager is hard work, let's not kid ourselves but seeing your team succeed and the joy on people's faces when they succeed is amazing.

    Area: blog

Don't write "Five Hidden Costs of X" but when you do I will reply

Wed 12 Mar 2014

Recently I was shown that Telerik did a "Five Hidden Costs of Selenium". I knew straight away from the title that this was purely a marketing document targeting teams with little to no automation skills to do automation. For what it is worth, if you want to do automation you should really hire the right engineers for the job.

My offence with the article is not that its wrong, there are a few items I disagree with which are documented below, but with it trying to sell snake oil or silver bullets. So let's even up the argument a bit. Note I am only comparing the WebDriver parts since if it were purely Selenium IDE vs Teleriks tool then I think it would be fair comments.

No Build/Scheduling Server

Telerik say we don't have those items and we don't. We don't want to be working on them since there are some awesome open source products with many years worth of engineering effort in them that you can use. These are free and allow a great amount freedom of customisation. They also work really well if you have hybrid systems as part of your test. Have you seen that ThoughtWorks has open sourced Go which is a great product from people who have been doing continuous integration for nearly, if not more, than a decade. Don't want to host it yourself, because managing servers is a hidden cost in all worlds, then look at the huge amount of Continuous Integration as a Service companies out there.

Execution Agents/Parallel Running

It says this is a 3rd party plugin which is not true. The Selenium server has a remote server system built in and if the correct arguments are passed in it can become a grid with a central hub managing which nodes are being used. This is called Selenium Grid.

The one thing, from the documentation, is that you have to host all these nodes yourself. Does it create hermetic environments to run against when scheduling? Hermetic environments is something that each core developer would want and if we can't give it then its not worth releasing. There are Infrastructure as a Service companies that WebDriver tests can be hooked up to so you don't need to maintain all the infrastructure yourself. The infrastructure costs can be quite expensive if you're a smallish team, using a service here can help. Unfortunately Telerik don't offer execute nodes as a service so you'll have to manage them yourself.

Also, its fine that nUnit doesn't support parallel execution, get your scheduling server to run a task for each browser and these tasks could be run in parallel.

Reporting

This is best done by the continuous integration server as part of the build and test. These take the output from the tasks they have told them to execute and then report. Having this as a selling point in marketing documentation feels like it is just targeting the untrained staff.

Multi-Browser Support

This is where you would think we would be even but Telerik is stuck to desktop browsers. WebDriver, due to its flexible transport mechanism (it's like we thought about it or something, means anyone can implement the server side and control a browser and all languages just work. We recently saw that with Blackberry creating an implementation for their devices. We have Selendroid for Android and iOS-Driver for iOS. Mobile is the future so only supporting the major desktop browsers is going to limit your future greatly.

Jim also mentioned that you would need to build a factory and teach it to get things running against multiple projects. You do have to but here is a link to the Selenium projects' way of handling it. We need to run our tests in multiple environments and we do it pretty well. This is just a one time sunk cost.

Maintenance of tests

I might be pulling a Telerik here but having tests look like the following
lolwat? What is SENDSELECTEDDiv even?.

Being able to code as an automation engineer is crucial. Being able to write good tests is useful too! I am biased but Mozilla has some really good examples that show good maintainable tests. Tests are trivial to write and update since they invested in a good pattern for tests. Record and playback tools have never had the ability to write maintainable tests and for them to be using meaningful API names. Also, it hampers (as in no one will take you seriously) your career calling yourself an automation engineer and only using record and playback tools.

Now beware of the snake oil that is being offered by vendors and for all that is holy... if you want to do automation don't do record and replay. I, and my peers, will not even let you past a phone screen if you don't show enough knowledge about coding and automation.

Also if Telerik was thinking straight they would be wrapping webdriver and then they would get everything that is happening in the webdriver world. Knowing your tests will always work in the browser no matter what platform (including mobile) is a huge selling point. And its standards based, feels like a no-brainer but i am obviously biased.

    Area: blog

WebDriver Face To Face - February 2014

Fri 28 Feb 2014

This week saw the latest WebDriver F2F to work on the specification. We held the meeting at the Mozilla San Francisco office.

The agenda for the meeting was placed, as usual, on the W3 Wiki. We had quite a lot to discuss and, as always, was a very productive meeting.

The meeting notes are available for Tuesday and Wednesday. The most notable items are;

  • Changing switchToFrame to only accept a WebElement or Index
  • Adding switchToParentFrame to the API
  • Potential changes to the way we do clicks on elements larger
  • Numerous bugs in the spec
  • Removing findElement(By.className and By.Id)
  • Landed the User Interactions spec and created an endpoint for batched actions

The other amazing things that happened was we had Blackberry join the working group, especially after their announcement saying they have created an implementation.

And... how can I forget about this...

The specification is getting a lot of attention from the people that we need and want which makes me really excited!

    Area: blog

Updated BrowserMob Proxy Python Documentation

Mon 03 Feb 2014

After much neglect I have finally put some effort into writing some documentation for the BrowserMob Proxy Python bindings however I am sure that they can be a lot better!

To do this I am going to need your help! If there are specific pieces of information, or examples, that you would like in the documentation please either raise an issue or even better create a pull request with the information, or example, that you would like to see in the documentation.

    Area: blog