Explaining how the real world works!     RSS Feed The Automated Tester on Twitter The Automated Tester on LinkedIn The AutomatedTester on github

TPAC 2013 - WebDriver Face To Face and more

Wed 20 Nov 2013

Last week I was at W3 TPAC for week of face to face meeting to discuss WebDriver and other W3C specifications that other working groups are working on.

Our initial agenda went up just before the meeting and we were lucky enough to get through all the items. If you would like to read the notes for the meeting (Monday and Tuesday.

Highlights from the meeting are

  • We are rescoping the specification to have the relevant things we need - we have been suffering from scope creep so we are removing things that aren't complete or won't be completed soon. We will be creating WebDriver "Level 2" specification which those features will go into.
  • We have roughly 36 weeks worth of work to get the spec and tests done. - I will be writing details on how anyone can help make this time line realistic so we hit all of our milestones!
  • We are going to be adding an API for when an element is intractable via mouse/touch, we haven't decided on a name yet but will be in the spec soon
  • Simon and I were speaking to other editors/working groups to see if we could offload definitions of things to them. It seemed to be positive, let's see where things head.

There are other actions from the meeting that need to be done but I think the items above cover the main points, at least for me, that came out of the meeting.

I found the rest of the week actually really useful from a networking perspective and from a learning perspective. I have a lot of changes that I need to put into the WebDriver spec and have been getting feed back about when i am doing it wrong which is great!

    Area: blog

Test The WebForward - Shenzhen, China

Tue 19 Nov 2013

Just over a week ago I was in Shenzhen, China helping with Test The Web Forward. A great initiative to get anyone and everyone to write tests for browser companies to use. The tests are conformance tests for W3C.

In the next couple of weeks I am going to document what is needed to help out and how you, yes you, can help! The WebDriver open source test suite needs to be checked against the WebDriver Specification and then ported over or bugs raised against the spec or implementations. There will be bugs which is a great thing!

P.S. We had 4 patches with ~10 tests from people who had never used WebDriver so I consider that a great success. We also noticed a number of pain points we need to fix before we get more people involved

    Area: blog

HackBMTH last weekend

Wed 17 Jul 2013

Last weekend there was another HackBMTH, this time hosted by Adido. This was my first time going to a hackathon in the Bournemouth area and it was a great. Jonathan Ginn and Adam Howard are the organisers and to be honest, they did a brilliant job of organising the event.

There were people from all over Bournemouth and as far away as Reading who came to work on their little projects or create new hacks. There were ops people helping people setup Apache and a few other things. There were people creating games, one of which was a tank game that took a picture from the webcam and turned that into the level. Finally a use for QR codes!

I worked on my GitHub+Travis Firefox Add-on. I worked to start pulling in the build history of Travis CI Projects to the project page on Github. Below is a screenshot of what it looks like at the moment. It's still rough around the edges.

You can install a pre-release from Github if you want play with it. Feel free to raise bugs about it on the Github page.

The only thing that I was disappointed with was that there was only 1 female. This I should point out I don't think is the fault of the organisers. When I worked in Southampton there weren't many females applying for positions so I know the area suffers from a lack of them. I know that there are some as students, having met them at B & W Meet, so I hope that more will come. After all, one of the main reasons there are not enough woman in tech is because there are not enough woman in tech, to paraphrase Sheryl Sandberg.

    Area: blog

W3C WebDriver Face To Face - June 2013

Thu 27 Jun 2013

A couple weeks ago was the latest face to face of the W3C Browser Automation (WebDriver) Working Group. We as a group don't meet anywhere near enough in my opinion but when we do the conversations we have are quite amazing.

Below is the Agenda that we used for discussions. If you want to read the minutes you can read them here and here.

If you have any questions please leave a comment!

    Area: blog

The tale of Selenium bug 141

Wed 26 Jun 2013

The selenium project has for a while now wanted to create a library that allows user emulation in the browser. The project has done a reasonable job at this so far with respect to this. We check if items are in the DOM, we check the visibility of items and we normalise text from the browser, amongst our other amazing talents!

Our default position is we enforce the idea of emulation. "How would a user do this action?" is the main question we ask ourselves! So when a bug that doesn't meet that we tend not to implement it. One of these times is bug 141 - WebDriver lacks HTTP response header and status code methods. This is one of the most contentious bugs in the Selenium repository. People see Selenium as more than browser automation library, they see it as a web testing framework. Yes, Selenium RC was like this but then again Selenium RC also had ~140 methods hung off one object. We made a mistake in the past, let's try not to repeat it! Anyway, I digress, in a web testing framework world there may be times where there are use cases where you need to find out data from the response header and status codes.

So is this a feature we want to add to WebDriver? No. "But David, we do!?!?!?" I hear you cry! The use case that comes up regularly is I want to know if a server is returning a 404 or 500 when I access a page. Why is there a need to start a browser to find this out? When I asked this question on twitter, my friend and fellow committer Kevin Menard said that it could be useful to to see the page response on failure and some pages can only be accessed with a real browser. The use cases that Kevin had is you want to check that the endpoint that you originally hit might might change based on the user agent passed in or there might be a site that follows the single page idea and breaks push state and therefore most of the web semantics. I admit Kevin is right with those use cases. However its only looking at a small part of the issue and there are a lot of corner cases. The major one for me is watching XHRs. What if one of your XHR returns a 500 and you get an error? Should WebDriver show you that information as well? I am sure you can see how quickly this can escalate. If you start worrying about all requests, and you should if you are worried about the initial request to a page, then you want the browser to capture all of this information and return it in a standard format. There just happens to be a format called Http Archive (HAR).

So does the project take the task of creating a mechanism for extracting a HAR from the browser? That could be near impossible on browsers that don't readily share network usage information. So we go from WebDriver being a browser automation framework for user emulation to browser automation framework for user emulation and networking information gathering. That might be a little obtuse but its meant to be.

While Selenium's main use case is for test automation, is not solely designed for that. We want to be the best at browser automation. Proxies are really good at managing network information and reporting on it in standard formats. Examples are fiddler and browsermob-proxy. And you can even hook them into Selenium WebDriver! Like the following:

from browsermobproxy import Server
server = Server("path/to/browsermob-proxy")
server.start()
proxy = server.create_proxy()

from selenium import webdriver
profile  = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)


proxy.new_har("google")
driver.get("http://www.google.co.uk")
proxy.har # returns a HAR JSON blob

server.stop()
driver.quit()

Yes it might be a little verbose to use them but if you really want to see that type of information then you should do it properly otherwise the real errors, that you originally wanted, will go missing.

Putting in half a solution into Selenium is just going to create scope creep and a large number of bugs because its not doing the "right" thing. The "right" thing is far too subjective for it to be done correctly.

    Area: blog

Selenium Conference 2013

Thu 20 Jun 2013

Last week was Selenium Conference in Boston, MA. This was the 3rd conference this year and it did not disappoint.This was the largest Selenium Conference ever! There were around 450 people there this year and we managed to get a large number of good high quality talks.

The main take aways from the conference, other than meeting some really brilliant people, was that:

  • Selenium 3 will be coming out around December.
  • Marionette, also known as FirefoxDriver 2 to some is available in the Nightly branch of Firefox. If you want background please read this post
  • A good automated test is indistiguishable from monitoring.
  • Mobile is the future

I managed to get myself doing 2 talks at the conference but they seemed to be well received. Below are the links to the slides and videos.

Thanks to the organisers who did a great job and worked many hours to make this years conference happen. I am already looking forward to being at next years conference!

    Area: blog

The value of locators and why everyone "owns" them

Tue 26 Feb 2013

The other day I got asked if I knew of a tool that would notice changes to IDs of elements and update Selenium tests accordingly because there was an incurred maintenance cost in updating these all the time because the test will fail.

The tl;dr; is there isn't a tool and I don't think there should ever be one

Why?

HTML documents are not complex things, far from it, so when we change them we should think about how this is going to impact everything that hangs off a page.

If a designer or front end engineer changes the ID of elements then they go around updating the JavaScript references to that element and then update any CSS that works with those elements!

The non-obvious thing that they tend to forget is the test automation like, Selenium, that uses that element too!

Why? Well 9/10 times its because they don't have any ownership in the process! They don't submit tests nor do they run tests so they don't see the problem.

Now you are probably going to say "We use XPath/ CSS/ Names because (insert excuse here)!"

The point is still the same, your testing process and your development process are silo'ed and you're feeling the affects of this! Just remember everyone involved has a stake in the final product and QA aren't gate keepers for the product.

    Area: blog

Why working on Open Source software makes you a better developer

Mon 18 Feb 2013

Recently there was a bit of a rant on the Selenium users mailing list about how there were a few bugs irritating the person and, because they are only testers with not enough development experience, didnt feel that they could help with fixing these issues.

Note: this person was not rude and was not trolling so was happy to reach out.

One thing that I want people to know is that you don't have to be a brilliant developer to work on Open Source projects. You don't even have to have any experience as a software developer. You are now wondering how you can help?

By submitting a patch, even with mistakes an inexperienced would do, you are solving the problem. Once you have submitted it one thing is going to happen. Someone is going to look at your patch, eventually and reply. The reply will be we landed it or your patch needs some work. The latter is a good thing, even if you see it as a negative, since you will be learning.

Now, and is the most important thing to remember, is that the developer looking at your patch could very well learn something from you! Committers on Open Source projects tend to have come up through the ranks, unless they created the project, and have learnt what code does what where. They have learnt from other developers and want to share the knowledge back.

Going through all of these processes is a good thing. As people in software we have to be constantly learning or we fall behind. Once you are behind it can be hard to get employment in some companies you would love to work.

Next time you feel negative about contributing to Open Source, try change the thought to how you can learn something new and make a difference to your favourite projects. Selenium has getting involved bugs and Mozilla has its good first bugs that people can solve.

Don't be afraid to contribute to numerous projects because you will likely learn from a number of different people. This way you will get many years worth of meaningful experience instead of just many years of doing the same thing over and over.

    Area: blog

Could CSS3 be making sites that are not testable?

Wed 06 Feb 2013

Could CSS 3, while is a great thing for the Internet and for web developers, be making websites that are extremely hard to automate?

As most of you know, in Selenium WebDriver we try an emulate what elements that a user can interact with. This means that we do a lot of DOM walking and gathering important little bits about the CSS on each of the elements to make sure that they are visible. You can see the details of what we do in the Determining Visibility section of the W3C Browser Automation Specification. Unfortunately every so often we get a situation where people start raising bugs about their Selenium WebDriver script with respect to allowing or allowing interaction with elements.

Recently we have started getting issues with how we handle CSS transforms. The first bug report came from the Web QA Team creating tests for Firefox OS. They were automating the Music app and noticed that tests were saying that an element was visible when it wasn't. I recreate the issue and make a minimised test case. You can see the element is miles away from the rest of the DOM below in the minimised test case below.

Element far from rest of DOM

Marionette, the future of FirefoxDriver and what we use to automate Firefox OS, thinks that is visible. So down the slippery slope we go, add in support CSS Transforms, because that is the root cause for this bug. Now if you are wondering how complex transforms can get have a look at this MDN article. It's just Algebraic transforms, no big deal right? But it can be when you start thinking in terms of cascading. Parent, or even great great grandparent element of the element we need to interact with could have the style applied. Now we need to walk the DOM from the element we want all the way back to body.

NOTE: Selenium has to walk the DOM a few times during various parts of the visibility checks which on large pages can be very expensive (therefore slow!). I am looking at you in particular large table element! We do the same when getting the text of elements

So while we are doing all of this work we used to assume that anything that mutates the element, like moving it across the page or changing its visibility, is going to happen in JavaScript and since the visibility code for Selenium is written in JavaScript, we will be blocking the JavaScript thread. CSS 3 has created things that have meant that we don't need JavaScript anymore to do animations

That means working with Selenium on something that is pure CSS animations might not work as you expect. So next time you decide to have a play with CSS 3 and it's awesomeness, have a play with Selenium at the same time and see how we fare. I suspect there are a lot of cases we don't cater for and we should!

    Area: blog

Using Dropbox as a source repository for book writing

Wed 16 Jan 2013

As many of you know I have recently released the 2nd edition of my Selenium book. When I wrote my first book I looked for a way to do version control of my book. Being a developer and test type, I used git to do all of my version control but I found that I would sometimes forget to commit and push my changes. This broke my own rule of keeping commits small.

The other problem is that if the text editor crashes, or even the OS crashes or I get a hard drive crash, there is a chance that I might only be able to go back to my last commit and push. Since I already said I am rubbish at doing the commit and push thing with documents git was definitely out of question.

I went to look for a different solution. I remembered that Dropbox keeps a history of files when syncing. So I went and tested the theory with a directory within my Dropbox folder. After a few times of playing with a file and syncing and reverting, I thought I would give it a go with my first chapter. I instantly fell in love with this approach. My publisher wanted me to use Microsoft Word to write all of chapters I instantly reverted to my old university habits of ctrl+s every few minutes. And every time I saved my chapter Dropbox would instantly synchronise the file and update the history of the file.

This meant that just by saving my file I was getting the small commit chunks I like without the worry of remembering to push my changes. I have lost the ability to do commit messages but in this scenario I honestly don't care. All I worry about is making sure I have the most up to date files I can if something were to go wrong.

I highly recommend people using Dropbox in this way.

    Area: blog

Build, test and deploy Firefox OS apps for 0

Thu 03 Jan 2013

Building Software can be a very expensive thing to do! Especially if you are building a web application that needs to be deployed.

An EC2 micro instance or Linode 512 machine will cost ~USD20 a month. This then needs to have all monitoring apps installed and maintained which costs more money. This might not be a lot to some of us but growing up in South Africa, that always felt like a lot of money. When you are building a new app for Firefox OS, paying for a server to serve your app might not be in your costs for building, testing and deploying your app.

What if you could limit your development and deployment costs to £0? Note your app needs to be open source to make the cost £0.

Source Control

The current "go-to" source control at the moment is GitHub. I quite like it and for our free CI and deployment, we are going to have to have to use GitHub.

Project template

The FirefoxOS Marketplace team have created some template projects called mortar that give you all the necessary pieces to build an app. There are a number of different templates, from normal to games, for all your different needs.

Continuous Integration

Travis-CI is my favourite CI as a service at the moment. They are running the company as an open source company so I have a lot of love for these guys but I am digressing.

If you are using mortar then your .travis.yml will look like the following

language: node_js
node_js:
  - 0.8

And you will also need to update your package.json file to specify how to run tests. I wrote my tests with python so it's the following

  "scripts": {
    "test": "py.test -n 3"
  }

Now, if you want to maximise the use of your app you will need to make sure it works on as many browsers as possible. My friends at Saucelabs have recently created an Open Sauce account giving you free usage and it works well with Travis CI. To get it working you just add the following to your .travis.yml

env:
  global:
    - secure: "sauce user name"
    - secure: "sauce labs key"
  matrix:
      - SAUCE_PLATFORM='Windows 2003' SAUCE_BROWSER_VERSION=18

You can see a working example here.

And the have something like the following in your test setup. It will also set the browser size to the default FirefoxOS view port

    def setUp(self):
        try:
            os.environ['SAUCE_USERNAME']
            desired_capabilities = webdriver.DesiredCapabilities.FIREFOX
            desired_capabilities['version'] = os.environ['SAUCE_BROWSER_VERSION']
            desired_capabilities['platform'] = os.environ['SAUCE_PLATFORM']
            desired_capabilities['name'] = 'Testing Get In The Habit'

            self.driver = webdriver.Remote(
                desired_capabilities=desired_capabilities,
                command_executor="http://%s:%s@ondemand.saucelabs.com:80/wd/hub" % \
                              (os.environ['SAUCE_USERNAME'],
                              os.environ['SAUCE_ACCESS_KEY'])
            )
        except Exception as e:
            print e
            self.driver = webdriver.Firefox()
        self.driver.set_window_size(320, 480)
        self.driver.get("http://localhost:8008")

Now that we have built and tested our app it's time to deploy!

Deployment

One of the things about our apps for Firefox OS is that it can be packaged on to the phone or can be served from a web server. GitHub pages make this a rather trivial exercise and also allows you to grow your user base without having to know how to maintain servers. You can also have a look at things like Launcher.io / AppFog / Heroku that have free tiers. If you need to have a backend for your app for synchronising of data then I highly recommend one of them.

    Area: blog