Tue 26 Feb 2013
The other day I got asked if I knew of a tool that would notice changes to IDs of elements and update Selenium tests accordingly because there was an incurred maintenance cost in updating these all the time because the test will fail.
The tl;dr; is there isn't a tool and I don't think there should ever be one
Why?
HTML documents are not complex things, far from it, so when we change them we should think about how this is going to impact everything that hangs off a page.
If a designer or front end engineer changes the ID of elements then they go around updating the JavaScript references to that element and then update any CSS that works with those elements!
The non-obvious thing that they tend to forget is the test automation like, Selenium, that uses that element too!
Why? Well 9/10 times its because they don't have any ownership in the process! They don't submit tests nor do they run tests so they don't see the problem.
Now you are probably going to say "We use XPath/ CSS/ Names because (insert excuse here)!"
The point is still the same, your testing process and your development process are silo'ed and you're feeling the affects of this! Just remember everyone involved has a stake in the final product and QA aren't gate keepers for the product.
Area: blog
Mon 18 Feb 2013
Recently there was a bit of a rant on the Selenium users mailing list about how there were a few bugs irritating the person and, because they are only testers with not enough development experience, didnt feel that they could help with fixing these issues.
Note: this person was not rude and was not trolling so was happy to reach out.
One thing that I want people to know is that you don't have to be a brilliant developer to work on Open Source projects. You don't even have to have any experience as a software developer. You are now wondering how you can help?
By submitting a patch, even with mistakes an inexperienced would do, you are solving the problem. Once you have submitted it one thing is going to happen. Someone is going to look at your patch, eventually and reply. The reply will be we landed it or your patch needs some work. The latter is a good thing, even if you see it as a negative, since you will be learning.
Now, and is the most important thing to remember, is that the developer looking at your patch could very well learn something from you! Committers on Open Source projects tend to have come up through the ranks, unless they created the project, and have learnt what code does what where. They have learnt from other developers and want to share the knowledge back.
Going through all of these processes is a good thing. As people in software we have to be constantly learning or we fall behind. Once you are behind it can be hard to get employment in some companies you would love to work.
Next time you feel negative about contributing to Open Source, try change the thought to how you can learn something new and make a difference to your favourite projects. Selenium has getting involved bugs and Mozilla has its good first bugs that people can solve.
Don't be afraid to contribute to numerous projects because you will likely learn from a number of different people. This way you will get many years worth of meaningful experience instead of just many years of doing the same thing over and over.
Area: blog
Wed 06 Feb 2013
Could CSS 3, while is a great thing for the Internet and for web developers, be making websites that are extremely hard to automate?
As most of you know, in Selenium WebDriver we try an emulate what elements that a user can interact with. This means that we do a lot of DOM walking and gathering important little bits about the CSS on each of the elements to make sure that they are visible. You can see the details of what we do in the Determining Visibility section of the W3C Browser Automation Specification. Unfortunately every so often we get a situation where people start raising bugs about their Selenium WebDriver script with respect to allowing or allowing interaction with elements.
Recently we have started getting issues with how we handle CSS transforms. The first bug report came from the Web QA Team creating tests for Firefox OS. They were automating the Music app and noticed that tests were saying that an element was visible when it wasn't. I recreate the issue and make a minimised test case. You can see the element is miles away from the rest of the DOM below in the minimised test case below.
Marionette, the future of FirefoxDriver and what we use to automate Firefox OS, thinks that is visible. So down the slippery slope we go, add in support CSS Transforms, because that is the root cause for this bug. Now if you are wondering how complex transforms can get have a look at this MDN article.
It's just Algebraic transforms, no big deal right? But it can be when you start thinking in terms of cascading. Parent, or even great great grandparent element of the element we need to interact with could have the style applied. Now we need to walk the DOM from the element we want all the way back to body.
NOTE: Selenium has to walk the DOM a few times during various parts of the visibility checks which on large pages can be very expensive (therefore slow!). I am looking at you in particular large table element! We do the same when getting the text of elements
So while we are doing all of this work we used to assume that anything that mutates the element, like moving it across the page or changing its visibility, is going to happen in JavaScript and since the visibility code for Selenium is written in JavaScript, we will be blocking the JavaScript thread. CSS 3 has created things that have meant that we don't need JavaScript anymore to do animations
That means working with Selenium on something that is pure CSS animations might not work as you expect. So next time you decide to have a play with CSS 3 and it's awesomeness, have a play with Selenium at the same time and see how we fare. I suspect there are a lot of cases we don't cater for and we should!
Area: blog
Wed 16 Jan 2013
As many of you know I have recently released the 2nd edition of my Selenium book. When I wrote my first book I looked for a way to do version control of my book. Being a developer and test type, I used git to do all of my version control but I found that I would sometimes forget to commit and push my changes. This broke my own rule of keeping commits small.
The other problem is that if the text editor crashes, or even the OS crashes or I get a hard drive crash, there is a chance that I might only be able to go back to my last commit and push. Since I already said I am rubbish at doing the commit and push thing with documents git was definitely out of question.
I went to look for a different solution. I remembered that Dropbox keeps a history of files when syncing. So I went and tested the theory with a directory within my Dropbox folder. After a few times of playing with a file and syncing and reverting, I thought I would give it a go with my first chapter. I instantly fell in love with this approach. My publisher wanted me to use Microsoft Word to write all of chapters I instantly reverted to my old university habits of ctrl+s every few minutes. And every time I saved my chapter Dropbox would instantly synchronise the file and update the history of the file.
This meant that just by saving my file I was getting the small commit chunks I like without the worry of remembering to push my changes. I have lost the ability to do commit messages but in this scenario I honestly don't care. All I worry about is making sure I have the most up to date files I can if something were to go wrong.
I highly recommend people using Dropbox in this way.
Area: blog
Thu 03 Jan 2013
Building Software can be a very expensive thing to do! Especially if you are building a web application that needs to be deployed.
An EC2 micro instance or Linode 512 machine will cost ~USD20 a month. This then needs to have all monitoring apps installed and maintained which costs more money. This might not be a lot to some of us but growing up in South Africa, that always felt like a lot of money. When you are building a new app for Firefox OS, paying for a server to serve your app might not be in your costs for building, testing and deploying your app.
What if you could limit your development and deployment costs to £0? Note your app needs to be open source to make the cost £0.
Source Control
The current "go-to" source control at the moment is GitHub. I quite like it and for our free CI and deployment, we are going to have to have to use GitHub.
Project template
The FirefoxOS Marketplace team have created some template projects called mortar that give you all the necessary pieces to build an app. There are a number of different templates, from normal to games, for all your different needs.
Continuous Integration
Travis-CI is my favourite CI as a service at the moment. They are running the company as an open source company so I have a lot of love for these guys but I am digressing.
If you are using mortar then your .travis.yml will look like the following
language: node_js
node_js:
- 0.8
And you will also need to update your package.json file to specify how to run tests. I wrote my tests with python so it's the following
"scripts": {
"test": "py.test -n 3"
}
Now, if you want to maximise the use of your app you will need to make sure it works on as many browsers as possible. My friends at Saucelabs have recently created an Open Sauce account giving you free usage and it works well with Travis CI. To get it working you just add the following to your .travis.yml
env:
global:
- secure: "sauce user name"
- secure: "sauce labs key"
matrix:
- SAUCE_PLATFORM='Windows 2003' SAUCE_BROWSER_VERSION=18
You can see a working example here.
And the have something like the following in your test setup. It will also set the browser size to the default FirefoxOS view port
def setUp(self):
try:
os.environ['SAUCE_USERNAME']
desired_capabilities = webdriver.DesiredCapabilities.FIREFOX
desired_capabilities['version'] = os.environ['SAUCE_BROWSER_VERSION']
desired_capabilities['platform'] = os.environ['SAUCE_PLATFORM']
desired_capabilities['name'] = 'Testing Get In The Habit'
self.driver = webdriver.Remote(
desired_capabilities=desired_capabilities,
command_executor="http://%s:%s@ondemand.saucelabs.com:80/wd/hub" % \
(os.environ['SAUCE_USERNAME'],
os.environ['SAUCE_ACCESS_KEY'])
)
except Exception as e:
print e
self.driver = webdriver.Firefox()
self.driver.set_window_size(320, 480)
self.driver.get("http://localhost:8008")
Now that we have built and tested our app it's time to deploy!
Deployment
One of the things about our apps for Firefox OS is that it can be packaged on to the phone or can be served from a web server. GitHub pages make this a rather trivial exercise and also allows you to grow your user base without having to know how to maintain servers. You can also have a look at things like Launcher.io / AppFog / Heroku that have free tiers. If you need to have a backend for your app for synchronising of data then I highly recommend one of them.
Area: blog
Thu 08 Nov 2012
One of the biggest things that always shocks me is the way people approach problems. Software developers are essentially pattern matchers using experience to figure out how to solve problems that are placed in front of them.
This can unfortunately lead them to a problem where developers use their current knowledge to solve problems. Imagine that you are in QA and your only experience of automation is using Selenium. You are told that you need to test a REST API, or as I have seen in forums to test stored procedures in your database, you go for the tool that you know and feel comfortable with. The downside to this is that you could possibly pick the wrong tool!
If you are thinking that people don't use Selenium to test Databases/SOAP/REST think again and go have a read of the Selenium User Forum!
Selenium is a wonderful tool! I am not just saying that because I have written two books or because part of my time is to try support the project, I say it because I love watching the tests run! Unfortunately it uses a browser to drive a web application. This means that you have the entire stack to worry about when testing. Actually you have more than your stack to worry about when testing. A Browser is a very complex bit of software. Both Google and Mozilla have shown that it can be its own operating system with Chrome OS and Firefox OS respectively.
That is a lot of lines of code that are run just to make sure that your tests do what they are expected. When you need a browser you use it and then make sure everything is working. The browser start up time annoys you but for the coverage you are getting it is acceptable.
But when you are testing something that isn't rendering there is a lot of wasted parts of the browser stack that are called and aren't really used.
So when running your tests thing about how much is being invoked and how much really needs to be invoked. Try not to be wasteful with the resources that are consumed. When little resources are consumed in a test then the tests will run run quicker making feed back loops take less time.
Area: blog
Mon 05 Nov 2012
Last week there was a very large gathering for the W3C in Lyon, France. The TPAC conference gets together all working groups and allows them to share their specifications with everyone else and demo implementations if there are some complete.
The Browser Automation Working group was there too to discuss moving the current specification further. We had 2 days of meetings and below is a break down of the topics we discussed.
On the first day we discussed the following
- Interoperability
(the spec is very loose on this, but interoperability is vital)
- Internationalised input
(do we need the IME methods from OSS WebDriver? Our
current solution is very desktop-centric)
- Possibly adding an ARIA locator
- Handling the shadow DOM
- Do we use strings or numbers for status codes
(there are arguments either way)
- Identifying areas of the WebDriver spec that better belong elsewhere
Discussion of open issues in the spec:
- Screenshots
- Non-HTML content (SVG, XML, XHTML, canvas, text/plain)
- Publishing a new working draft
- Start work on the conformance tests
The main items that came from the work this week are:
- Browser Vendors are to provide a shim that understands the HTTP JSON Wire Protocol. This allows browser vendors to use the best transport mechanism they want but gives implementors of the local end library to use the same approach for all browsers ensuring interoperability between browsers.
- We are going to be changing error codes to error strings. Using strings allows us to pass back a meaningful error without it needing to be translated from an integer to allow easier development of libraries for local ends.
- We are going to be adding the ability to get a screenshot of a specific element on the page.
- We have started work on the conformance tests for the specification. The first tests actually found a bug in the Selenium WebDriver implementation which is great.
We will be cleaning up a few sections and hopefully be pushing the next version of the specification in the next few weeks. If you have thoughts or you spot any errors please let us know!
Area: blog
Thu 25 Oct 2012
It has been two years to the day that I joined Mozilla. I have mentioned in the past that I came in having missed the contributor to employee route and, as I said then, feel like I have missed out.
So what have the last two years been like? If I was only allowed one word it would be "Rollercoaster". Now I know that you are probably thinking that I chose it for the cliche of ups and downs. I chose the word because it is full on, and at a tremendous speed, and with out a doubt, fun!
So what have I done in the last 2 years at Mozilla? I have worked in 3 different teams. I started out in the WebQA team helping standardise the way they write their Selenium tests. They have gone on to do much much more than I expected and I love watching what this team does. It is after all my first home in Mozilla.
The next team that I was in was Automation Services. This was a team that I co-lead with Henrik Skupin and we did some really good things. This team was then merged with the Automation and Tools team, the A-Team as they are known internally, which is my current team.
I have worked on everything from web applications to Firefox Mobile to Firefox itself. I am co-editing a W3C standard with Simon Stewart to make sure that web developers and software testers can increase the quality of the products that they deliver to the web. I have worked on Frameworks and production code and I have written a lot of tests.
All the while knowing that everything I do is making a difference to the way the web is being used and worked on. It is one of the main things that drives to me to want to work at Mozilla. One of the other main things that I have found interesting, and to be honest still shocks me, is the warmth that people exhibit when they find that you work for Mozilla.
So after two years at Mozilla I can honestly say that I am working with a large number of brilliant people, though I have always been lucky with my previous employers, who continue to teach me, guide me and to give me opportunities to show how brilliant our craft and industry is!
P.s. Want to help Mozilla? Have a look at What can I do for Mozilla?
Area: blog
Tue 09 Oct 2012
So it is official, the 2nd version of my book is officially out! It has been a lot of work but I am happy with the current result. Selenium 2 Testing tools is a beginners guide that walks the user from creating easy throw away tests with Selenium IDE to being able to use Selenium WebDriver on desktop Operating Systems to being able to run them on mobile devices.
It is a beginners guide so if you know how to code you should be able to get started and by the end of the book be pretty confident at using Selenium WebDriver on a number of different platforms and devices.
You can buy the book from Packt Publishing or from Amazon.com or from Amazon.co.uk.
I hope that you enjoy it!
Area: blog
Wed 25 Jul 2012
We all know that Selenium WebDriver is on track to becoming a W3C standard. First we saw this happen with Opera who have integrated this with their browser when they created OperaDriver. Google Chrome followed with the release of their ChromeDriver executable. This is a separate download to the browser and client bindings. So who is next? Mozilla.
Why do we need a new FirefoxDriver for Selenium?
Mozilla has been working on a project called Marionette which will become a future version of FirefoxDriver. Marionette was created as a way to automate Firefox OS, since Firefox OS is just a kernel then starts up Gecko, the core engine of Firefox. I appreciate that is overly simplified but for now it all that we need to know. Firefox OS did not have a mechanism that allows Firefox extensions, when the work started on Marionette, to be installed which means that the current way that FirefoxDriver works by installing an extension and updating preferences on start-up will not work.
I have been working on Mozilla Open Web Apps project as one of my many things. This is a project that allows you to install web applications natively on to your OS and then use them like native applications. This relies on AppCache and other offline tricks but when you start the installed version of a web application you will not have access to extensions either. So both Firefox OS and Open Web Apps both block the current approach that FirefoxDriver has.
So in step the Automation and Tools Team, affectionately known as the A-Team internally at Mozilla, and more specifically Malini Das and Jonathan Griffin. They started working on a way to get an automation framework and build system going. Malini and Jonathan started working from the Selenium JSON Wire Protocol to create Marionette so that it can fit in with our plans for making Selenium a W3C Standard. Malini and Jonathan have stabilised Marionette for the current need in the Firefox OS project. I have been working to try to stabilise the desktop version, purely for my own needs on the Open Web Applications project at the moment but we be trying get this as complete as possible in the next couple of quarters.
How does it work?
Marionette works by using the socket transport of the JavaScript remote debugger as the JavaScript Remote debugger that is coming to Firefox. This just opens a socket in the browser that we then connect to and pass through commands that we want the browser to do. Does that mean that the JSON Wire Protocol isn't followed exactly? Yes, we have created our own take on the JSON Wire Protocol mainly just to fit in with the transportation mechanism that we are using. Since this is just a socket one could open a telnet connection to the browser and just send the commands. It is an extremely lightweight approach when we come to mobile and therefore benefits desktop implementations.
Does it only work on web content?
No! Marionette can also work against the browser chrome. Marionette is ultimately going to be the central item when it comes to automation frameworks within Mozilla. All of which are homegrown since we have unique requirements. We need to make sure that what we can do at least most of their functionality via Marionette if not all of it.
Bonuses
With Marionette being maintained in-house by the people that know the browser best, and being part of Mozilla build process, if Marionette gets broken it will get backed out very quickly. That means FirefoxDriver is going to be part of the Mozilla build process!
So when can I play with it? It is available now as part of Firefox debug builds. Why only debug builds? It still has to go through our security review since just having an open socket can drive the browser is quite scary and an opportunistic hacker could just run a selenium script and steal details. It will probably be preferenced off to start but is still a potentially scary problem.
Area: blog
