Thu 26 Jan 2012
Back in 2008, David Henderson and I wanted to try automate collecting client side performance data for the web application that we were working on at the time. We were getting a large number of complaints from users about load time and we need to try solve this. The way that we decided on doing this was to use Selenium, which was already running our tests, and hack the Selenium server to give us the information we needed. What we came up with was showcased at GTAC 2009. It was really good for the time and we were happy.
About a year later just as the new movement for HTTP Archive or HAR as its commonly known was taking off I found that Jan "Honza" Odvarko had created a Firebug add on to export the network tab to HAR. This, when used with Selenium, could mean that we could get the same data that the browser wanted with little to no effort. Run your tests as you were with the new WebDriver API and get it to programmatically install the necessary addons as well as set all the preferences needed. I wrote this for Python and for .NET. This works really well but now limits this type of data to only being collected within Firefox. Some modern web applications can send back totally different javascript and sprites dependant on which browser hits.
What if you could run your Selenium tests and collect the same info as the if you were using the Firebug Net tab but using any browser. Browsermob Proxy is a good way to collect this information and it has a programmatic interface that allows us to set it before our tests start. It can also return a HTTP Archive of the traffic that it is routing. I have released the Python Library for BrowserMob Proxy that can be injected into tests quite easily. I have put an example below.
from browsermobproxy import Server
server = Server("path/to/browsermob-proxy")
server.start()
proxy = server.create_proxy
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)
proxy.new_har("google")
driver.get("http://www.google.co.uk")
proxy.har # returns a HAR JSON blob
proxy.stop()
driver.quit()
If we now feed the data from proxy.har into HAR Viewer we can get a nice waterfall of the traffic as below! Jari Bakken has a useful commandline version called Har.
I recently showed this off at London Web Performance. Unfortunately it wasn't recorded but you can view the slides here
Area: blog
Thu 12 Jan 2012
WebDriver, the browser automation framework we all love, is on its way to becoming a browser standard. WebDriver as an OSS project is 5 years old and started being merged into the Selenium project about 3 years ago. From there we have seen WebDriver grow. It has had a lot updates in that time. The team is currently averaging ~100 commits a week.
In the short time, from GTAC 2009 when they were created, the Ruby bindings created by Jari Bakken have had over a million downloads! The Python bindings have around 3000 downloads a week and growing. Jim Evans work on the .NET bindings is great, I don't have figures because of Nuget. Same with the Java bindings. Maven makes it difficult to know how many downloads.
We have also seen that Selenium Jobs have overtook the commercial counterparts. All of these things point to us needing a standard to make sure if we can do the same thing on multiple devices and multiple OSes.
Yesterday we held a W3 meeting in the Google UK office. It was great to be able to get in a room and discuss what needed. A plan was hatched on what needs to be done next, and it is going to be a lot of work.You can see the minutes of the meeting here to see what we discussed.
Mozilla, Opera and Google are happy with the direction we are going and long may it last! It will be great to get Microsoft and Apple involved soon.
Remember that all of this work is happening out in the open and you can watch the Hg repository for the editors draft updates.
Area: blog
Thu 08 Dec 2011
How did you get involved with your favourite Open Source project? I started helping on the Selenium project in 2006 by helping out in the forums and then over time submitted patches and eventually got my commit bit by helping Jim Evans . I have always been a huge proponent Selenium even when it made up a small amount of my job. However I never looked to help Mozilla for some strange reason even though my personal views and the company mission were ideally matched.
Lately there have been a number of different blog posts from Mozillians about how they got involved with Mozilla. The posts describe how these people went from being volunteers, or using forks of Mozilla code, to being employees. A good example is Clint Talbert's post. Clint wrote his post in response to David Boswell's post. These stories make me feel like I have missed a step in getting into Mozilla. And to be honest it does make me a little sad.
One of the hardest things to do, and most OSS committers on any project will agree, is creating a community around your project. Mozilla has over the years done an amazing job at getting contributors. One of the main pulls is the Mozilla Misson! The simple message is "Mozilla's mission is to promote openness, innovation and opportunity on the web.". This is something that I didn't really know before joining Mozilla. I started using Mozilla Suite when it came out many many years ago. I was in University at the time and it made my life easier. I, like most students, had an amazing amount of free time and spent it playing with new technology and learning to program better but not contribute back to an OSS project.. In other words I could have given back.
I have been working on other OSS projects as and when I came across them but didn't see the OSS project that was taking me to the ones I was working on. Next time that you feel like that you want to help out on an OSS project but don't know which one have a look at have a look at Mozilla. If you want to see about helping my team, have a look at our team site and come have a chat on irc.mozilla.org#automation.
Area: blog
Wed 30 Nov 2011
I am one of the few privileged people in the world to only work on open source software for a living. All the work I get paid for as well as all my private projects is open source. It was one of the main things that drew me to work at Mozilla. Creating software that I can give away and when there are bugs I can fix, or even better receive a patch, and I can learn from my mistakes all in the open. I can show other people my approach to problems and perhaps they can learn from it too! I find this really exciting for me.
OSS is what makes the software world tick. Let's face it, a large number of libraries, frameworks or even our key datastores that we use in our day to day jobs are open source! There is also a chance that you use Linux as your main OS. A lot of these projects are run entirely by volunteers who freely give up there time to make something that they think is useful to the world!
Unfortunately there does appear to be a increasing trend for users of the OSS projects to feel they are allowed to treat developers as if they were paying for the code. Perhaps I am more aware of this trend since I am part of OSS landscape.
I am constantly seeing messages that say "how can I use your product that is full of bugs", "I am embarrassed to see this happening", "Urgently fix my problem...".
To be honest, that is the extent of the bug or query to the mailing list and looking past the trolling comments I couldn't help if I wanted to.
We have seen examples of this quite publicly with posts about Node.Js being cancerous and the article about Mozilla Places database crashing one users Firefox instance. But if you were to go through bug reports of your favourite OSS tool you will see these comments. They are not useful, helpful and can be down right hurtful.
Speaking to Simon Stewart he said sign of a OSS product making it big is the amount of comments that come in that are not useful and/or trolling increases. While I can understand that I don't understand the need to make comments like that.
We can see this a similar idea from Mikeal Rogers in one of his tweets.
Both my Selenium colleagues and my Mozilla team are made up of some of the cleverest and friendliest people who bend over backwards for the community at the drop of a hat.
So next time you make a comment on OSS think about how much you would feel if you gave up your free time to be treated this way by users. Be constructive in your criticism and make sure to make sure to give steps to replicate bugs even down to the smallest detail!
Area: blog
Thu 03 Nov 2011
GTAC is one of my favourite conferences that I like to go to. It has the right mix of people who attend. The one thing they have in common is that they want to learn and don't want any sales pitches.
This year did not disappoint. I have put my thoughts on each of the talks below that really stuck out.
Testing is Dead - Alberto Savio
This was an interesting talk about how we need to stop thinking being gatekeepers of products. We need to rethink our approach to creating products as a whole and then go from there. He started talking about prototyping and getting feedback. We should constantly dogfooding our application. Dogfooding, a term coined by Microsoft, is where you get people from with in the organisation using the application as soon as humanly possible.
Those of you who follow Eric Ries and the whole Startup movement will be well aware this idea. Eric pushes the concept of a minimum viable product so that you, as an entrepreneur or intrapreneur, can get people using it and feeding back ideas and bugs. This approach won't work with all applications but it solves the 80% problem.
Lightning Talks
The talk that stood out the most was a project called scriptcoverage. It is a Chrome extension that when coupled with testing tools could tell how much of your JavaScript is being exercised. This is a project that hopefully I will have time to port to a Firefox extension.
WebConsistency - Kevin Menard
Most people will know Kevin for being one of the key people on Selenium Grid. Kevin talked about how we need to stop worrying about pixel perfection between multiple browsers and that we can automate some checks. Since we access the DOM we can work out what CSS is being applied where and how this influences everything.
This is the basis for Kevin's start up, and MogoTest, so suggest signing up and giving it a try. In Testing Tools at Google talk the Googlers mentioned it was a much better product than they had created internally. Great comment from a company that suffers from "Not invented here"
How hackers see bugs - Hugh Thompson
Hugh did a really great talk on Security. Hugh talked about how numbers can really skew what is really happening in the real world. He was also talking about how social engineering is becoming the tool of choice. Years ago we had emails from Nigeria were sending fairly standard emails with no personalised parts and a number spelling mistakes. That has changed to perfect English with personalisation.
This was the most entertaining talk of the whole conference. When the videos are out I will be watching this again.
WebDriver - Simon Stewart
Simon did good talk on how to scale out your tests. Simon and a number of other Googlers have built out a Selenium farm so that they can run tests on many different browsers and OSes. While Simon was saying that a number of tools that Google use were built in house there are many different OSS alternatives. Selenium Grid and the plugins that can pull VMs easily. Simon did mention that people should be wary of running tests in the cloud if you don't have there own. Latency can make tests run slowly. Simon phrased it as "You are putting 4x the Internet in the way". Something we should be aware of when using cloud providers.
Lightning Talks Day Two
The lighting talks on day two where quite good, especially the ones for mobile by the Googlers. It was good to see the hard work that they had done to make WebDriver API work on Mobile is great. I did a talk on how we can create HeatMaps from WebDriver tests and David Clarke from my team did a talk on JsTestNet. It's a way to drive JsTestDriver in a more distributed fashion.
Jellyfish and other JavaScript testing tools - Adam Christian
Adam did a great job showing the power of JavaScript. Adam is the creator of numerous testing frameworks, and tells people not to do it, shows off his latest tool. Jellyfish can run tests on any device. Adam did a very similar talk at Selenium Conference but has updated it quite a bit so won't go on too much. Adam was extolling the virtues of doing very good JavaScript testing.
Conclusion
The other talks weren't really my thing but was surprised by how Google heavy this conference was but happy there wasn't the obvious recruiters. I would say for a feature talk make sure you practice and don't do fake banter.
Area: blog
Mon 12 Sep 2011
One of the major buzzwords floating around the Internet these days is continuous deployment. Continuous Deployment is the process of pushing code to servers straight from trunk at any moment. A number of companies do this on a commit/push and most of those companies have chosen to push the code straight to their production servers. This means they could be releasing up to once every 11 seconds. For those who want to learn more about it I suggest that you look up continuous deployment at Imvu, easy, flickr, Amazon and Mozilla.
The approach that I took for my setup was to have a deployment process similar to if you were to use Heroku or Joyent's no.de service. When I am ready to deploy I just want to do git push myserver master. I know this process won't scale well but for most sites this is all you need.
I have been developing an IRC bot for my team to use. It does lookups in Bugzilla, it can give people links to our Pivotal Tracker projects and does logging of what is happening in the channel and a few other things. So far my team has been making really good use of it, and being people who work in QA, they tend to find bugs or think of new features.
I wanted a way that I could fix bugs, or create features required, and release with out too much effort.
The following is what I do on my free AWS EC2 Micro instance. Please note that my IRC bot is written to run with Node.Js so some things will have a Node.Js slant but the my overall process will work for every one.
On the server you want to deploy do the following
mkdir project-folder.git # note the .git prefix
mkdir project-folder
cd project-folder.git
git init --bare
git config core.worktree project-folder
git config core.bare false
git config receive.denycurrentbranch ignore
cd hooks
mv post-receive.sample post-receive
Edit the post-receive file and add git checkout -f to the file
Now on our development machine a simple
git remote add micro ssh://user@aws-instance/~/project-folder.git
git push micro master
If you go into project-folder on your server you should see your code now. If you received a security exception make sure that you have copied your ssh public key, like the one you may have put on Github, in to ~/.ssh/authorized_keys on your server. Add a new line and then the ssh key. Try push again once that is correct.
Now we need a way to restart Node.Js when there is new code is deployed. I used Nodemon by Remy Sharp. Instead of doing node myapp.js just do nodemon myapp.js. Nodemon will restart the node process running you application when it sees changes to the JavaScript files.
If you do all of the above things you should have a basic Continuous Deployment process. I haven't created any rollback scripts for if something goes wrong but since you can push fixes quickly that shouldn't matter too much.
Area: blog
Mon 27 Jun 2011
HTML5 is becoming on the of the biggest things to hit the internet. Unfortunately it has also meant that Selenium 1 has hit a wall in with which technologies it can interact with. It can't play with these new technologies like canvas.
The new Selenium 2 API is trying to solve a number of these problems with the way that it drives the browser. One of the good things that has come out of the work on Selenium 2 is the ability to control elements on the page like a user would. This could be dragging items around a page or even working with canvas applications. Using the mouse, or keyboard, at a low-level to do some really good things.
The Advanced User Interactions API allows us to do things like dragging and dropping as well as gives us the ability to do something like hold down a key and move the mouse around or carry on typing. Below I have put together a couple examples below that you can how we can use the new Advanced User Interaction API that is part of the Selenium 2 API. A quick note, this API only works on Linux and Windows.
The first example uses the first demo page. The page allows you to draw what you want on a canvas element on the page. The script below will draw a triangle on the canvas.
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
class TestCanvas:
def setup_method(self, method):
self.firefox = webdriver.Firefox()
def teardown_method(self, method):
self.firefox.quit()
def test_that_we_can_draw_on_canvas(self):
self.firefox.get('http://www.theautomatedtester.co.uk/demo1.html')
canvas = self.firefox.find_element_by_id("tutorial")
drawing = ActionChains(self.firefox)\
.click_and_hold(canvas)\
.move_by_offset(-40, -60)\
.move_by_offset(30, 20)\
.move_by_offset(100, 200)\
.release(canvas)
#Now we know what we want to happen, let's perform the actions
drawing.perform()
Now that we have done something cool, let's have a look at using the API to do every day things with our sites. The next example will drag an element on the page. The following demos will use the following demo page
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
class TestDraggingAround:
def setup_method(self, method):
self.firefox = webdriver.Firefox()
def teardown_method(self, method):
self.firefox.quit()
def test_that_we_can_drag_a_div_around_the_page(self):
self.firefox.get('http://www.theautomatedtester.co.uk/demo2.html')
draggable = self.firefox.find_element_by_class_name("draggable")
dragging = ActionChains(self.firefox)\
.click_and_hold(draggable)\
.move_by_offset(30, 20)\
.move_by_offset(100, 200)\
.release(draggable)
#Now we know what we want to happen, let's perform the actions
dragging.perform()
But there are times where we need to be able to drag and element and drop it on another element. To do this there is a basic method called drag_and_drop(). See the example below.
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
class TestDragAndDrop:
def setup_method(self, method):
self.firefox = webdriver.Firefox()
def teardown_method(self, method):
self.firefox.quit()
def test_that_we_can_drag and_drop(self):
self.firefox.get('http://www.theautomatedtester.co.uk/demo2.html')
draggable = self.firefox.find_element_by_class_name("draggable")
droppable = self.firefox.find_element_by_id("droppable")
dragdrop = ActionChains(self.firefox)\
.drag_and_drop(draggable, droppable)
#Now we know what we want to happen, let's perform the actions
dragdrop.perform()
I suggest having a play with this.
Area: blog
Fri 10 Jun 2011
After the success of my last test day I thought I would share some how I, and my fellow Mozillians, go about organising an event.
For those who don't know what a test day, let me explain. A test day is a day where the Mozilla asks community members to help test something. The day is spent focusing everyone's effort on this application. The application could be a desktop, mobile or web application.
So what are the benefits of a test day? The main benefits are that we will have worked through an application and raised bugs or triaged them.
As an example, the last test day for automation ended with over 14 patches from employees, contractors and contributors. That is phenomenal. Another example is the test day on Friday that had 27 bugs raised or triaged while working through the various about: pages within Firefox. It has a huge benefit in that people learn from each other. The test day is run on IRC with everyone discussing bugs, asking for thoughts on the look and feel. For those new to testing we help with areas to focus on. We guide people through creating high quality bug reports with all the bits we need. For Mozilla QA we get the much needed, and much appreciated, help from the community as well as making them feel part of making great product.
So, how do you organise a test day?
- pick a product or feature
- pick a day to hold it on
- advertise it.
- learn from it
The advertise step may seem obvious but this is where you get people really excited about a project or feature. My approach is, and I think it is a great selling point to community members, is reminding them that all the bugs, bug triaging, patches are all publicly available so can be placed on your CV. Visible proof you know how to do what you do makes a huge difference when it comes to looking for your next job.
Read up on our Wiki page for more info, our products wouldn't be as good as they are without these test days!
P.s. There is a test day today for Developer tools and would be great if you could help out!
Area: blog
Wed 25 May 2011
Yesterday, 24 May 2011, was the second test day that I had organised for Mozilla WebQA team. This is my team that I work in mainly trying to help them with their automation.
Yesterday was the turn of Addons.mozilla.org (AMO) to get the attention. And attention it got. Over the day, we managed to get a 13 pull requests from Mozilla employees, contractors and contributors. For a 24 hour period I think this is great. A small project, dedicated to Selenium tests, can get that attention just astounds me!
We have in the region of 10 new tests, tests that we never had before. I know this doesn't sound a lot but one thing that many people don't realise is that Mozilla has a large number of websites that need to have test coverage, so having one site gain 10 tests over a day means that we can concentrate on the more mission critical parts of sites. As we all know, AMO is quite a mission critical site for Mozilla.
On the side of the test day, Marlena Compton ran a GitHub WTF course to help people get up to speed with using Git. This course was quite useful both internally at Mozilla for people wanting to learn but also to contributors on the day.
Remember that a Test day is a focused day for contributors but you don't need a test day to contribute. We are always available on IRC if you don't know where to start. I look forward to seeing more faces at the next test day!
Area: blog
Thu 12 May 2011
Since the last exercise at doing Selenium tests with Mozillians was a success I want to do it again!
On Tuesday 24 May 2011, I will be hosting a Test Day for Mozilla with help of some of my colleagues to help expand coverage on one of our projects and would love to have the Selenium Community help out!
So what is a test day? A test day is where Mozilla asks the community to help out on a project so we can make sure that it is properly tested and share ideas about testing the application. In this case we would like to expand our selenium coverage for http://addons.mozilla.org and you can help by going through the details from our Quality.mozilla.org (QMO) blog post. All details will be updated there if need be so register and keep an eye on it!
It starts around 10am GMT on the Tuesday through to 1am GMT on the Wednesday so hopefully giving everyone from around the world a chance to participate. Come talk Selenium and help us build some amazing tests. Your name could be added to the contributors list if we use the test!
P.s. This is a great way to practise Selenium and show what you can do to future employers!
Area: blog
