Wed 04 Aug 2010
Measuring the speed of how long a page takes to load is becoming one of the most important metrics that
we as software developers or testers can measure. This is because milliseconds can mean the difference between
a sale being paid for or abandoned. A couple of years ago Amazon did research that showed for every 100ms slower
the page was in loading they lost 1% of sales.
The downside to all of this is that we, as developers, testers and even ops, can only control the speed of things
within our code and within our datacenters. With this in mind we should be trying to record all of the information
that we can from within the browser. Last year at GTAC 2009 (
presentation video) David Henderson and I discussed how we used YSlow and Selenium to record page metrics and load
times of the web application. We made the total primed page size decrease by 85% with the work we did.
The unfortunate thing about the way we did this was that we had to hack YSlow because it didn't give us all the information
that we wanted. We also needed to hack Selenium because it blocked a few of the headers to do with element expiry. This
is not really ideal but with the new Web Timings API we are starting
to be able to do all of this with just a bit of JavaScript.
To access this we need to get hold of the performance object and then get it to return a dictionary of all the timing data.
The values that are returned are milliseconds from 1/1/1970. They give you all cool bits of information like how long it
takes to unload a page you're on before navigating to the new page and how long that takes to load. It also tells you how long
domain lookups take and how long redirects take. These are all neat tidbits of information when performance tuning your web application.
So below is a C# extension method that I have created that extends the WebDriver object to collect the information for you. Not all
browsers support this at the moment but as web timings becomes the standard the code will start returning data. Current browsers that
support this are Chrome 6 and IE9 pre3. A patch has been submitted to Mozilla for Firefox 4 so hopefully this will be applied soon.
using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using OpenQA.Selenium;
namespace AutomatedTester.PagePerf
{
public static class Extensions
{
public static Dictionary<string,object> WebTimings(this IWebDriver driver)
{
var webTiming = (Dictionary<string, object>)((IJavaScriptExecutor)driver)
.ExecuteScript(@"var performance = window.performance || window.webkitPerformance || window.mozPerformance || window.msPerformance || {};
var timings = performance.timing || {};
return timings;");
/* The dictionary returned will contain something like the following.
* The values are in milliseconds since 1/1/1970
*
* connectEnd: 1280867925716
* connectStart: 1280867925687
* domainLookupEnd: 1280867925687
* domainLookupStart: 1280867925687
* fetchStart: 1280867925685
* legacyNavigationStart: 1280867926028
* loadEventEnd: 1280867926262
* loadEventStart: 1280867926155
* navigationStart: 1280867925685
* redirectEnd: 0
* redirectStart: 0
* requestEnd: 1280867925716
* requestStart: 1280867925716
* responseEnd: 1280867925940
* responseStart: 1280867925919
* unloadEventEnd: 1280867925940
*/
return webTiming;
}
}
}
Area: blog
Mon 05 Jul 2010
First let me start by thanking those who followed the Selenium StackExchange Proposal. We managed to meet all the criteria that was set by the StackExchange Team and have moved onto the second stage.
The second stage is the commitment stage. This is where we get people to commit that they are going to use it to ask questions or answer questions when they are put on the site. This is a very important stage because it shows how strong the Selenium Community is and in my opinion we do have a strong community. Unfortunately, my opinion doesn't matter that much in the StackExchange process so if you haven't committed to the Selenium StackExchange Site please click on the link and commit.
You never know when you may need an answer to a question that may already have been asked and it's easy available to you with the new site
Area: blog
Tue 22 Jun 2010
A question I get asked regularly, and is regularly asked on forums I visit, is how do "I make my tests maintainable?".
This is normally from a tester upset that the developers are commenting out tests because they feel its too much work to update the tests.
The next popular question is "how can I get my non-technical colleagues creating automated tests?". For me the answer to both is very similar.
Make the tests self-documenting and easy to read (maybe using a DSL). You, as a developer, would do this with your production code so why not do it with your tests.
Now, what do I mean by self-documenting and easy to read?
- Methods should explain what they are for. This goes for test and production.
I like to format my test methods like ShouldDoX. This is similar to what BDD practioners do for their tests.
If another prerson comes along to look at some code that you have written, and your tests,
they will be able to see what you meant it to be doing.
public void ShouldLoginAndChangeStatus()
for a social networking app explains exactly what they are expecting.
- Methods and tests should be succinct
Methods that are huge, in test and production, are likely to have bugs in them because people have to
scroll to get to the bottom. They are likely to have business logic bugs when that happens. I also like abstract
as much as I can so that it makes the code readable.
Which is better for readability?
selenium.click("//a[contains(@blah,'someIdButNotAllOfIt)]")
or
// linkToCoolStuff is a constant holding the XPath above
selenium.click(linkToCoolStuff)
-
Maybe using a DSL?
We could take this one step further and create a DSL so that we do not have to care how it does it, we just know it will happen.
For UI testing I like the idea of the Page Object design pattern for creating tests.
You split the UI and the tests apart so that you can have a test looks like the following.
[Test]
public void ShouldLoadHomeThenGoToXpathTutorial()
{
Home home = new Home(selenium);
SeleniumTutorials seleniumTutorials = home.ClickSelenium();
SeleniumXPathTutorial seleniumXPathTutorial = seleniumTutorials.ClickXpathTutorial();
Assert.True(seleniumXPathTutorial.
IsInputOnScreen(SeleniumXPathTutorial.FirstInput));
Assert.True(seleniumXPathTutorial
.IsInputOnScreen(SeleniumXPathTutorial.SecondInput));
Assert.True(seleniumXPathTutorial
.IsInputOnScreen(SeleniumXPathTutorial.Total));
}
I have created a tutorial on how to use the Page Object Pattern with Selenium
to make tests that are extremely maintainable and also allows non-technical people to write tests without having to understand how Selenium works.
Area: blog
Fri 11 Jun 2010
A while ago Dave Hunt, a Selenium Contributor and London Selenium User Group organiser, suggested that perhaps we should have a look at creating a StackExchange site for Selenium. He suggested it after the StackOverflow team announced changes to StackExchange
On Wednesday Dave created our proposal so that we can get the ball rolling. I have signed up and want to help this proposal as much as I can. There are a couple reasons why I am advocating this so much.
- Its a lot friendlier to use than Google groups
- It will show up possible duplicates in questions. Chances are that if you are having an issue, someone else has had that issue before
- It's like a game! It's points based so people will try get the most points that they can.
So what can you as a reader of this blog do? Well, if you are a Selenium User I suggest following to the proposal by logging into Area51 and clicking the follow button. The next thing to do is add some example questions if they aren't there. The more example questions the better as it will give people a better idea of what the StackExchange will be used for. Finally, start voting on the example questions. You do this by clicking on one three links on the left of the question as shown in the image below.
I have benefitted from using the StackOverflow type sites(StackOverflow, ServerFault, etc) since their launch and I think that as a community we can all benefit from this site so please help the cause
Area: blog
Sat 05 Jun 2010
This morning I work up this morning thinking that today was going to be a really good day. The sun was shining and it was really warm. The thoughts going through my head was "I need coffee then check my emails then phone my friend Chris and see if he wants to cycle over to the Isle of Purbeck today"
I did all of those things quite successfully and then thought, lets have a look at what is happening on twitter. This was both a good idea and a bad idea. It was a good idea as I got to see what was happening in the world. The bad idea part was I saw battle of the sexs happening between Lanette Cream and James Bach about diversity in the industry.
This appears to have upset a number of people as their were some negative thoughts from both parties. The main crux of the argument was one was saying we need to encourage diversity and the other saying thats not true.
For me I think we need to encourage individuality. I grew up in South Africa and have seen the aftermath of Apartheid and have been victim to affirmative action. Affirmative Action was the main reason why I decided to move to the UK after university. Filling positions to meet quotas is always going to do more harm than good. There were stories, don't know how true, of secretaries becoming directors of companies in South Africa so that the "balance" was corrected.
This idea makes me mad! Everyone should get to somewhere on their own merits and driving forward this idea that we need to get more woman into IT is going to do more harm. We need to get more people in general into the industry. This shouldn't matter if they are male, female, pink, green or indifferent. Over the last decade there has been a drop in graduates for maths and science degrees. There are fewer Computer Scientists, engineers and mathematicians. We should be driving that forward and hopefully that will organically get people from diverse backgrounds into our industry.
I had a brief discussion over twitter with Michael Bolton where we seemed to agree that we need to have different perspectives on applications when we test. This does comes from hiring different sexs and races but that doesn't guarantee anything. People who go to similar schools will have similar perspectives on things. Being from South Africa I had people of different races and ethnicities in my classes but we all have similar perspectives. This alone makes me hate the diversity argument on sex/race.
I am an IT professional so treat me like one!
Area: blog
Thu 03 Jun 2010
I have recently come back from a well deserved holiday in South Africa. Now that I am back I thought that I would carry on working on the .NET Selenium 2 code. I am just about to port the Support project over but thought that I would check that ther weren't any blocking issues that people were experiencing with the .NET Code.
So over the last couple days I have been going though the issues that are on Google Code and have been suprised by the quality of the bug reports that people have been submitting. This is from people who spend most of their day writing bug reports or spend their days reading bug reports to fix issues.
The question that I kept asking myself is why do they think this is a good enough bug report? Maybe they have to pay their company per character they type when not on intranet sites? They think that Selenium Developers are psychic? The make certain assumptions like everyone in the world writes tests like them against applications that are exact clones to their application.
I am both a developer and a tester so have the unique perspective when looking at bug cases. When I raise a case I make sure that I think about the what is needed for someone, who may not know the system, to fix the issue. This, for me, is an extremely powerful way to raise cases. I always raise my cases with every exact step needed to solve the issues. And if the bug is UI related I will try add a screenshot just in case I may have forgotten something.
Marlena Compton says that Being a writer made her a better tester. I have to agree with that but I think being a developer has made me a better tester too. Being a writer has made me articulate the issue clearly. Being a developer has made me write down what is needed to fix this issue without causing the other person to go grey.
Next time you raise a case, think about how you have phrased it!
Area: blog
Thu 06 May 2010
Last night I was trying to show a client of mine how a Selenium script looks when running on my linux box. My client has an issue that when they run the test the browser just dies. Screen sharing was an option but when we have done this before the screen sharing doesn't show the screen that well. The other issue was trying to get time when we were both free.
I wanted just a video of what was happening and then I can watch it to see what the issue may be. This way my client could do it in their own time when they have a moment. There are tools out there to record a person's screen. There is camtasia which is really good. Unfortunately I did not want my client to worry about when to start recording and when to stop recording. I wanted a no frills no fuss solution for them. They are not technical in the slightest which is why they hired me.
My next thought came back to something that Jason Huggins said to me last year at GTAC 2009. Jason had put a small API onto pyvnc2swf and had created Castro. Castro is a python library that allows users to programmatically record what is happening on the screen and then create a SWF file so you can view it within a web page for everyone to see. This is exactly what I was after since I could write a script for my client to run and that would record the test. I would hopefully be able to then see where the issue may lie.
It did solve my problem brilliantly and this has become the standard way for me to help my clients. If something isn't working perfectly for them I get them to work through the steps below and then record what the issue could be.
I have only done this on Ubuntu 9.10, hence the use of apt-get, but it should work on all machines except x64 Windows. This is because Python SetupTools doesn't really work on x64 Windows that well because of the way Windows controls its registry.
Setup VNCServer :
System > Preferences > Remote Desktop Prefences
Check "Allow others to view your desktop
In a terminal window run the follow:
sudo apt-get install python-dev python-pygame python-tk python-setuptools ffmpeg flvtool2
sudo easy_install castro
Once all that has been installed you are now ready to create your video of your test.
from selenium import selenium
import unittest, time, re, castro
class mypms(unittest.TestCase):
def setUp(self):
self.verificationErrors = []
self.video = castro.Castro()
self.video.start()
self.selenium = selenium("localhost", 4444, "*firefox", "http://www.theautomatedtester.co.uk/")
self.selenium.start()
self.selenium.window_maximize()
def test_my(self):
sel = self.selenium
sel.open("/")
sel.click("link=blog")
sel.wait_for_page_to_load("30000")
def tearDown(self):
self.selenium.stop()
self.video.stop()
self.video.process()
self.assertEqual([], self.verificationErrors)
if __name__ == "__main__":
unittest.main()
If you were to run the script above it would dump the file into /tmp and you can view the video over there. This is a very handy tool and a quick way to see where scripts are going wrong and hopefully be happy like me.
Area: blog
Wed 21 Apr 2010
Last night was the 2nd ever Selenium User Group in London and what a night it was! Arriving at their office we were ushered up to 4th floor where in true Google style we were given food, lots of food and beer so that we could all hang out and have a chat.
The event was mainly organised by Dave Hunt and he did a brilliant job organising the speakers and with the help of Simon Stewart organising a venue with food and beer.
Dave Hunt did a talk on how to move from Selenium 1 in the IDE and move to Selenium 2. He showed how you can use the new Selenium IDE Plugin Architecture that Adam Goucher got going. Dave created the WebDriverBacked Formatter plugin with Adam and he showed it off to good effect.
It was then my turn to speak. I was speaking about Selenium 2 and what it means for the Average QA. In the talk I described the major differences like the API is a lot more concise in Selenium 2 and also describes some of the things I described in my "Why Should I move to Selenium" blog post I did earlier this year. I have put my presentation below for all to see.
And I have put my demos on GitHub for all to see and play with.
Unfortunately I had to leave at this point so I missed Eran's talk and the pub session afterwards but from what I hear it was a really good way to finish the night.
Thanks to all that turned up for the User Group meeting and making a great event!
Area: blog
Sat 06 Mar 2010
Today I officially received my contract to write a book on Selenium. I am going to be writing a beginners guide showing people how they are start test automation working using Selenium IDE then moving on to Selenium RC and Selenium Grid. There will also be chapters on Selenium 2.
The outline I went for was to help people who want to learn Selenium so they can maintain existing tests but also start writing tests using Selenium 2 and migrating their Selenium 1 tests over to Selenium 2.
So look out for it later this year!
Area: blog
Sat 20 Feb 2010
Over the last month or so I have been spending my time writing .NET code for the new Selenium 2 Bindings. I have spent most of my time working on the Google Chrome Driver. But below is the story of 1 Selenium bug and 1 Mono bug and my exploration of how processes work in .NET on a Mac.
I was quite confident about the .NET bindings since it was extremely easy to get them working on my laptop. I admit I got nervous about how it would work on Non-Windows platforms since it was my first time writing .NET code on a platform that isn't windows. I shouldn't have been nervous. It was fairly easy to install Mono and MonoDevelop on my Ubuntu VM and ran the NUnit tests that we have. Google Chrome started and we had tests passing. Brillant because there was little code changes, I just updated the code where to find the Google Chrome application.
I was confident until I started working Mac OS X. I have very little experience on a Mac. Other than using it to run Selenium Grid or using it with a Browser I know very little about how the OS works.
On the Mac I installed Mono and MonoDevelop and fired them up. I went in and updated the code so that if the .NET code was running on a Mac it knew where to find the Google Chrome application and would then start up the process using the standard Process.Start() method from System.Diagnostics library. Feeling extremely confident I fired up NUnit and clicked Run. They all failed. Why? Well the browser wasn't loading. Selenium without a browser, excluding when used with HTMLUnit, isn't going to work.
I started debugging my code to see why the browser wasn't loading. The first issue that I found was the Selenium 2 .NET code wasn't recognising the Mac OS platform. We have logic that checks which platform you are on and we hadn't put a conditional for Mac OS. Updated my code and ran the tests again and they went red again. Why? No browser again.
Debugging again I came across a rather large and very annoying bug in Mono. Mono when running on Mac OS platform reports that it is running on Unix/Linux. This isn't too ridiculous since Mac OS is based on BSD however it means now that when ever the code needs to launch the browser it needs to up to 3 hits to the disk to see if it is where I think its. It also means the fix I added in Selenium earlier doesn't really matter. Anyway the code now has the ability to launch the browser...or does it? It doesn't as my tests are still failing.
I started debugging again. The error I was getting was that I was trying to open the application and open wouldn't accept the arguments I was passing in because they are not valid for Open. Since I have never developed on a Mac I had to do some research and found out that the defaults for Process.Start() want to Open the file rather than execute it. This is perfect if you want to open text files but not applications. So I tried to see about creating a bash script to start up Google Chrome. One obvious downside if it did start up was that I wouldn't have an easy way to kill it. I was starting to think that I was going to be doing Yak Shaving to get this to work.
I next had a look at using Monobjc. It is a library that allows you to write Cocoa code in .NET and there it uses a bridge to get your code working. It is something that one of the Watin guys had done to get Watin working on a Mac. After a lot of pulling my hair out for a while I gave up and asked a question on StackOverflow.
The 5th answer I was given, and the one I accepted, was that I had forgot to set a property called UseShellExecute. My code for starting up went from
chromeProcess = Process.Start(GetChromeFile(), String.Concat(Arguments, serverUrl));
to
chromeProcess = Process.Start(new ProcessStartInfo(GetChromeFile(), String.Concat(Arguments, serverUrl)) { UseShellExecute = false });
The subtle change of explicitly creating a new instance of ProcessStartInfo and updating the property { UseShellExecute = false } meant that my code went from opening an application to executing a application. The difference tells Process not to open a shell according to MSDN but on a Mac means it does what I want. I hope this is helpful for anyone else in the future.
Area: blog
