This week I needed to write some automated UI level tests for my current web project (LogMyTime Zeiterfassung). After a bit of research, I narrowed the list of candidates down to Selenium and Visual Studio UI Automation testing (a.k.a. Coded UI). Then I spent a couple of hours with each tool to see what works better for me.
So here’s my first impression:
Selenium consists of an IDE that comes as a Firefox Add-On, a remote control for automated playback of tests and Selenium Grid, which is used for distributing tests across multiple servers.
This is what I found out after two hours of playing with the Selenium IDE:
- Very easy to get your first tests running.
- Highly intuitive interface, at least when it’s working properly.
- Good workflow since the IDE is visible side-by-side with your browser.
- Inline help shown inside the IDE for quick reference
- Generated source code (html) is very small and easily readable
- Tests can be exported in many different programming languages
- Test recording and playback is very fast.
- Easy to extend with custom actions and assertions
- Extremely buggy IDE. Within about two hours of usage I lost some test cases, saw some exceptions and noticed some problems with the source code export feature.
Re-use of recurring test steps and assertions does not seem to be possible without custom coding and a lot of copy and paste.Edit: I just found a workaround to this. See Edit1 below the article.
- Simple assertions (such as check if a given string is shown on the web page) can be added with a few mouse clicks. But more thorough tests such as “check if the string “foo” appears in a div with the ID “bar” require custom coding.
- No out of the box support test setup routines. This again requires custom coding.
Coded UI is a new addition to Visual Studio 2010. Tests can be recorded with the new MS Test manager (which I can’t use since it requires Team Foundation Server) or from within Visual Studio. Tests can be replayed from within Visual Studio, MS Test Manager or via command line on any Windows machine with an installed Test Agent.
In contrast to Selenium, Coded UI can test many different flavors of user interfaces, such as MFC, WPF, Web apps and even Silverlight applications.
So here’s my first impression regarding Coded UI:
- Everything seems very stable.
- You can test many different kinds of user interfaces, not just he web.
- Frequently recurring test steps can easily be re-used. It’s up to you how fine-grained you want to record your test steps. Calling a step (consisting of one or more actions) is just a matter of a single function call.
- Generates C# and XML code by default.
- Features fuzzy matching of UI elements. This seems to make the tests more robust when updating the user interface.
- Even very simple tests generate a lot of code.
- Test steps are stored in an XML file (called UIMap) which in turn compiles to C# code. The UIMap is big and clunky and there currently is no editor or documentation for it. So if you want to make some changes, things can get complicated unless you want to re-record an entire test step.
- Creating very simple assertions (such as “look for the string “foo” on this web page”) is a bit clumsy and requires too many mouse clicks. An IE accelerator would be great!
- No support for Safari, Opera or Chrome. On the bright side, Coded UI has an open interface, so anyone could implement it.
- While fuzzy matching works great in most cases, It can get in the way in others.
- Not quite as intuitive as Selenium at the beginning.
- Tests record and run quite a bit slower than with Selenium.
In essence Selenium looks like a great lightweight tool with a lot of potential, but also with big disadvantages. I wouldn’t even have had a look at Coded UI if only the Selenium IDE was more stable and test steps could be wrapped into groups without custom coding.
But as it is right now, I would only work with Selenium by recording each test separately, exporting it as a C# NUnit test and copying / pasting code into my own test library. This should work around the stability issues, but it is quite tedious.
In the end I’ve decided to use Coded UI. It’s not perfect, but it is very good first product iteration. Due to Selenium’s stability issues it looks like the better choice for me at the moment.
Some useful links for getting started:
- The official Selenium documentation
- Useful Selenium extensions
- OwenG’s introduction to Coded UI
- Some tips on structuring CodedUI Tests by Gautam Goenka
Edit from 12/05/2010: Actually it’s a draw…
As I just found out , there is a way to re-use test steps in Selenium IDE without custom coding:
- Create a separate test case for each sequence of steps that should be made reusable. (such as “create a blog article”)
- Create separate test suites for each test that should reuse those test steps. For example you might have a test suite with the tests “Create a blog article” “Edit the blog article” and one with the tests “Create a blog article”, “Delete the blog article”. Note that this is just a poor example since you could also cover both in one single test suites, but as tests become more complex having all reusable test steps in on single test suite becomes impractical.
- You have several different options for running all test suites in sequence.
This makes Selenium a much more attractive choice. Since I have not written too many tests in Coded UI yet and I prefer the workflow of the Selenium IDE, I am now implementing some of my tests in both frameworks to see which approach leads to more robust and maintainable tests in the long run.