Adrian Grigore

User Interface Testing with Selenium Vs. Coded UI

In Testing on May 6, 2010 at 21:06

This week I needed to write some automated UI level tests for my current web project (LogMyTime Zeiterfassung). After a bit of research, I narrowed the list of candidates down to Selenium and Visual Studio UI Automation testing (a.k.a. Coded UI). Then I spent a couple of hours with each tool to see what works better for me.

So here’s my first impression:

Selenium

Selenium consists of an IDE that comes as a Firefox Add-On, a remote control for automated playback of tests and Selenium Grid, which is used for distributing tests across multiple servers.

This is what I found out after two hours of playing with the Selenium IDE:

The Good:

  • Very easy to get your first tests running.
  • Highly intuitive interface, at least when it’s working properly.
  • Good workflow since the IDE is visible side-by-side with your browser.
  • Inline help shown inside the IDE for quick reference
  • Generated source code (html) is very small and easily readable
  • Tests can be exported in many different programming languages
  • Test recording and playback is very fast.
  • Easy to extend with custom actions and assertions

The Bad:

  • Extremely buggy IDE. Within about two hours of usage I lost some test cases, saw some exceptions and noticed some problems with the source code export feature.
  • Re-use of recurring test steps and assertions does not seem to be possible without custom coding and a lot of copy and paste. Edit: I just found a workaround to this. See Edit1 below the article.
  • Simple assertions (such as check if a given string is shown on the web page) can be added with a few mouse clicks. But more thorough tests such as “check if the string “foo” appears in a div with the ID “bar” require custom coding.
  • All custom code has to be written in JavaScript. Not a big issue, but I still prefer type-safe languages.
  • No out of the box support test setup routines. This again requires custom coding.

Coded UI

Coded UI is a new addition to Visual Studio 2010. Tests can be recorded with the new MS Test manager (which I can’t use since it requires Team Foundation Server) or from within Visual Studio. Tests can be replayed from within Visual Studio, MS Test Manager or via command line on any Windows machine with an installed Test Agent.

In contrast to Selenium, Coded UI can test many different flavors of user interfaces, such as MFC, WPF, Web apps and even Silverlight applications.

So here’s my first impression regarding Coded UI:

The Good:

  • Everything seems very stable.
  • You can test many different kinds of user interfaces, not just he web.
  • Frequently recurring test steps can easily be re-used. It’s up to you how fine-grained you want to record your test steps. Calling a step (consisting of one or more actions) is just a matter of a single function call.
  • Generates C# and XML code by default.
  • Features fuzzy matching of UI elements. This seems to make the tests more robust when updating the user interface.

The Bad:

  • Even very simple tests generate a lot of code.
  • Test steps are stored in an XML file (called UIMap) which in turn compiles to C# code. The UIMap is big and clunky and there currently is no editor or documentation for it. So if you want to make some changes, things can get complicated unless you want to re-record an entire test step.
  • Creating very simple assertions (such as “look for the string “foo” on this web page”) is a bit clumsy and requires too many mouse clicks. An IE accelerator would be great!
  • No support for Safari, Opera or Chrome. On the bright side, Coded UI has an open interface, so anyone could implement it.
  • While fuzzy matching works great in most cases, It can get in the way in others.
  • Not quite as intuitive as Selenium at the beginning.
  • Tests record and run quite a bit slower than with Selenium.

 

Summary

In essence Selenium looks like a great lightweight tool with a lot of potential, but also with big disadvantages. I wouldn’t even have had a look at Coded UI if only the Selenium IDE was more stable and test steps could be wrapped into groups without custom coding.

But as it is right now, I would only work with Selenium by recording each test separately, exporting it as a C# NUnit test and copying / pasting code into my own test library. This should work around the stability issues, but it is quite tedious.

Coded UI on the other hand is clunky, but stable. Workflow is not nearly as smooth as with Selenium, but still quite good. And it works with C# code instead of JavaScript, which is a big plus for me. Now if only there was a good editor for previously recorded tests built into Visual Studio…

In the end I’ve decided to use Coded UI. It’s not perfect, but it is very good first product iteration. Due to Selenium’s stability issues it looks like the better choice for me at the moment.

Some useful links for getting started:

Edit from 12/05/2010: Actually it’s a draw…

As I just found out , there is a way to re-use test steps in Selenium IDE without custom coding:

  • Create a separate test case for each sequence of steps that should be made reusable. (such as “create a blog article”)
  • Create separate test suites for each test that should reuse those test steps. For example you might have a test suite with the tests “Create a blog article” “Edit the blog article” and one with the tests “Create a blog article”, “Delete the blog article”. Note that this is just a poor example since you could also cover both in one single test suites, but as tests become more complex having all reusable test steps in on single test suite becomes impractical.
  • You have several different options for running all test suites in sequence.

This makes Selenium a much more attractive choice. Since I have not written too many tests in Coded UI yet and I prefer the workflow of the Selenium IDE, I am now implementing some of my tests in both frameworks to see which approach leads to more robust and maintainable tests in the long run.

 

kick it on DotNetKicks.com   

Visual Studio 2008 randomly hangs when opening .aspx .ascx or .master files

In ASP.NET on January 29, 2010 at 15:12

Visual Studio 2008 has been running quite smoothly on my computer until this week. But then it started randomly hanging when opening an .aspx .ascx .master file. The problem even occurred when opening the file in isolation, instead of opening the file as part of a solution. Deactivating all add-ins did not help.

Fortunately It seems I just found the solution: After uninstalling the “Microsoft Visual Studio Web authoring Component”, the problem seems have to have disappeared. I’m not even sure what this component does, but it seems to be a relict from Visual Studio 2005 and so far I have not noticed any missing features since I uninstalled it.

ASP.NET MVC Tip #4: Client-side form validation made easy – Part 2

In ASP.NET MVC on January 11, 2010 at 15:00

In my previous article about ASP.NET MVC Client-Side validation, I showed how to set up your project so that you don’t have to write any custom JavaScript code for any new validation rules. This approach also covered remote client-side validation – these are rules which require server-side resource in order if a particular input field is valid or not.

Since I wrote the first part of the article, Steve Sanderson adopted my idea of automatic client-side validation and baked a similar feature right into the latest version of xVal.

One important thing is still missing though: What do you do when validation depends on server-side resources but also involves several different form fields? For example, let’s assume you were implementing a form for your website users to update their contact information.

Follow

Get every new post delivered to your Inbox.