Adrian Grigore

User Interface Testing with Selenium Vs. Coded UI

In Testing on May 6, 2010 at 21:06

This week I needed to write some automated UI level tests for my current web project (LogMyTime Zeiterfassung). After a bit of research, I narrowed the list of candidates down to Selenium and Visual Studio UI Automation testing (a.k.a. Coded UI). Then I spent a couple of hours with each tool to see what works better for me.

So here’s my first impression:

Selenium

Selenium consists of an IDE that comes as a Firefox Add-On, a remote control for automated playback of tests and Selenium Grid, which is used for distributing tests across multiple servers.

This is what I found out after two hours of playing with the Selenium IDE:

The Good:

  • Very easy to get your first tests running.
  • Highly intuitive interface, at least when it’s working properly.
  • Good workflow since the IDE is visible side-by-side with your browser.
  • Inline help shown inside the IDE for quick reference
  • Generated source code (html) is very small and easily readable
  • Tests can be exported in many different programming languages
  • Test recording and playback is very fast.
  • Easy to extend with custom actions and assertions

The Bad:

  • Extremely buggy IDE. Within about two hours of usage I lost some test cases, saw some exceptions and noticed some problems with the source code export feature.
  • Re-use of recurring test steps and assertions does not seem to be possible without custom coding and a lot of copy and paste. Edit: I just found a workaround to this. See Edit1 below the article.
  • Simple assertions (such as check if a given string is shown on the web page) can be added with a few mouse clicks. But more thorough tests such as “check if the string “foo” appears in a div with the ID “bar” require custom coding.
  • All custom code has to be written in JavaScript. Not a big issue, but I still prefer type-safe languages.
  • No out of the box support test setup routines. This again requires custom coding.

Coded UI

Coded UI is a new addition to Visual Studio 2010. Tests can be recorded with the new MS Test manager (which I can’t use since it requires Team Foundation Server) or from within Visual Studio. Tests can be replayed from within Visual Studio, MS Test Manager or via command line on any Windows machine with an installed Test Agent.

In contrast to Selenium, Coded UI can test many different flavors of user interfaces, such as MFC, WPF, Web apps and even Silverlight applications.

So here’s my first impression regarding Coded UI:

The Good:

  • Everything seems very stable.
  • You can test many different kinds of user interfaces, not just he web.
  • Frequently recurring test steps can easily be re-used. It’s up to you how fine-grained you want to record your test steps. Calling a step (consisting of one or more actions) is just a matter of a single function call.
  • Generates C# and XML code by default.
  • Features fuzzy matching of UI elements. This seems to make the tests more robust when updating the user interface.

The Bad:

  • Even very simple tests generate a lot of code.
  • Test steps are stored in an XML file (called UIMap) which in turn compiles to C# code. The UIMap is big and clunky and there currently is no editor or documentation for it. So if you want to make some changes, things can get complicated unless you want to re-record an entire test step.
  • Creating very simple assertions (such as “look for the string “foo” on this web page”) is a bit clumsy and requires too many mouse clicks. An IE accelerator would be great!
  • No support for Safari, Opera or Chrome. On the bright side, Coded UI has an open interface, so anyone could implement it.
  • While fuzzy matching works great in most cases, It can get in the way in others.
  • Not quite as intuitive as Selenium at the beginning.
  • Tests record and run quite a bit slower than with Selenium.

 

Summary

In essence Selenium looks like a great lightweight tool with a lot of potential, but also with big disadvantages. I wouldn’t even have had a look at Coded UI if only the Selenium IDE was more stable and test steps could be wrapped into groups without custom coding.

But as it is right now, I would only work with Selenium by recording each test separately, exporting it as a C# NUnit test and copying / pasting code into my own test library. This should work around the stability issues, but it is quite tedious.

Coded UI on the other hand is clunky, but stable. Workflow is not nearly as smooth as with Selenium, but still quite good. And it works with C# code instead of JavaScript, which is a big plus for me. Now if only there was a good editor for previously recorded tests built into Visual Studio…

In the end I’ve decided to use Coded UI. It’s not perfect, but it is very good first product iteration. Due to Selenium’s stability issues it looks like the better choice for me at the moment.

Some useful links for getting started:

Edit from 12/05/2010: Actually it’s a draw…

As I just found out , there is a way to re-use test steps in Selenium IDE without custom coding:

  • Create a separate test case for each sequence of steps that should be made reusable. (such as “create a blog article”)
  • Create separate test suites for each test that should reuse those test steps. For example you might have a test suite with the tests “Create a blog article” “Edit the blog article” and one with the tests “Create a blog article”, “Delete the blog article”. Note that this is just a poor example since you could also cover both in one single test suites, but as tests become more complex having all reusable test steps in on single test suite becomes impractical.
  • You have several different options for running all test suites in sequence.

This makes Selenium a much more attractive choice. Since I have not written too many tests in Coded UI yet and I prefer the workflow of the Selenium IDE, I am now implementing some of my tests in both frameworks to see which approach leads to more robust and maintainable tests in the long run.

 

kick it on DotNetKicks.com    Shout it

Advertisements
  1. User Interface Testing with Selenium Vs. Coded UI | devermind.com…

    Thank you for submitting this cool story – Trackback from DotNetShoutout…

  2. You may give Watin a try, the IDE is as buggy as Selenium’s but you write tests in C#

  3. You can also write tests for Selenium in C#, but I prefer to use an IDE for recording.

  4. Just a note of caution, Element names(ex: xpath) can be brutal when using Selenium IDE

  5. @Rakesh: I know, I noticed it several times already. As I said, none of the contestants is Ideal. But I haven’t found a better alternative yet.

  6. I’ve used Selenium (with Ruby) for a long time but since I’m now working for a company that is using MSFT products all out I need to start looking at VSTS coded ui. This is what I found a couple of months ago:
    http://social.msdn.microsoft.com/Forums/en/vsautotest/thread/1562dd8c-7691-4a9e-a5f0-09fed5b1f628

    I will look at it again but I just can’t work out the maintainability of coded ui test…

  7. @Mattias: Yes, I found that thread before shortly before I dived into Codedui as well. It is halfway true and one of the reasons that made me reconsider using CodedUI. But I think it depends on how you create new tests and the UI maps to make them work. It’s actually possible to create a quite tidy UI map with CodedUI, but if you aren’t careful, it is indeed going to be a mess.

  8. Could you elaborate this? “It’s actually possible to create a quite tidy UI map with CodedUI”

  9. @Mattias: When you create CodedUI tests with the recorder, it creates a kind of map which holds fuzzy references to your user interface. Depending on the kind of software you are testing (web based, multi-domain in our case) the auto-generated UI map can get quite messy because CodedUI fails to recognize the system behind the UI. It just looks for input ids, classes, urls. For example, it might create two maps for testing the same web page in two different tests. This is usually not what you want, so you can alter the UI map (for example by hand-editing the generated XML file) to keep your map tidy. But CodedUI has no editor for the map yet, so keeping the map tidy is a LOT of work. Which is one of the main reasons I am not using it anymore.

  10. can some one give an idea how to record a windows application with coded UI

  11. @daya: It’s all described in detail at MSDN

  12. Hi,

    I am new to Coaded UI test.
    I have one query on Coaded UI Test.

    What happnneds If I record winform application on 1024x 786 resolution and placyback on 800x 600?

    Thanks
    – Naga

  13. @Naga: I haven’t tried out CodedUI with Winforms apps, but I don’t think that click events are resolution-dependent. If it works similarly to web tests, then CodedUI would try to find controls based on on several fuzzy criteria such as ID, caption, etc.

  14. Hi,

    Is there any way to record and generate code without using VS 2010 & Microsoft Test Manager ?
    I mean, Can launch Codec UI Test Builder without using VS & MTM ???

    Thanks
    – Naga

  15. Depending on what you are testing, using a Performance Web Test in Visual Studio 2010 has many advantages to using Coded UI. Don’t let the word “performance” fool you — they can be very effective for functional testing.

    To determine which is the best tool for the job, look at what you are testing. If you are working with the HTML content Web Performance Tests are a great choice. You can click on objects, navigate, deal with AJAX, etc. You can validate that specific content exists or doesn’t exist, that a value is or isn’t equal to a known value, that a particular attribute of a tag has a specific value and many other things. You can also use extraction rules to get information from the result of one request to use in a future request. In summary, if you are working primarily with the HTML, Web Performance Tests are much easier, provide a nice development UI, and make things like data bining easy. I would only use Coded UI if you are working with client-side javascript or browser-specific situations.

  16. Hi Adrian,
    Great post introducing the differences, i just wondered nearly 1 year on, if you are still loving selenium or you have switched back to visual studio or found another alternative etc. The only thing i found with my brief experience of selenium IDE is that its firefox only and only seems to work stable on early versions of firefox and not say firefox 4 onwards as the browser continually evolve…

    Brad

  17. @Brad: Actually Selenium IDE seems to have gotten a bit better during the past year. It’s still not a love relationship, but I’m still using it in lack of better alternatives.

    It also works fine more me on Firefox 4. It’s just the silly compatibiliy mechanism in Firefox that’s blocking Selenium IDE by default. You can switch that off by installing the compatibility reporter add-on though: https://addons.mozilla.org/de/firefox/addon/add-on-compatibility-reporter/?src=api

  18. […] some additional recent information on user-interface testing for generic JavaScript applications at this post which discusses Selenium and Coded UI as […]

  19. Hi Adrian,
    Coded UI Test has a new editor since Nov 2010. Reading your post , just thought of sharing it with you. Doc is here at http://msdn.microsoft.com/en-us/library/gg269469.aspx. However its not free, you might need an MSDN subscription to download it.

    Thanks
    Anindya

  20. @Anindya: This sounds great! I’ll have a look and write a follw-up article when I have the time.

  21. Highly recommend this article. The idea of PageObjects is very useful and something that we use in Selenium and Ruby. These practices are useful in small applications. Maintainability of your test suites is as important as your production code so I’m not a fan of the way they title this for “Large Applications”

    http://msdn.microsoft.com/en-us/library/ff398056.aspx

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: