Extend MSTest : TestCaseSource aka runtime inputs for test cases

Extending MSTest

Read this great post first by William Kempf. And It's true!
 http://www.digitaltapestry.net/blog/extending-mstest

 Next if you're willing to bear with all that, these are the only 2 posts from circa-2009 on the intertubes that give you any hope

MSDN Blogs - Bruce Taimana Part 1. Part 2
 

Writing an extension in 2013

The details have changed since 2009 and combined with the paucity of information, it was three days before I had something to show.

So my target extension was going to bring in NUnit's TestCaseSource functionality, whereby you could supply the parameters for a parameterized test via a method (at runtime. Compile-time is already supported via XML and DBs.. though cumbersome.)

[TestMethod] 
[TestCaseSource("DivideCases")]
public void MultipleParams(int n, int d, int q)
{
   Assert.AreEqual( q, n / d );
}

public static object[] DivideCases()
{
   return new[] 
   {
       new object[] {12, 3, 4},
       new object[] {12, 2, 6},
       new object[] {12, 4, 3}
   };
}

GUI Testing rehab : Can we start saying NO?

Testing GUIs has been hard, tedious, painful... just bad. But they have been an occupational hazard due to lack of feasible alternatives..

There's a hard-earned confidence you get when you see a dancing UI twisting, turning... testing itself. And vendors smelt that from miles away.. and then they homed in with tools. Over-simplified demos were given, Influential people were influenced, buying decisions made, tools thrust on unsuspecting people... "The horror... the horror..."

But I digress. GUI tests were problematic because

• Flaky: Running tests would just fail without reason one fine day. Reissuing a test run would pass that test (the flashing lights tests anti-pattern) but could fail cause a different intermittent test failure. Trust goes down.. tests get commented.. a dangerous path to tread.
• Fragile: Vulnerable to UI/UX changes - Some test broke because someone turned a textbox into a combobox.. or worse someone just redesigned a commonly used dialog. Time to throw someone into "the hole" again.. record-n-replay/Fix all those broken tests.
• Time to develop/write: Writing UI tests = tedium. Getting them to "stabilize" takes a while. But 'we use a record-n-replay tool!'.. put a pin on that one.
• Time to execute: Don't hold your breath, these tests take a while. Waiting for windows to pop up, scraping info out of controls, etc.
• Quirky controls: There are always some automation party poopers. Third party controls that don't exhibit standard behavior / the tool simply refuses to "see" them. But the UI is already "done".. Time to call in some specialists..
• Vendor lock-in and Specialists: Our resident expert has vanished without a trace.. who can write/fix the tests? (Shrugs all around) Instant Emergency: "We surely can't swap tools now. How quickly can we hire someone who speaks ToolX ?"
• Misc dept: Handle possible failure error dialogs so that the test doesn't block or wreck the subsequent tests, test sensitive to type of OS, theme, screen resolution, etc.

"Enough!" you say. Is there any hope in this post at all?

Let's tackle them one at a time.

Fragility / UX sensitivity

What if we could extract named actions ( a set of building blocks ) that we could then use to build up our tests. Think Lego blocks (named actions) combining to become models (tests) limited only by your imagination and time.

e.g. Let's say I want to test if my (unnamed) mail client

test CanReceiveEmails
     testMailServer.Setup(DummyEmails).For(username,password) 
     mailClient.Start()
     mailClient.AuthorizeOfflineMailStoreAccess(datafile_password)
     mailClient.LoginToMailServerAs(username, password)
     mailClient.SendAndReceiveAllFolders()
     var actualEmails = mailClient.GetUnreadEmails()
     Assert.That(actualEmails.Count).Is.EqualTo(DummyEmails.Count)
     // more comprehensive checks for message content...
     mailClient.Stop()

So there, we have identified the actors (I'll call them as Drivers henceforth) in our test and the corresponding keywords/actions that we need them to offer. How did that help us - you ask?

We have removed any traces of the UI out of the test. So let's say the LoginToMailServerAs changes from a modal application window to an inline standard widget provided by the specific mail server implementation. All I need to go fix now is the implementation of the LoginToMailServerAs action and all my tests should stay unchanged.
Also now everyone can just invoke LoginToMailServerAs as a magic incantation without worrying about how it works...it just does!

Separate intent (WHAT you want to do) from implementation (HOW you're doing it): Compare that to a run-of-the-mill UI test, the above test is much much more readable. Easier to read, understand, fix/maintain.

Time To Write - it still takes time but decreases as the store of named actions grows. Every keyword/action needs to be implemented once.. write once use wherever you need it.

We've lowered the technical expertise needed to write a test. Given the "drivers":cohesive clumps of named actions, requisite tooling and a brief walkthrough of the existing "drivers", someone can discover the APIs to choreograph a specific script - a test. Focus on testing/thinking rather than automation/coding.

Vendor lock-in and Specialists

• The decline of the specialists : "That looks almost like a xUnit style test!" You're observant. Yes you could leverage whatever it is your developers are using for unit tests - this means anyone can now write a test. No more dependency on specialists, learning curves for mastering a proprietary tool, No magic-tool licenses to buy. More money to distribute among the team (that last part is still fiction.. but I'd bet you'll have a really motivated team the day after :)
• Encapsulate Tools : The tool is bounded by the box exposing the keywords. No one outside this box (the driver) knows that you're using White for instance. This makes the tool replaceable and the choice of tool a reversible decision.

But how do we implement the HOW i.e. the keywords? The Drivers themselves.

You could use an open-source library like White (or equivalent) that can launch/attach to a running instance of a GUI app, find windows/controls and poke them. (Anything that helps you implement the ControlFinder role shown later)

Flakiness 

Depends on your choice of UI controls and your Automation library. e.g. with C#/WPF applications that run on Windows, I've found White to be pretty robust. < 5% chance of White playing truant.

Enter VM/PM Tests

"That's it ??!! These are still UI Tests! What about writing all that nasty UI Automation code?" White has wrapped the nastiness within a bunch of control-wrapper types. (You could add your own too). However for special controls, you'd still need to get your hands dirty.

"But these tests still crawl!"

Beyond this, the target application has to be (re-?) structured or as the self-righteous phrase goes 'designed for testability'. Here's one idea that should work...

All of the remaining issues are due to the GUI. There are so many types of UI controls to automate. Waiting for windows and finding controls in large hierarchies takes time. What if I slice the UI out ?

e.g. let's consider the Login.. named action (which involves bringing up the login dialog, entering the username, password and clicking OK)

If we design it such that the UI is thin (devoid of any code/logic) and merely "binds"/maps to fields and methods in a backing class. This means updating the control will trigger the backing field to update and vice-versa. Doing an action like clicking a button will trigger the underlying method.

This technique has been known for some time now (Presentation Model (MVP) - Martin Fowler (2003) OR Microsoft's variation called MVVM, which leverages .Net's in-built data-binding feature to make WPF apps faster to develop).

The only thing that the UI contains is the layout, the controls and the wiring to the underlying class. (Even that can be automatic if you move into the realm of advanced MVVM - look for Rob Eisenberg's MIX talk which uses convention to auto-bind). The more important thing is that most of the code (and bugs as a corollary) has moved into a testable class - the ViewModel / PresentationModel. The whole app is basically a symphony orchestrated by multiple presenters.

So instead of fidgeting with the UI, I can now just assign desired values to corresponding properties and invoke the OK method to simulate the whole Login process. Much better - plain method calls. What if I can load the whole app from the ViewModel layer down in my test process? That'd be great.

Benefits:

• Time to develop: No need to write UI automation code. Just call existing methods and set properties that the developers have (already) created as part of the implementation. Quick, simple and easy.
• Time to execute: No more flashing windows, looking for controls and manipulating them. If you're able to load the whole app sans the UI layer within your test process, you are effectively creating a bunch of objects, toying with them and then letting the garbage collector clean them up. It's way slower than a unit test (because you're using all the real services, data stores, devices etc..) but would be faster than a traditional GUI test (Presentation Intensive tests will show a bigger gap as compared to something that spends most of its time talking to a slow hardware device. YMMV)
• Quirky anti-automation Controls: Buh-Bye! Instead of grappling with a third party tree/grid that doesn't want to be found, you can just reach into the VM/PM layer & grab a friendly in-memory collection (that it binds to) within the corresponding ViewModel/Presenter.

But wait it gets better..

• Decoupled the testers from the implementation : This means as long as you give them some key information, the testers can start writing the tests

    
    public interface ControlLocator
    {
        Window GetWindow(string title);
        T GetControl<T>(Window parentWindow, string automationId) where T : Control;
    }

    public interface ViewModelLocator
    {
        T GetViewModel<T>() where T : ViewModel;
    }

What testers need to know to implement the Drivers.

1. for UI Tests: ParentWindow + ControlType + unique ControlId 
2. VM Tests: ViewModelType + PropertyName/CommandName

• Enable Test-first: they don't have to wait till the whole thing is implemented to write the tests. e.g. With record-and-replay style of tools, you'd have to wait till the development team gives you the running application to begin test automation. Especially important for teams practicing one of the Agile methods. You could now enable the teams to move up to ATDD.

So let's do a recap, 

Current: We started at the top where teams have heavy investments in GUI Testing. These tests are work magnets : maintenance-heavy... sucking in team resources... high cost to benefit as compared to the PM/VM Tests.

Target: By identifying reusable actions with intention-revealing names, we can construct tests much faster than before and with less cost (Programmers will recognize this as the Extract Method refactoring).
Further by peeling off the UI layer, we get a scriptable interface (an API so to speak) for the target application. We can write most of the system-level tests without the UI.. most teams still like to write some UI tests just as backup.

Stretch: Finally IF a team resolves to write comprehensive unit tests (such that most bugs don't make it past the green section), uses VM tests to catch integration defects and makes every defect an opportunity to fix the process: you could STOP writing UI tests at all (James Shore is a proponent and seems to have had success with this). The time saved in UI Automation can be put to better use - exploratory testing. Not all teams will get here.. but if you make it here, you'll never want to go back. You'd be able to deliver more features per unit time.

So what do you need to do: 

• Get enough user-context to create a library of named actions; called keywords by some. Tests are written in terms of these keywords. Remember What - not how. e.g. EnterUserNameField() or ClickLogin() is bad; ask "WHY?" to chunk up and you should reach Login(username, password)
• Let testers step into the shoes of the user & shape this interface outside-in. Pair them with a good programmer to ensure you have a "Discoverable API" i.e. easier to figure out on your own given tooling support (e.g. IDE Intellisense).

For UI-less tests,

• Follow a design technique like MVP or MVVM. Minimize code in the UI.. so that it's easier to test. 
• Ensure that you do not need the UI to start an instance of your SUT/application. Have a composition-root (e.g. a Main() function where the app comes together) 
• Abstract out the User-interaction. So you can't pull a MessageBox or ShowDialog() out of thin air in the ViewModel code. You create a Role e.g. User. The production implementation of User will probably pop up dialogs. When you need to test without the UI, you replace it with a fake-object controlled by your test.

Now the preceding bullets are easier said than done for zillion-line legacy apps. For greenfield projects, I find this a very enticing alternative - there is no reason to not build it in test by test. We've crossed out most of the perils of UI Tests.

Feel free to question / enhance / criticize with objective reasons / list pros and cons.... In the words of the Human Torch: "Flame on!"

NUnit vs MSTest - 2011 Edition

I have tried to be as objective as possible. Disclaimer: NUnit user since 2005-06.

Legend:

• MSTest as an alias for the unit-testing fwk bunded with VS2010 v10.0.30319 throughout this post (although technically it is the just the runner). It's much easier to say than VisualStudio QualityTools UnitTestFramework. For NUnit, I'm using v2.5.9
• Class-Setup/Teardown - to be executed ONCE before/after ALL tests
• Test-Setup/Teardown - to be executed before/after EVERY test

Major Differences

Migrating a test suite between the two runners might be non-trivial depending on your test code. MSTest and NUnit are very different in the way they go about running a set of tests. e.g. Consider a set of 2 classes containing tests - Class A with 2 tests, B with just one. If I insert logging at some critical points.. (I am looking at the default behavior - no customization)

The A Team

Contd from the previous post..

So how do we succeed?
Lock up Enemy #1 - Accidental Complexity

Empower teams to choose

All projects / teams are not the same. Different needs, different tools. This may be difficult in BigCo especially if the tools have already been bought. But make some noise - stand up for your tools, the "users" need to channel some feedback upstream to the "purchasers".

• Explore options instead of resigning to the golden hammer. Prefer tools that don't get in your way. Ones that don't require you to learn yet another proprietary language. Ability to write extensions in your team's native language is a +. This also opens avenues for developers to assist with automation work, if required.
  
• Use existing tools instead of writing your own - they're likely to be functional, tried and tested
• Avoid putting all eggs in one basket. Keep tools/test-runners swappable by defining layers. To migrate to a different test runner, you should only need to migrate your thin tests layer which calls into an automation layer below it, which does most of the heavy lifting. More on this soon..

Collaboration
forms a reinforcing loop against Silos/Specialists. Increasing collaboration decreases opportunities for specialization, which in turn facilitates more collaboration. Of course, the reverse is also true - Silos can finish off collaboration. Only one shall survive, you just have to choose.

Outside-in / Test-first + Wishful Thinking
If you've tried the XP Practice of TDD, you'd know the liberating feeling of letting the unit tests drive the design of the production code. You fix the end-goal, you make it work, make it clean and repeat.
Starting with the test prevents any bias (arising from implementation details, existing tools at your disposal, etc.)

ATDD is the corresponding practice at the system level. However it is not an easy practice to quickly latch on to. So work towards it in baby steps
For starters, concentrate on a DONE definition + writing tests first (before implementation) from the users' perspective on a piece of paper. Make sure everyone has the same idea of DONE before you start the iteration.
As the team matures, you can even move up to ATDP (from the BDD world) where you write test before or during iteration planning & use them for estimation.

WHAT over HOW
Ensures that the test is at the right level of abstraction (the ol' forest over trees adage). It makes the tests shorter and readable. It also works beautifully to bring out the intent (as opposed to the implementation) of the test.
Specify the bare minimum ; things that are relevant to the test at hand.. all other details need to be out of sight.

Stable DSL for testing
You employ wishful thinking to imagine the ideal interface you'd like the system to expose for testing. Since the tests are another client to the system, they can also drive beneficial design changes. The tests stand-in for real users, so if the system is difficult to consume for the tests it follows that it might be for the users too. You could start with a plain C# interface to begin with and then work your way up to a custom DSL. It

• abstracts away incidental details like the GUI, underlying technology and other implementation details.
• abstracts away the tools used for automation from the tests.
  
• decouples the people interested in writing tests from the automation personnel. This allows both to play to their strengths and offers the best of both worlds. e.g. the testers could define the automation interface changes for the sprint and the developers could implement them with production-code like quality.
  
• makes it easy to write new tests with relatively little boot-up time. Writing a test then is just combining the reusable building blocks offered by the test DSL. The tests layer is a good training ground for new hires.
  

Imagine (wishful) a robot that will operate the system for you and think of the commands that you'd issue to the robot. The set of commands are your starting point.
e.g. robot.DoX(params) or robot.GetY()

Programming Skills
Automation is programming. Without good programming techniques and discipline, sustainable pace would be difficult.
This means you need to raise the bar for automation personnel and/or leverage devs. If the team lacks the skill-set required, take countermeasures... Training, get some experts onboard, etc. The average skill level of the team can also be increased by frequent pairing.

Refactoring
Your #1 weapon against complexity. Beck's 4 rules for simple design, the techniques from the Refactoring book (Martin Fowler) + the SOLID principles are a must-read. Top that off with an introductory text on programming (e.g. Clean Code - Robert Martin) and you should be good to go.

Good Naming & Discoverable Design
Taking the time to pick good names goes a long long way. Good names make it easy to find things, facilitate understanding, help zone in on a specific area to change & reduce duplication
This also helps in being able to discover the design / API using just the IDE (learn by intellisense) and programmer intuition. Choose names that are likely to be searched. Operate by the principle of least surprise (code that works as expected the first time around); Avoid hidden side-effects. Document and use team conventions to preserve consistency.

Communicate Intent / Distill the essence
This takes WhatOverHow to the next level. Explaining Why e.g. by extracting another "coarse" method to move up one level OR differentiating sets of inputs by using explanatory names. This reduces the test further to the essence - where the tests turn into readable system documentation... the kind that never gets out of date.

Learning Curve
Refactoring well and often keeps accidental complexity down to manageable levels. The supporting cast of Pairing, a discoverable design, intention-revealing code and a good testing DSL make it easy for new team members to learn the ropes.
This inhibits cargo-cult behavior and the changes made are deliberate/intentional rather than hopeful. Another source of complexity wanes.

Test Maintenance - the last frontier
Test Maintenance like complexity can be minimized not eliminated. As complexity decreases, maintenance effort reduces too.
The testDSL makes it possible to write-and-test the blocks once & use anywhere. Simple designs (no duplication, intention-revealing code, minimal classes) make maintenance much easier.

Transitively, the cost of automation goes down as well.

Let's refactor our diagram to remove the accidental nodes and edges and things get clearer now. Refactoring code is even more rewarding.

Towards better acceptance test automation...

This started out as a sketch of a Causal Loop Diagram (CLD) for bad acceptance test suites... and then it got away from me :) The black arrows indicate "+ve links" e.g. Duplication and Complexity increase/decrease together. The blue arrows ending in dots indicate "-ve links" e.g. Refactoring and Duplication work against each other. Increase in Refactoring causes decrease in duplication.
Click on it to be able to zoom in.

Automated tests != Free Lunch

Disclaimer: I'm a proponent of XP and truly believe it has made me a much better programmer. This post just aims to let readers see through the fog-of-agile caused due to data-transfer loss as it passes from person to person. Please do not misinterpret this as an anti-agile rant. I'm just saying it doesn't work out always unless you're willing to put in the effort to make the change.

Legend
What you hear… (good) the Promised Land

• What was left unsaid (bad.. or downright ugly)

You have an automated regression-safety net to make future changes. Make a change, Push a button and you will know if you broke anything. CI can provide near instant-feedback. Better confidence.

• You have 2-3X the code to maintain. If you have people who don’t care/are too busy/ not passionate about code quality and writing good tests, the tests are the first to put a chokehold on your productivity. Bad tests are as good as (or possibly worse than) having no tests. You could see yourself in a situation where a section of your team is permanently siphoned to keeping the build/tests green. This turns into a daily bottleneck. Tests need to be self-checking, thorough, readable, professionally written, independent, repeatable, concise and FAST. All this takes effort!

Documentation - the tests can be "live specs" of the application - They never get out of date like documentation.

• It takes a significant level of discipline and skill to write readable spec-tests. An essential skill :to see the What and Why without getting entangled in the How. Most teams get this wrong... without noticing it.
  
• Sidenote: The practice of ignoring failing tests is criminal (but usually not punished accordingly) and can lead to misleading specs.

Quality & Productivity: Leads to high-quality production code. Fewer Bugs. More features added / unit time (because you spend less time in debugging and manual testing)

• IF you let the tests drive/shape your design (ATDD and TDD). Client-first design is an unstated requirement.
• The quality of the code is usually a direct reflection of the people writing it. This means you need craftsmen (> 30-50% of a team) and NOT armies of cargo-cult programmers.
• If you're using automated tests exclusively for regression (or getting your 'agile badge'), you'll slowly grind to a halt. Writing tests for "untestable blobs implemented from a non-negotiable handed-down paper design" is frustrating. People can be stuck on “how do I test this?” – Usually leads to silent trade-offs & non-thorough test which will let in bugs and put you in the net negative w.r.t. productivity.

Less rework/thrashing: The dialogue / conversation (that you have with the customer to come up with the acceptance tests) makes it highly likely that you’re building the right thing..

• Assumes that the customers want to collaborate with the developers and testers. This is not often true.. Real users are sometimes real hard-to-find. Even if you manage to snag one of them, you can only procure a small slice of their time. Real users rarely want to write tests.


• If the customers give a “vision” and delegate the responsibility of mapping them to executable specs to the technical team (or worse the QA/testers), you still run the risk of “This is not what I asked for” late in the game. Regular demos may help shorten the feedback time.. but you may still waste an iteration.The magic potion here is collaboration and conversation.. the tests are just a beneficial byproduct.

Simple: Red-Green-Refactor. How hard can that be?

• Sounds simple.. but is deceptive. True OO is a minority sport. Refactoring is a skill that you need to work on. Off the job practice is mandatory.
  You may need to "hire" a good coach for an extended period (I'd say 6 months-1 release) to get the team rolling. Spot trainings/Just-in-time learning won't work for most teams.

How to get Thoughtworks' White to LogStructure

Struggled for some time getting it to work. I've been playing with Thoughtworks' open library for UI Testing for a while.

Sometimes UISpy doesn't get you to the UIElement hierarchy and you need to take out the big guns ala White's window.LogStructure to dump out the UIElement tree.
As always it's simple when you know how.

Step#1: Modify the app.config of your executable to include the sections related to White. See a sample file on White's page here.

Step#2: Add the following line to your AssemblyInfo.cs file

[assembly: log4net.Config.XmlConfigurator(ConfigFile = "Log4Net.config", Watch = true)]

Step#3: Add a file called Log4Net.config under your assembly (configure it such that it is always copied to the output folder). This contains the configuration information for Log4Net (e.g. the following config logs to console and to file named example.log in your output folder.). That's it you should now be able to see/capture window.LogStructure output from White. Godspeed !

<log4net>
  <appender name="Console" type="log4net.Appender.ConsoleAppender">
    <layout type="log4net.Layout.PatternLayout">
      <!-- Pattern to output the caller's file name and line number -->
      <conversionPattern value="%5level [%thread] (%file:%line) - %message%newline" />
    </layout>
  </appender>

  <appender name="RollingFile" type="log4net.Appender.RollingFileAppender">
    <file value="example.log" />
    <appendToFile value="true" />
    <maximumFileSize value="100KB" />
    <maxSizeRollBackups value="2" />

    <layout type="log4net.Layout.PatternLayout">
      <conversionPattern value="%level %thread %logger - %message%newline" />
    </layout>
  </appender>

  <root>
    <level value="DEBUG" />
    <appender-ref ref="Console" />
    <appender-ref ref="RollingFile" />
  </root>
</log4net>

An effective test strategy to grow applications

This is a post to summarize my understanding (as of today) of an effective (low ceremony) way to build applications.
Let's take a look at (a slightly modified version) of Mike Cohn's layered test pyramid.



The choice of a pyramid indicates the robustness/stability of a lower layer directly affects the effectiveness of the upper layer. Also the number of tests decreases as you move up. As the agile testing book says ROI is maximum at the bottom (speed of feedback over time invested) and wanes towards the top.
e.g. without robust unit-tests, DSL tests or GUI tests would catch a bunch of errors without the essential feedback needed to fix it quickly. More and more errors would make it up to the middle and top layers, where it is more time-consuming/expensive to find-n-fix. Nothing replaces well-written, quick, professional unit tests.
To prevent ambiguity and misinterpretation, let's go over each layer.

Contract Tests with Slim/Fitnesse : Running a test suite against multiple implementations of an interface

Recently I had a need for something similar to "xUnit contract tests" in Fitnesse (defined as: a test suite, which any object must satisfy so that it can be considered as a valid implementation of a Role. I got the term from one of J.B.Rainsberger's talks. )

To illustrate my point, let me cook up an example.

Imagine a contest for BoneyM impersonators. We would quiz each impersonator with a set of questions that they must answer correctly to win the contest.

i.e. we have a set of tests (questions) for each implementation of an Impersonator (Role). We also have multiple impersonators that I want to validate against the contract tests. The tests need to be dry - since they don't change for each implementation.

Testing .Net code with Cucumber and IronRuby

As promised, this is the culmination of this trilogy. The previous two posts were a jump-start on Cucumber to test Ruby code. Next I moved towards using Cucumber's plain text stories to test .Net code.Aslak Hellesoy has a wiki-post on how to do this ; however I found that you get eaten and spat out by a lot of IronRuby dragons along the way. So hold on..

1. Install the new DLR

First up install .Net Framework 4.0 (I have Beta1) - this has the new Dynamic Language Runtime (DLR) that makes things like IronRuby and IronPython possible.

2. Get the latest IronRuby release

Next we need to get IronRuby as a zip. Extract it to say d:\ironruby-0.9.0 Add the path to the bin folder to your PATH environment variable to avoiding lengthy paths. Drop to a command shell, type 'ir' to invoke the interactive IronRuby console. Type something simple to test if it works. It should.

>>>puts "Hello IronRuby"

3. Dragons ahead. Use diversion

But I hit a snag with this version of IronRuby, which had A BUG.

• So I had to get the latest (8 Sep 2009 master to be precise) version of the source from the github repository to overcome that. You can download it as a zip or use git as you prefer. I got the zip
  ironruby-ironruby-90cdda82fd60f4b7e6d7d940501c586d55954466.zip (Could have used a shorter name)


• Extract it to say d:\ir-src
  I hit a few path-too-long errors during extraction. Just keep skipping them. I think it's because of the rather large alphanumeric string which is the name of the top-level folder.


• Once extracted, navigate to D:\ir-src\ironruby-ironruby-90cdda82fd60f4b7e6d7d940501c586d55954466\Merlin\Main\Languages\Ruby
  and open Ruby.sln in VS2008 (I have Dev Edition of VSTS.. Although it should build in VS Express Editions as per the docs - I couldn't get the solution to open in it.)
  Build Solution. At the end of it you should find the built binaries in
  D:\ir-src\ironruby-ironruby-90cdda82fd60f4b7e6d7d940501c586d55954466\Merlin\Main\bin\Debug


• Rename that directory with the huge name to something like "ir".


• Copy all built binaries and overwrite the ones from the 0.9.0 release's bin folder i.e. D:\ironruby-0.9.0\bin.



• Open ir.exe.config in the same folder and update the Library paths element to the proper paths to folders within your ir-src folder. 'Handle with extreme care' or you'll lose hours chasing error messages. It should read
  Path#1 - D:\ir-src\ir\Merlin\Main\Languages\ (NOT D:\ir-src\ir\Merlin\External.LCA_RESTRICTED\Languages)
  Path#2 & #3 - D:\ir-src\ir\Merlin\External.LCA_RESTRICTED
  
  My xml looks like
  

  <set language="Ruby" option="LibraryPaths" 
  value="D:\ir-src\ir\Merlin\Main\Languages\Ruby\libs\;D:\ir-src\ir\Merlin\External.LCA_RESTRICTED
  \Languages\Ruby\redist-libs\ruby\site_ruby\1.8\;D:\ir-src\ir\Merlin\External.LCA_RESTRICTED\Lang
  uages\Ruby\redist-libs\ruby\1.8\" />

4. Wrapper script ICucumber

Lastly we need a wrapper script to invoke cucumber with Ironruby. Create a file called icucumber.bat under your Ruby bin folder i.e. D:\Ruby\bin\icucumber.bat with the following text.

REM Update with appropriate values for GEM_PATH, Ruby bin and the path to ir.exe
@ECHO OFF
REM This is to tell IronRuby where to find gems.
SET GEM_PATH=d:\ruby\lib\ruby\gems\1.8
"D:\ironruby-0.9.0\bin\ir.exe" -D -X:ExceptionDetail "d:\ruby\bin\cucumber" %* 

5. Back to our example from the previous two posts.
The .feature file stays unchanged.
Delete c:\cukes\dot_net_features\support\BowlingGame.rb ; since we're going to implement the same in C# this time around as
c:\cukes\dot_net_features\support\BowlingGame.cs

namespace CukesDemo
{
   public class BowlingGame
   {
      public void roll(int pins_knocked_down)
      {
        Score += pins_knocked_down;
      }

      public int Score
      {
         get; set;
      }
      public bool Over
      {
         get { return false; }
      }
   }
}

Also create a small batch file to compile it to a DLL (Assumes csharp compiler is on the PATH).
c:\cukes\dot_net_features\support\Compile.bat

IF EXIST bin GOTO COMPILE
MKDIR bin
:COMPILE
csc /t:library /out:bin/BowlingGame.dll bowling_game.cs

Finally back to our step definitions to check the glue. Four changes needed - explained in comments.
cukes\dot_net_features\step_definitions\bowling_game_steps.rb

# CHANGE 1 : Add bin folder to load-path
$:.unshift(File.dirname(__FILE__) + '/../support/bin')   
# CHANGE 2 : Get BowlingGame.dll
require 'BowlingGame'      

Given /^I am starting a new game$/ do
   # CHANGE 3 : Use Namespace::ClassName.new  
   @game = CukesDemo::BowlingGame.new   
end

When /^I roll (\d+) gutter balls$/ do |count|
  count.to_i.times{
    @game.roll(0)
  }
end

Then /^the score should be (\d+)$/ do |expected_score|
  @game.score.should == expected_score.to_i
end

Then /^the game should be over$/ do
  # CHANGE 4 :  be_over passes even if Over returns false. Don't know what is the equiv of over? 

in .Net
  #~ @game.should be_over == true 
  @game.over.should == true
end

When /^my rolls are (.*)$/ do |rolls|
  rolls.split(' ').each{|roll|
    @game.roll(roll.to_i)
  }
end

Now for the grand finale, run Cucumber to verify our .Net DLL via IronRuby !!!


HTH

Scenario Outlines and Tagging in Cucumber

Post#2 in this trilogy.

Scenario Outline

Scenario tables are similar to Fit's ColumnFixture and NUnit's RowTest. You run the same scenario with different inputs each time.
Let's go back to our plain text feature and add the following.

Scenario Outline: score should be as per the std rules
  Given I am starting a new game
  When my rolls are <rolls>
  Then the score should be <score>

 Scenarios: lets go bowling
 | rolls   | score |
 |5 2      | 7     |
 |5 5 5    | 15    |

The key things to remember here are the 'Scenario Outline:' marker to indicate that it is an outline. We then have placeholders within angular brackets. The Outline is then followed by one or more tables identified by the marker 'Scenarios:'
The next line should contain column headers which correspond with the placeholders in the outline. Cucumber would substitute the values to run each row against the Outline.

SARC and it should fail. Let's go fix that up in bowling_game.rb

class BowlingGame
 attr_reader :score
 def initialize
   @score = 0
 end
 def roll( pins_knocked_down)
   @score += pins_knocked_down
 end
 def score
   @score
 end
 def over?
   true
 end
end

SARC. Now you should see this nice listing.


Tagging

You can tag a scenario (or a feature) with one (or more) tags.

@important, @fast
Feature: Bowling Game Score Calculation
 In order to let the player know his score

By default, cucumber runs all .feature files in the features subfolder. You can group features in different subfolders as well e.g. I can define a new.feature within a new_features subfolder, which can then be run with

>cucumber new_features

Overtime, it can get crowded with lots of subfolders. But tagging is here to save the day..
To run only features/scenarios marked 'fast'

>cucumber --tags @fast

To run features/scenarios that are not marked 'fast'

>cucumber --tags ~@fast

So that's another way to quickly sort out your tests.

Language Support

The next thing you'd probably want to know (just in case you need it) is that cucumber speaks multiple languages. You can write your .feature file in any language - provided that the necessary entries are made in the resource file - languages.yml.

Next post - getting cucumber to test a .Net app with some help from IronRuby. Piqued?


Resources on BDD / Cucumber (although I felt the majority were too entwined with Rails (e.g. WebRat for testing Web Apps via free step definitions) but then that very well could be the major user-base for cucumber right now.)

• ScreenCast by Aslak Hellesoy Executable User Stories with RSpec and BDD


• ScreenCast by Ben Mabey Outside in development with Cucumber


• Blogpost series by Dave Astels Cucumber: More advanced


• RailsCast episodes by Ryan Bates Beginning with Cucumber

Green in 30 mins : Getting Started with Cucumber (a test framework for BDD)

Prologue:
BDD is TDD with an outside-in (top-down) bent.
The way to tackle any new feature request is to ask Why 5 times? An you should arrive at one of Protect Revenue, Increase Revenue or Manage Cost. If not, chances are you’re building something that is not needed.
A feature can be summarized in a few lines as (ala Mike Cohn’s user story format)

As a <role>
I want <feature>
So that <value>

Dan North began this journey with JBehave.
It evolved over time into a Ruby gem - RSpec courtesy David Chelimsky n co. RSpec consisted of two parts - example runner and a story runner. The story runner runs features (think big cog) written up as plain text files which can be ‘executed’ (via some developer-added connecting-glue code in a separate location). The example runner (think small cog) is a bunch of module specs that make the feature happen. (think xUnit for BDD).
The RSpec story runner has now been simplified/improved and become Cucumber – a really neat little tool by Aslak Hellesoy. There are others too who have contributed to this movement like Dave Astels, et. all


You can still use RSpec example runner in tango with Cucumber or you could stick with xUnit under the hood instead of RSpec.
Enough prologue for today.

Now I've spent 3-4 days chasing hyperlinks and watching screencasts (links at the end of Post#2 in this series), this post should give you a rolling start on cucumber.

Step1# Installation

You need a few Ruby Gems to get rolling. (RSpec & Win32Console not mandatory)

gem install rspec
gem install cucumber
gem install win32console

I have rspec (1.2.8) | cucumber (0.3.99) | win32console (1.2.0). Try ‘cucumber –help’ to verify this step. Win32console is for color output on Windows.

Step2# Lets go bowling

I’ll take the popular ‘Score a bowling game’ TDD Kata as an example.
So we begin with a feature. (I’ll skip the pop-the-why-stack ; this feature falls into the ‘protect revenue’ category). Find a nice empty directory let’s say c:\cukes. Create a features subdirectory within it.
We create a new file : features\bowling_score.feature and type this in. This is called the feature narrative – it’s just a descriptive block of text. However the form is slightly different the context/value clause rises to the top with the prefix In order to <business value>. Spotlight on business value!
Note: Indentation matters! Spaces preferred.

Feature: Bowling Game Score Calculation
  In order to let the player know his score
  As a CEO of TheBowlingInc
  I want to calculate the score at the end of a bowling fame

Next we Save And Run Cucumber from the c:\cukes directory. This step I hereby alias to SARC. This outputs

c:\cukes>cucumber
Feature: Bowling Game Score Calculation
  In order to let the player know his score
  As a CEO of TheBowlingInc
  I want to calculate the score at the end of a bowling fame

0 scenarios
0 steps
0m0.000s

So now that prompts us : we need Scenarios. A Feature may contain multiple scenarios (which collectively validate the feature).
A Scenario is executable. A Scenario takes the form

Scenario: <text description>
  Given <context>
  When <action>
  Then <user-visible outcome> 

So we expand our feature like this.

Feature: Bowling Game Score Calculation
  In order to let the player know his score
  As a CEO of TheBowlingInc
  I want to calculate the score at the end of a bowling fame

  Scenario: All Gutters score 0
     Given I am starting a new game
     When I roll 20 gutter balls
     Then the score should be 0
     And the game should be over

In addition you can write And <step> to have multiple steps – the above example is equivalent to 2 Then clauses. You can use it under Given/When too. Save and run cucumber again. (or SARC from here on)



For some reason the snippets don’t show up. So we play a little trick. Create a subfolder under features called step_definitions. Create a new blank file called bowling_game_steps.rb in it. Run cucumber again. Now you should see some snippets. Copy them from the console and paste into the blank file. SARC. You should now see that the first step is shown as a TODO and the rest are skipped (and in a different step-color to boot)

The first snippet looks like

Given /^I am starting a new game$/ do
pending
end

We need to define a “step” – specify the action to be taken i.e. when cucumber encounters a matching Given clause for the regex, what action should it take? You define that as the content of the block. Let’s say we want to create a new instance of a game. So replace `pending` with
`@game = BowlingGame.new`
SARC and now we have a familiar color – Red (of the Red-Green-Refactor fame). We’re notified that our step has failed.




We have no class called Bowling Game yet. Create a new subfolder under features called support to house our app classes. Create a new file in there ; features\support\bowling_game.cs

class BowlingGame; end

SARC and we’re green…. partially.. better than red. Cucumber now points us to the next step.



I’ll define all the steps like this.. while you take a breather. There. The updated version

Given /^I am starting a new game$/ do
  @game = BowlingGame.new
end

When /^I roll (\d+) gutter balls$/ do |count|
  count.to_i.times{
    @game.roll(0)
  }
end

Then /^the score should be (\d+)$/ do |expected_score|
  @game.score.should == expected_score.to_i
end

Then /^the game should be over$/ do
  @game.should be_over
end

When /^my rolls are (.*)$/ do |rolls|
  rolls.split(' ').each{|roll|
    @game.roll(roll.to_i)
  }
end

NOTE:

1. The second step is an example of a parameterized step. If the regex in the step definition contains groups, the matched contents are passed in as parameters to the block for the step. Parameter passed in are strings so you need to convert it to the right type before use. Standard regexp rules apply for matching groups. So now the step can match both ‘When I roll 5 gutter balls’ and When I roll 20 gutter balls’. You also see parameters highlighted distinctly in cucumber output. Cool!
2. Then steps use a new should method which is added to all objects. You can read more about such helper methods here and here
3. Cucumber tells after each step what needs to be done next. This rhythm is similar to the TDD approach.

Here’s the updated bowling_game.rb with ‘the simplest thing that could possibly work’

class BowlingGame
 def roll( pins_knocked_down)
 end
 def score
   0
 end
 def over?
   true
 end
end

SARC this time we specify the no source
option as a CL argument

 >cucumber –s 

And we’ve reached the promised land - plain text file story/features that can be verified automatically at the push of a button. That's pretty cool.

Just to reiterate the folder structure.




In the next post, we move up the learning curve with Scenario tables and tagging.

NUnit RowTest Extension : Running the same test with different inputs

Update 2010-03-10: The following extension has now been superseded by the TestCase attribute which is now a part of the NUnit core (v2.5 and later). (nunit.framework.dll)

For an equivalent code sample, see the end of the post.

End of Update

This extension allows you an elegant way of handing the scenario where you need to run with different sets of input values. Usually tests shouldn't take any inputs... (the test runner doesn't know what inputs to supply the test with).

[Test(Description="PlainVanilla")]
public void TestBasicMath()
{
Assert.AreEqual(1, 1, "Someone broke mathematics");
}

But then there are always 'exceptions to the rule'. For example I'm writing a class called the Tokenizer that reads tokens from an input string. So I give it "10 + 15", the first token returned by the class should be the number 10.
Now I need to exercise the test code block below with different inputs for sInput like "10", " 10 + 15"

Tokenizer t = new Tokenizer(sInput);
Assert.AreEqual( 2, t.GetNextToken() );

Now back in the old days, you'd need to write a test case for each possible data value. Now with Andreas Schlapsi's RowTest Extension which is bundled with NUnit, things are much simpler.
Prerequisites:

• Needs NUnit 2.4.7 or later. I 'm using NUnit 2.4.8 for .Net 2.0. Get it here as always
  
• Add a reference to the nunit.framework.extensions assembly (in addition to the usual nunit.framework to your test project

using NUnit.Framework;
using NUnit.Framework.Extensions;

namespace TestDynCalc
{
[TestFixture]
public class TestTokenizer
{

  [RowTest]
  [Row("10 + 15")]
  [Row("10")]
  [Row(" 10 +15", TestName = "WhiteSpaceBeforeFirstNumber")]
  [Row("+10+15", Description = "Read number from +10+15", ExceptionMessage = "No number found at beginning of input stream!", ExpectedException = typeof(ArgumentException))]
  public void ReadNumberFromInputString(string sInput)
  {
      Tokenizer t = new Tokenizer(sInput);
      Assert.AreEqual( 2, t.GetNextToken() );
  }

  [Test(Description="PlainVanilla")]
  public void TestBasicMath()
  {
      Assert.AreEqual(1, 1, "Someone broke mathematics");
  }
}
}

Whoa! Let me step through all that. The using directives are self explanatory.

1. The RowTest attribute (instead of Test) over a NUnit test method allows you to parameterize the test method and gets the ball rolling.
   
2. Next for every unique set of inputs, you need to run this test with, you add a Row attribute and specify the inputs as constructor arguments. (The extension is vocal about any mismatch between number of test method parameters and the number of inputs you supply. )
3. The Row Test also has some named parameters

• TestName : Lets you specify a descriptive name for the specific 'sub-test'. See how the last child node of the RowTest has a different name in the the attached GUI Screenshot below.
• Description: This seems to be broken. It's a NUnit feature.. allowing you to tag a test with some comments that will show up when you bring up Properties for the test case. (Right click > Properties)
• ExpectedException, ExceptionMessage: Ideally I'd like this as a different test case. However you have the option to mark a set-of-inputs to indicate that 'this should cause this type of Exception with the following message'. See last Row attribute.

This is how the NUnit GUI renders a RowTest. Quite nice. (Of course, You should choose better names for your tests :) Each Row attribute is rendered as sub-node of the test with the relevant name and all the input params specified in brackets (comma seperated in case of multiple params).




Update

Moving from RowTest to TestCase: There are no big changes with using the new 2.5 TestCase attribute. You don't need an explicit RowTest attribute. Replace each Row attribute with a TestCase attribute.
The Exception properties have been renamed (inline with the 2.5 ExpectedException revamp. So check the docs on ExpectedException if you can't get something to work).. but easy to figure out via Intellisense.
Another improvement is an explicit Result property, which as you can guess would be used to verify the output of your test case. Before TestCase, you had to pass in another parameter in the RowTest named expectedOutput and take care to use it only for Asserting at the end of the test.

       [TestCase("10 + 15")]
       [TestCase("10")]
       [TestCase(" 10 +15", TestName = "WhiteSpaceBeforeFirstNumber")]
       [TestCase("+10+15", Description = "Read number from +10+15", ExpectedMessage = "No number found at beginning of input stream!", ExpectedException = typeof(ArgumentException))]
       public void ReadNumberFromInputString(string sInput)
       {
           Tokenizer t = new Tokenizer(sInput);
           Assert.AreEqual(2, t.GetNextToken());
       }

Getting NUnit to go all STA

As I was trying to help out someone who had trouble with writing a unit test for a data bound WPF Window.. first of all I had to ref WindowsBase, PresentationCore and PresentationFramework and then I ran into a curt warning from WPF when you're trying to instantiate a WPF window in a NUnit test case.. (Grumble Grumble... I hate UI in unit tests)
TestProj.TestBindings.TestTextBoxBinding:System.InvalidOperationException : The calling thread must be STA, because many UI components require this.
What is STA? Something you only wish 'stays out of the way'.
Interesting.. this means... its not running in STA. Elementary. Now how do I get NUnit to go STA. It was a long arduous road.
First up if you have NUnit 2.5 or above, I believe its as easy as this.

[Test, RequiresMTA]
public void MTA()
{
   Assert.AreEqual(ApartmentState.MTA, Thread.CurrentThread.ApartmentState);
}

[Test, RequiresSTA]
public void STA()
{
   Assert.AreEqual(ApartmentState.STA, Thread.CurrentThread.ApartmentState);
}

If not, which is where I was. It's slightly more complicated.
First go to

C:\Program Files\NUnit 2.4.8\bin

Look for a file called NUnitTests.config. Next copy this over to your test dll folder which has TestProj.nunit (or make one.. you'll need it). Rename the copy of the file to 'TestProj.config'.. Open it up in an editor.. Time for adding bringing some more XML into the world.

<configuration>
  <configSections>
    <sectionGroup name="NUnit">
      <section name="TestRunner" type="System.Configuration.NameValueSectionHandler"/>
    </sectionGroup>
  </configSections>

  <NUnit>
    <TestRunner>
      <add key="ApartmentState" value="STA"/>
    </TestRunner>
  </NUnit>
  <!-- all the other fluff stays -->
</configuration>

That's it, fire up NUnit.. if all goes well, you'll be running in STA and WPF will 'show itself'.

ShowMeTheMoney-16 - Tallying up the expenses

Iteration#2 Day 3/10

Some TODOs I have jotted down.

• place all view strings (displayed to the user) under the control of ApplicationHelper::ResourceManager
• inflow/list with the table style similar to outflow/list
• credit and expense rows are listed in order of ID. Ordering by entry date is what is required ( Found this out while I was actually using the app.. Real users! nothing comes close to them for feedback)
  

For the last item – lets take expense first (Credit should be identical), we add another record to the fixtures file, but out of order – make the year 2007 as shown below. We need to update some existing tests to update @fixtureRecordCount to 3. Done!

file: /test/fixtures/expenses.yml

rent_expense_entry:
id: 1
created_at: 2008-01-19 00:00:00
description: rent
amount: 10000
apparel_expense_entry:
id: 2
created_at: 2008-02-20 00:00:00
description: A pair of Jeans
amount: 1000.50
movies_expense_entry:
id: 3
created_at: 2007-12-20 00:00:00
description: Taare Zameen Par
amount: 300

Next, we need to update the controller test to check if the records are displayed in order. For that we inspect the controller’s @expenses variable and assert that they are retrieved order by date i.e. 3 – 1 - 2

def test_list
  get :list
  assert_response :success
  assert_template 'list'
  assert_not_nil assigns(:expenses)
  assert_equal(@fixtureRecordCount, assigns(:expenses).size )
  assert_equal(3, assigns(:expenses)[0].id, “not ordered by date”)
  assert_equal(1, assigns(:expenses)[1].id)
  assert_equal(2, assigns(:expenses)[2].id)
end

Run the tests to see if it fails. It does. That validates our test. Next some research on how to implement this, how do we tell paginate to order the records by “created_at” ? Search search.. online... can't find it. Once again the book comes to the rescue, Pg 352, Section 17.7 Hurray!

def list
  @expense_pages, @expenses = paginate :expenses, :order=>'created_at', :per_page => 10
end

The :order key-value pair is all we need. It’s really that simple!!! I even tried out via the browser – It just works!! I LOOVVEEE Rails!!!
Do it again for the credits page. TODOs are done! Quick run of all tests. Green!

Alrighthy! now we got to build ourselves a balance tracker.

file: /test/unit/balance_tracker_test.rb

require File.dirname(__FILE__) + '/../test_helper'

class BalanceTrackerTest < Test::Unit::TestCase
fixtures :credits, :expenses
def test_balance
  assert_in_delta(19700, BalanceTracker.balance, 0.01 )

  Credit.create(:description=>"D", :amount=>300.50)
  assert_in_delta(20000.5, BalanceTracker.balance, 0.01 )

  Expense.create(:description=>"E", :amount=>10000)
  assert_in_delta(10000.5, BalanceTracker.balance, 0.01 ) 
end
end

file: /app/models/balance_tracker.rb

class BalanceTracker
def BalanceTracker.balance
  models = Credit.find_by_sql("select sum(amount) as totalAmount from Credits");
  fTotalCredit = models[0].attributes["totalAmount"].to_f

  models = Expense.find_by_sql("select sum(amount) as totalAmount from Expenses");
  return fTotalCredit - models[0].attributes["totalAmount"].to_f;
end
end

Done. Now we have everything to go and work on our acceptance test fixtures to make the actionfixture table and second rowfixture table pass.

file: /AcceptanceTests/track_balance.rb

require File.dirname(__FILE__) + '/../test/test_helper'
require 'inflow_controller'
require 'fit/fixture'

class CreditsHelper < Test::Unit::TestCase
def setup
  Credit.delete_all
  @controller = InflowController.new
  @request = ActionController::TestRequest.new
  @response = ActionController::TestResponse.new
end
def test_dummy
end
end
class ExpensesHelper < Test::Unit::TestCase
def setup
  Expense.delete_all
  @controller = OutflowController.new
  @request = ActionController::TestRequest.new
  @response = ActionController::TestResponse.new
end
def test_dummy
end
end
module AcceptanceTests
class TrackBalance < Fit::Fixture

  #enter attributes
  attr_accessor :description, :amount, :select_entry

  def initialize
    @creditHelper = CreditsHelper.new "test_dummy"
    @creditHelper.setup
 
    @expenseHelper = ExpensesHelper.new "test_dummy"
    @expenseHelper.setup
    super
  end

  #press methods
  def add_credit_entry
    @creditHelper.get :new
    @creditHelper.post :create, :credit => {:description => @description, :amount => @amount}
  end
  def delete_credit_entry
    @creditHelper.post :destroy, :id => getID_ForSelectedEntry(Credit)
  end
  def add_expense_entry
    @expenseHelper.get :new
    @expenseHelper.post :create, :expense => {:description => @description, :amount => @amount}
  end
  def delete_expense_entry
    @expenseHelper.post :destroy, :id => getID_ForSelectedEntry(Expense)
  end 
  #check methods
  def current_balance
    BalanceTracker.balance
  end

  private
  def getID_ForSelectedEntry model
    obSelectedEntry = model.find(:all, :order => "id")[@select_entry.to_i - 1]
    obSelectedEntry.id
  end
end
end

file: /AcceptanceTests/get_inflow_records.rb

puts File.dirname(__FILE__) + '\get_inflow_records'
require File.dirname(__FILE__) + '\get_inflow_records'
require 'fit/row_fixture'
module AcceptanceTests
class GetOutflowRecords < Fit::RowFixture
  def query
    recordset = []
    Expense.find(:all, :order=>'created_at').each{|x|
      recordset << InflowRecord.new(x.created_at, x.description, x.amount)
    }
  end
  def get_target_class
    InflowRecord
  end
end
end

Our Reward !!!