Showing posts with label NUnit. Show all posts
Showing posts with label NUnit. Show all posts

NUnit vs MSTest - 2011 Edition

I have tried to be as objective as possible. Disclaimer: NUnit user since 2005-06.

Legend:
  • MSTest as an alias for the unit-testing fwk bunded with VS2010 v10.0.30319 throughout this post (although technically it is the just the runner). It's much easier to say than VisualStudio QualityTools UnitTestFramework. For NUnit, I'm using v2.5.9
  • Class-Setup/Teardown - to be executed ONCE before/after ALL tests
  • Test-Setup/Teardown - to be executed before/after EVERY test

Major Differences
Migrating a test suite between the two runners might be non-trivial depending on your test code. MSTest and NUnit are very different in the way they go about running a set of tests. e.g. Consider a set of 2 classes containing tests - Class A with 2 tests, B with just one. If I insert logging at some critical points.. (I am looking at the default behavior - no customization)

What do I use for acceptance tests (ATDD) : NUnit vs Fitnesse ?

Lately I've had some teams in my getting-agile ecosystem debating on which tool to use for acceptance testing. Now my first response is always "It depends" (You really can't go wrong with that. :) ).

Notation: I'm gonna use
  • NUnit when I actually mean any member of the xUnit family of unit-testing frameworks. 
  • Fit/Fitnesse stands for itself and now Slim/Fitnesse.
  • Customer means the XP's Customer role or Scrum's Product Owner role.

Fit-Fitnesse was the de-facto tool for automated acceptance tests till sometime ago.. Around 2009, there were strong murmurs that Fitnesse isn't working that well (read as some teams weren't executing very well and Fitnesse was part of their toolkit ).

So what does it "depend" on ??

NUnit ExpectedException Bug in 2.4.7

"Bug In NUnit ? It can't be!" Exactly my words ; However it turned out that the behavior is due to a newer NUnit runner (Gui v 2.5.3) running a test fixture referencing an older version of NUnit.Framework.dll (v2.4.7)

Try this in a NUnit test fixture referencing NUnit.Framework v 2.4.7


// 2.4.7 - test passes even if an exception of wrong type is thrown
[ExpectedException(ExceptionType=typeof(FileNotFoundException))]
[Test]
public void Test_ExpectedException()
{
throw new ArgumentException();
}


I found that the test passes if "any" exception is thrown. It doesn't have to match the specified type. A false positive!

To fix this, upgrade your NUnit version to 2.5.2. You'd also need to fix a compile error. 'ExceptionType' property has now become ExpectedException.


//2.5.2 'works as expected'
[ExpectedException(ExpectedException=typeof(FileNotFoundException))]
[Test]
public void Test_ExpectedException()
{
throw new ArgumentException();
}



You live... you learn.

NUnit RowTest Extension : Running the same test with different inputs

Update 2010-03-10: The following extension has now been superseded by the TestCase attribute which is now a part of the NUnit core (v2.5 and later). (nunit.framework.dll)

For an equivalent code sample, see the end of the post.

End of Update

This extension allows you an elegant way of handing the scenario where you need to run with different sets of input values. Usually tests shouldn't take any inputs... (the test runner doesn't know what inputs to supply the test with).




[Test(Description="PlainVanilla")]
public void TestBasicMath()
{
Assert.AreEqual(1, 1, "Someone broke mathematics");
}


But then there are always 'exceptions to the rule'. For example I'm writing a class called the Tokenizer that reads tokens from an input string. So I give it "10 + 15", the first token returned by the class should be the number 10.
Now I need to exercise the test code block below with different inputs for sInput like "10", " 10 + 15"


Tokenizer t = new Tokenizer(sInput);
Assert.AreEqual( 2, t.GetNextToken() );



Now back in the old days, you'd need to write a test case for each possible data value. Now with Andreas Schlapsi's RowTest Extension which is bundled with NUnit, things are much simpler.
Prerequisites:
  • Needs NUnit 2.4.7 or later. I 'm using NUnit 2.4.8 for .Net 2.0. Get it here as always
  • Add a reference to the nunit.framework.extensions assembly (in addition to the usual nunit.framework to your test project



using NUnit.Framework;
using NUnit.Framework.Extensions;

namespace TestDynCalc
{
[TestFixture]
public class TestTokenizer
{

[RowTest]
[Row("10 + 15")]
[Row("10")]
[Row(" 10 +15", TestName = "WhiteSpaceBeforeFirstNumber")]
[Row("+10+15", Description = "Read number from +10+15", ExceptionMessage = "No number found at beginning of input stream!", ExpectedException = typeof(ArgumentException))]
public void ReadNumberFromInputString(string sInput)
{
Tokenizer t = new Tokenizer(sInput);
Assert.AreEqual( 2, t.GetNextToken() );
}

[Test(Description="PlainVanilla")]
public void TestBasicMath()
{
Assert.AreEqual(1, 1, "Someone broke mathematics");
}
}
}
Whoa! Let me step through all that. The using directives are self explanatory.
  1. The RowTest attribute (instead of Test) over a NUnit test method allows you to parameterize the test method and gets the ball rolling.
  2. Next for every unique set of inputs, you need to run this test with, you add a Row attribute and specify the inputs as constructor arguments. (The extension is vocal about any mismatch between number of test method parameters and the number of inputs you supply. )
  3. The Row Test also has some named parameters
  • TestName : Lets you specify a descriptive name for the specific 'sub-test'. See how the last child node of the RowTest has a different name in the the attached GUI Screenshot below.
  • Description: This seems to be broken. It's a NUnit feature.. allowing you to tag a test with some comments that will show up when you bring up Properties for the test case. (Right click > Properties)
  • ExpectedException, ExceptionMessage: Ideally I'd like this as a different test case. However you have the option to mark a set-of-inputs to indicate that 'this should cause this type of Exception with the following message'. See last Row attribute.
This is how the NUnit GUI renders a RowTest. Quite nice. (Of course, You should choose better names for your tests :) Each Row attribute is rendered as sub-node of the test with the relevant name and all the input params specified in brackets (comma seperated in case of multiple params).




Update

Moving from RowTest to TestCase: There are no big changes with using the new 2.5 TestCase attribute. You don't need an explicit RowTest attribute. Replace each Row attribute with a TestCase attribute.
The Exception properties have been renamed (inline with the 2.5 ExpectedException revamp. So check the docs on ExpectedException if you can't get something to work).. but easy to figure out via Intellisense.
Another improvement is an explicit Result property, which as you can guess would be used to verify the output of your test case. Before TestCase, you had to pass in another parameter in the RowTest named expectedOutput and take care to use it only for Asserting at the end of the test.



[TestCase("10 + 15")]
[TestCase("10")]
[TestCase(" 10 +15", TestName = "WhiteSpaceBeforeFirstNumber")]
[TestCase("+10+15", Description = "Read number from +10+15", ExpectedMessage = "No number found at beginning of input stream!", ExpectedException = typeof(ArgumentException))]
public void ReadNumberFromInputString(string sInput)
{
Tokenizer t = new Tokenizer(sInput);
Assert.AreEqual(2, t.GetNextToken());
}

Getting NUnit to go all STA

As I was trying to help out someone who had trouble with writing a unit test for a data bound WPF Window.. first of all I had to ref WindowsBase, PresentationCore and PresentationFramework and then I ran into a curt warning from WPF when you're trying to instantiate a WPF window in a NUnit test case.. (Grumble Grumble... I hate UI in unit tests)
TestProj.TestBindings.TestTextBoxBinding:System.InvalidOperationException : The calling thread must be STA, because many UI components require this.
What is STA? Something you only wish 'stays out of the way'.
Interesting.. this means... its not running in STA. Elementary. Now how do I get NUnit to go STA. It was a long arduous road.
First up if you have NUnit 2.5 or above,
I believe its as easy as this.



[Test, RequiresMTA]
public void MTA()
{
Assert.AreEqual(ApartmentState.MTA, Thread.CurrentThread.ApartmentState);
}

[Test, RequiresSTA]
public void STA()
{
Assert.AreEqual(ApartmentState.STA, Thread.CurrentThread.ApartmentState);
}


If not, which is where I was. It's slightly more complicated.
First go to
C:\Program Files\NUnit 2.4.8\bin
Look for a file called NUnitTests.config. Next copy this over to your test dll folder which has TestProj.nunit (or make one.. you'll need it). Rename the copy of the file to 'TestProj.config'.. Open it up in an editor.. Time for adding bringing some more XML into the world.


<configuration>
<configSections>
<sectionGroup name="NUnit">
<section name="TestRunner" type="System.Configuration.NameValueSectionHandler"/>
</sectionGroup>
</configSections>

<NUnit>
<TestRunner>
<add key="ApartmentState" value="STA"/>
</TestRunner>
</NUnit>
<!-- all the other fluff stays -->
</configuration>


That's it, fire up NUnit.. if all goes well, you'll be running in STA and WPF will 'show itself'.