What do I use for acceptance tests (ATDD) : NUnit vs Fitnesse ?

Lately I've had some teams in my getting-agile ecosystem debating on which tool to use for acceptance testing. Now my first response is always "It depends" (You really can't go wrong with that. :) ).

Notation: I'm gonna use
  • NUnit when I actually mean any member of the xUnit family of unit-testing frameworks. 
  • Fit/Fitnesse stands for itself and now Slim/Fitnesse.
  • Customer means the XP's Customer role or Scrum's Product Owner role.

Fit-Fitnesse was the de-facto tool for automated acceptance tests till sometime ago.. Around 2009, there were strong murmurs that Fitnesse isn't working that well (read as some teams weren't executing very well and Fitnesse was part of their toolkit ).

So what does it "depend" on ??


Customer Collaboration: Probability of Non-developers reading/modifying individual tests
  • Is your customer willing/able to collaborate with the team over a piece of paper to extract examples/scenarios for a particular use-case ? 
  • Are customers interested in creating truth-tables/spreadsheets of 'how this rule should work' or story descriptions of 'how the ideal system should work'? 
  • Do Customers / QA-personnel want to add/modify tests (without dev support) to improve their understanding / the system? Is the team using a shared medium to come up with more tests? 
If yes, choose Fit-Fitnesse... that is what it was created for. Collaboration.
Alternate Scenario: Developers are given a bunch of requirements and are responsible for conversion to executable tests. Customers/Proxies are available over email to answer questions during ear-marked hours.  Test-list generation is done offline (read without an active customer) followed by a verbal/email review. The transformation to executable tests has no participation from the customer. Acceptance tests are being used primarily for closing stories (definition of done) and regression. If a Pass/Fail result is all you need, NUnit might be a better choice.

Positive Pressure on Test readability
As-intended use of Fitnesse, causes test authors to express intent before implementation.. a wiki page is like a blank of paper - wishful thinking ahead! This causes the pages to read like scripts or a sequence of intent-revealing steps with intermediate checks/asserts. If the team is struggling to create readable-n-maintainable tests, use fitnesse as a guardrail.
You could do the exact same thing with NUnit. You just need to see the world from the users' perspective and a professional programmer's instincts for refactoring and simplicity. A blank test is as good as a blank wiki page. Flesh out the step-methods later. A common trait (of developers) is to jump the gun to implementation details at great harm to readability. Curb your enthusiasm and no reason why it shouldn't work

Deployment Footprint
Fitnesse offers a Web interface to easily view/modify/run tests from the comfort of your favourite browser. Point to your fitnesse server URL and voila! Zero install. You can even do 'What if' analysis without much assistance. (e.g. What if number of concurrent clients is raised from 100 to 150)
NUnit can't do it on its own. You would need a CI System to offer the web dashboard to execute/view results. Viewing the internals of a test is still difficult. You would need an IDE and 'coding skills' to modify the tests for any what-if analysis. Of course, if the development team is the only one interacting with the tests, this is a moot point.

Homesickness
If dev >> testers, then it might be easier to teach the testers NUnit - in crunch time, free devs can pitch in to help the test teams get to the finish line. Most developers already know NUnit; Getting all of them to learn another tool for acceptance tests could be a drag (atleast in the short term). Personally (Disclaimer: more dev/NUnit hours than QA/Fitnesse) I find that Fitnesse has a longer learning curve than NUnit (different table styles, reusing sections of a page, etc) before you can use it effectively. NUnit also wins hands down in the refactoring department (moving HTML around can get tiresome very soon to a person used to a statically typed language + a good IDE). Stuff like 'attaching a debugger to a failing test', 'how do I get my pages into source control', 'feedback on progress of test execution' with Fitnesse might restrain/annoy speed-addicted xUnit-developers. NUnit also has a richer assertion library, which I missed during my time in Fitnesse.
        On the other hand, if you have customers and testers who can 'converse in fitnesse' and have a few projects under their belt, don't get in their way. Testers can get homesick too. Go with Fitnesse and pair Devs with Testers to diffuse Fitnesse in the team.


End-game
Both are tools - it is how your team executes with these tools, that makes the difference. To quote Ron Jeffries,  "people can screw up everything" ; that doesn't reflect badly on the tool they used.
It is a reversible decision. If you design your tests to be independent of the test runner, you can swap out the existing runner at any time. e.g. the idea below (see image) that I've had some success with (for rich desktop clients). Maybe you'd find that none of them work for "your team", look for something new that works (e.g. maybe Cucumber is a better fit). Instead of spending hours in analysis paralysis, choose one, get started, reflect and adapt.

No comments:

Post a Comment