Automated tests != Free Lunch

Disclaimer: I'm a proponent of XP and truly believe it has made me a much better programmer. This post just aims to let readers see through the fog-of-agile caused due to data-transfer loss as it passes from person to person. Please do not misinterpret this as an anti-agile rant. I'm just saying it doesn't work out always unless you're willing to put in the effort to make the change.

Legend
What you hear… (good) the Promised Land

• What was left unsaid (bad.. or downright ugly)

You have an automated regression-safety net to make future changes. Make a change, Push a button and you will know if you broke anything. CI can provide near instant-feedback. Better confidence.

• You have 2-3X the code to maintain. If you have people who don’t care/are too busy/ not passionate about code quality and writing good tests, the tests are the first to put a chokehold on your productivity. Bad tests are as good as (or possibly worse than) having no tests. You could see yourself in a situation where a section of your team is permanently siphoned to keeping the build/tests green. This turns into a daily bottleneck. Tests need to be self-checking, thorough, readable, professionally written, independent, repeatable, concise and FAST. All this takes effort!

Documentation - the tests can be "live specs" of the application - They never get out of date like documentation.

• It takes a significant level of discipline and skill to write readable spec-tests. An essential skill :to see the What and Why without getting entangled in the How. Most teams get this wrong... without noticing it.
  
• Sidenote: The practice of ignoring failing tests is criminal (but usually not punished accordingly) and can lead to misleading specs.

Quality & Productivity: Leads to high-quality production code. Fewer Bugs. More features added / unit time (because you spend less time in debugging and manual testing)

• IF you let the tests drive/shape your design (ATDD and TDD). Client-first design is an unstated requirement.
• The quality of the code is usually a direct reflection of the people writing it. This means you need craftsmen (> 30-50% of a team) and NOT armies of cargo-cult programmers.
• If you're using automated tests exclusively for regression (or getting your 'agile badge'), you'll slowly grind to a halt. Writing tests for "untestable blobs implemented from a non-negotiable handed-down paper design" is frustrating. People can be stuck on “how do I test this?” – Usually leads to silent trade-offs & non-thorough test which will let in bugs and put you in the net negative w.r.t. productivity.

Less rework/thrashing: The dialogue / conversation (that you have with the customer to come up with the acceptance tests) makes it highly likely that you’re building the right thing..

• Assumes that the customers want to collaborate with the developers and testers. This is not often true.. Real users are sometimes real hard-to-find. Even if you manage to snag one of them, you can only procure a small slice of their time. Real users rarely want to write tests.


• If the customers give a “vision” and delegate the responsibility of mapping them to executable specs to the technical team (or worse the QA/testers), you still run the risk of “This is not what I asked for” late in the game. Regular demos may help shorten the feedback time.. but you may still waste an iteration.The magic potion here is collaboration and conversation.. the tests are just a beneficial byproduct.

Simple: Red-Green-Refactor. How hard can that be?

• Sounds simple.. but is deceptive. True OO is a minority sport. Refactoring is a skill that you need to work on. Off the job practice is mandatory.
  You may need to "hire" a good coach for an extended period (I'd say 6 months-1 release) to get the team rolling. Spot trainings/Just-in-time learning won't work for most teams.