Click on it to be able to zoom in.
And now the thousand err... few thousand words that correspond to that picture :)
I'll open with the worst-kept secret: Automation is Code. You're asking the computer to perform a set of steps to check something. That sounds like... programming. (There is also considerable overlap with unit testing.) If you have built a few non-trivial systems, you'd admit that the main challenges are minimizing accidental complexity and maximizing communication (with other humans).
All roads lead to accidental complexity.
Complexity is unavoidable for non-trivial programs. Essential complexity is that which is absolutely required to solve the problem at hand. Accidental is everything else (e.g. complexity introduced by the tools or language or algorithm used). The art is in identifying accidental complexity and minimizing it.
If we just concentrate (really hard enough) at the corresponding node in the center, we see the following inward black arrows (causes).
Wrong choice of Tools
You shouldn't use a hammer to unscrew a nail although it might work given the right level of persistence and resistance to pain. Sadly there are a lot of expensive 'golden hammers' being forced on QA teams. A bad tool can
- get in your way.
- coerce you to go against your better judgement. e.g. forces you to write repeated boilerplate code with no facility for extracting reusable blocks; even though you know 'Duplication is Evil'. You own the tools; don't let it be like the other way around.
- lock you in . It is prohibitively expensive to switch. The vendor can also choose to charge for enhancements and fleece you with their well-thought out pricing and support models
- refuse to play with other tools. Often this leads to the need for inventing wheels (in-house) ... which end up kinda square. Meanwhile the other folks using those round existing (open-source?) wheels are racing ahead
- can breed specialists (for lack of a better term. Thanks Elizabeth H.)
- drag in other relatives (tools from the same vendor). Go to bullet 1.
Cumbersome tools can raise certain specialists to an elite status. These are the only people who dare to touch the codebase ; no one else understands it / knows how to use the tools or run the tests. All the automation work funnels through them. They become a difficult bottleneck limiting team throughput. Imaginary walls pop up, they may have illusions of grandeur and work in isolation unquestioned. Now you have a knowledge silo in the team (god forbid they leave or fall sick or take a vacation :).
Worse as the diagram shows, they form a reinforcing loop with accidental complexity. The complexity precludes the non-elite from making changes reinforcing the specialist status. The specialists continue to add complexity unchecked... widening the gap further.
Cargo-Cults
If this is the first time you've come across this term, read Eric Lippert's excellent post.
They form another reinforcing loop with complexity. Lots of cult members furiously spot-fixing without understanding the design or "Why did that work?" leads to the design turning opaque. Loss of structure leads to BBOMs, which breed the next generation of clueless fixers.
Speculative Architecture
Over-engineering is a leading cause of accidental complexity. Designing for future needs isn't as rampant as on the development side of the fence where the architects live.
Reuse is best when extracted out of 2 or more functioning clients - designing for reuse inside-out usually leads to abstractions that make all clients mad.
Intricate interfaces/classes/abstractions with non-intuitive names all add to the complexity and lead to a steep learning curve for new team members. Not sure who I am quoting here but - "you really can't use what you do not understand" so make sure that the code is within the grasp of the team supposed to work on it.
Duplication
It is the easiest problem to fix and somehow the most widespread code smell. Changing something in one place is a breeze.. changing it in 47 places identically is a mistake-prone chore. Doing this every few weeks is ... inviting bugs over.
Duplication forms another reinforcing loop with accidental complexity. Greater the duplication, greater the confusion / complexity. As the complexity goes up, it can be harder to find the right way/spot reusable pieces. People write their own copy - duplication goes up!
Test-After
This is a throw-back from the way things used to be. Programmers code till they are happy or time runs out. Throw it over the wall to QA. QA finds bugs, throws it back over. This continues for a while.
Cut to year 2011. The wall has been torn down. We can't wait till the development team is done to begin with automation. There won't be enough time. Even if you manage with some end-of-sprint heroics, you're likely to miss out on exploratory / attribute testing. The automation isn't likely to be the best either - in fact it is most probably going to be "the first thing that worked" (TM) considering the pressure and not wanting to be the ones that let the team down. The bugs - they live! (and yes, they're all out of bubblegum)
Test-after tests are usually long granular only-machine-readable scripts that over-specify and coupled to the GUI to boot! After about 4 sprints of heroics, the team is tired and complexity is taking the elevator to the top! Stop and change something.. the next sprint is NOT going to be better.
The heavies : Wrong Tests, Rework, Bad tests, Maintenance, $$$.
Wrong tests
They're usually a symptom of lack of communication/collaboration.
It's bad enough that the team built and wrong thing, you also expended a significant amount time & money in automating it & other testing. Now you got to do it all over again - Rework. That is resources could have been used to give the customer more features. There is no excuse for getting the tests wrong.
Eliminate middle-men & barriers in the communication path. Get the trinity of the customer+tester+developer to start talking! If the customer doesn't have the time to explain things to the team, quantify the implications in monetary terms. You will have someone's attention.
Bad tests
Write a failing test, how hard can that be? But what if I say write a failing test that is automated, thorough, repeatable, independent, precise and employs the best programming ideas? And did I mention it has to run as FAST as possible.
It takes effort, programming skill and discipline. Expecting historically non-coding testers to jump this chasm on their own in a short-time frame is crazy.
Getting this wrong can lead to tests that are
- Spotty - the tests aren't thorough. As a result, bugs keep coming through the net of automated tests
- Repeatable - the tests give you different results each time you run it. Given enough time, this can irreparably damage trust in the tests & consequently the testers. The tests turn into another hurdle for the development team.
- Dependent - the tests have insider information about each other. Run it alone - it fails. Run the whole suite - everything passes. So to check one of them, you need to run and wait for a whole bunch of them.
- Fragile - the existing tests keep breaking all the time. A tiny flutter in a remote corner of the system causes loads of tests to break.
- Slow - "this thing usually takes 2 mins to comeback. But sometimes it takes upto 3. Let's Thread.Sleep for 5 in the test just to be safe." So "usually" you're twiddling thumbs for 3min/test... and for 100+ tests, that comes to 5 hours of waiting. What's the way out? (Hint: No. The solution is not to run it at night.)
- Over-specification - the tests are a cure for insomnia. KLoC after KLoC of prescriptions for the computer.
- Messy - It's code. Possibilities are endless... especially if you factor in some constant deadline pressure and no training
Maintenance and Cost of Automation
Maintaining a bad codebase is only better than maintaining bad tests. No one wants to babysit a tantrum throwing CI build server every day.
Automation costs real money. All "accidental maintenance" is money lost. Since the end-goal is to make money, it follows that automation has to have ROI. It has to pull its weight. If not it might take the team down with it.
Is there any good news at all in this post?
No but the next one does :)
No comments:
Post a Comment