The Agile Manifesto makes not one single reference to test automation in agile software development projects. The manifesto was never written with a prescriptive ‘you should do it this way’ approach in mind. As such there’s no formal statement that you should use automated testing. There’s not even a statement that you should test! What there is, however, are some guiding principles that state things like:
Working software is the primary measure of progress.
And another states:
Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale
The agile software development manifesto itself and the twelve principles make absolutely no reference to automated testing. Yet, two of the principles do stress the importance of working software. In an Agile project the ability to implement an automated testing tool is just ONE tool you have access to, to help you achieve that goal of :
Delivering working software frequently
It goes without saying that if you want to release frequently, AND you want those releases to work, then you need to test frequently too. Automation is JUST ONE method to help you increase the volume, frequency and quality of your testing. It is NOT the only method. It is NOT necessarily the preferred method. And it may well NOT be the best method for your project. Remember this is just a “means to an end”. The end goal being – well tested software releases.
Keeping this end goal in mind it’s always worth asking this question first…..
Automated software testing is not always the right tool for every Agile Software Development team. In fact many teams get so caught up in the ‘ideal’ that they forget what the real end goal is. They don’t step back to consider if automation is the right route to that end goal. In fact for some teams it’s clearly the WRONG solution. It’s the wrong solution because a handful of manual testers would have done a better job, in a shorter time frame and at a lower cost.
Some teams in the agile software development domain implement an automation solution that ends up with such high overheads (both in terms of setup and maintenance) that the only sensible conclusion is that they’ve implemented the WRONG solution. The wrong solution for the goal they needed to hit. If they’d only stepped back and considered the totally cost of automation (QA staff, tools, training, maintenance, etc) they would see that that the Return on Investment (RoI) was NON-EXISTENT or even -VE.
We’re not saying automation isn’t the right solution. For many it is the right solution. Just that many software development teams approach it so badly, never looking at the RoI, and continue blindly down a dead end path. A path that would have been far more alive if they’d opted for the equivalent manual testing path.
Or, if they’d stepped back and…
… they would stand a good chance of coming out on top.
Defining your Goal, Strategy and Objectives are part of our tool agnostic G.R.I.S.T. system for implementing test automation in agile projects. If you’d like to learn more about G.R.I.S.T. sign up for 5 emails in 5 days. We’ll take you through the five key implementation aspects you need to consider as part of your agile software development project.
Approached in the right way, with a keen eye on RoI from day one, agile automated testing tools can deliver huge benefits in the pursuit of that Agile goal where we have….
working software delivered frequently.
In short it can be the right answer IF it delivers the quality, coverage, speed, consistency and visibility (with a higher RoI) than the comparable manual testing effort.
If Automation IS The Right Answer then how do we ensure we succeed? To answer that you first need to consider ….
Your ability to add automated tests at speed. You need to keep up with the ever increasing demands for adding new functionality to each software release whist adding automated tests to ensure you maintain those high quality levels that everyone demands of you.
Complex application design and business logic isn’t written down. It only exists in individuals heads. If you can’t get people to write stuff down how can you possibly automate the tests? The result is that large swathes of the application never get tested leaving you with huge release risks.
The ever increasing matrix of devices, platforms and browsers makes it very difficult to deliver comprehensive regression coverage. This is yet another area of significant weakness that threatens your reputation as the next release looms.
For agile testing you’ll probably end up wholly dependent on just one test automation engineer. An engineer who’ll end up holding all the keys to the massive investment you’re about to make. Lose him or her and the drain on your developers picking this up is likely derail, not one, but many future releases.
Agile testing demands a huge investment in people, infrastructure and tools. It’s an essential task yet you may find the results somewhat disappointing (e.g. few defects found). You risk spending more time managing the automation framework than you do writing and executing tests. Potentially a big investment for comparatively little return.
Each of these issues on it’s own is challenge enough. All of these issues taken together presents a significant hurdle. Ignoring any one of them is likely derail or significantly restrict the success of any effort you put in.
The key to success then is to make sure your approach addresses each of these big challenges. For that then you need Big Solutions.
At the outset of any automation project, when you define your Goals, Strategy and Objectives (see G.R.I.S.T. for more info on this) you’ll want to make sure your strategy will deliver on these solutions:
If you have 1 new feature added every day, and you need say 3 automated regression tests added for each feature then you need to be capable of adding 3 automated tests a day. Say it takes you 4 hours to define, write, test, stabilise and validate one automated test. It doesn’t take a genius to work out that one automation engineer is quickly going to get left behind.
Your automation strategy will need to deliver one or other (or both) of the following:
1. Ability to add tests fast
2. Ability for more people to add tests
You either need to have more people writing those tests or you need to make it faster to add them. Or both!
The age old issue of testers finding enough information about what it is they need to test hasn’t gone away. In many ways it’s more prevalent than it’s ever been. Why not kill two birds with one stone though?
What if the application architect (and developers) found it easy enough to outline and document tests in your automation tool? If that’s achievable with little overhead then you’re on your way to Test Driven Development. Maybe not the whole way there but part way there. You’re in a place where your test automation engineers have a head start and in many cases you’re eliminating significant amounts of documentation.
Find an automation solution that makes it fast and easy to write the test once then scale across many devices/platforms. From the outset make sure you’re looking for
Easy to say – not so easy to implement. That is unless you focus on picking the right automation tools at the outset that allow you to abstract out the case creation from the platform/device test execution.
There’s never enough time to test. Even less time to automate your tests. For many teams there is no luxury of having a dedicated automation engineer.
If you don’t have the luxury of dedicated resources you’d better make it easy for those who don’t specialise in this type of testing. That might mean building a framework that allows team members to write tests in Gherkin or Cucumber. Or it might mean investing in a tool that makes it really easy for everyone in the team to develop tests. Either way if you’re clear about who and how you’ll implement things you can spread the load. As they say, “Many hands make light work”.
It’s also worth remembering the old adage “Little and often”. Many a test automation project has succeeded from just one or two people being dedicated to writing tests for just 30 or 60 minutes a day. You just need to make sure you have a strategy that makes little and often, easy.
Success with automation comes from adding a good number of high quality, reliable tests. Success does not come from spending 80% of your time building and managing the automation framework and infrastructure.
When it comes picking a strategy make sure you consider just how much time (and money) it’s possible to waste on aspects like managing infrastructure. It’s a false economy. Money well spent up front, on simplifying and implementing proven solutions is never wasted. Never lose sight of the fact that it’s all about the number of tests you’re adding and execution.
Don’t under estimate the criticality of these requirements when defining your strategy for implementing automated testing tools in software development projects. If you consider the issues up front, address then with solutions as you implement your strategy then there’s no reason why you shouldn’t reap the following benefits:
Big Benefit Number ONE : The faster you can add tests the less technical debt you build up.
Big Benefit Number TWO : Outlining tests in an automation tool makes it easier to share knowledge
Big Benefit Number THREE : Execution across large numbers of devices and platforms saves significant amounts of time on every release.
Big Benefit Number FOUR : More people writing tests means more test coverage you get.
Big Benefit Number FIVE : More time spent writing and running tests – less time spent maintaining and managing.
If you’re in the agile software development game keep your focus on ..
Living by the 80/20 rule.
Making sure that you get it the right way round.
20% managing/maintaining and 80% writing/executing.
Whatever solution you go for, if you’re close to this you can consider your agile automated testing efforts a success.