Test Automation Investment - The Holy Grail?

By Stephen Johnson, CEO - Roq 

So, you want to automate everything? Do you think that test automation is the holy grail of solving your software quality issues? You have been drawn in by the analyst companies with the pretty graphs and charts that promote DevOps, Agile and iterative development as the only way forward and that test automation makes everything fine. 

Well, guess what? That would be a high-risk approach and a very expensive experiment. I say experiment, as it’s likely to fail. 

But you know me. I am not a doom-monger or against an automation-first approach. However, what I would not advise is wasting money at chasing the unachievable dream. 

Having spent a lot of time speaking to clients, the thought of test automation still throws up many familiar scenarios of discussion; Can I reduce headcount? Can we save money on using free tools? Can we do more testing for less? 

The answer to all these is invariably yes, but my concern is that the driver for these questions is usually from one-track – quick cost savings. In my view, this is a dangerous approach.   

I have always associated test automation as a development project. One that needs a real objective, a good notion of business requirements, treated as an investment and something that will have longevity if it is built well, maintained, and continues to serve the purpose of the user. 

As a result, there is a slightly different play to the discussions that follow. A key one is asking why? Why is automation being considered? Are the systems being considered easily automated? What value will automation deliver? What are the drivers – increased coverage, speed to market, regular releases – and so on? Determining these answers can help set the stall out about you understanding the return on the investment of test automation. You can apply this to an application or an estate – similar rules apply. 

And whilst the above points are important, I want to continue the theme of cost and leveraging test automation as an investment and look at it through a slightly different lens. 

A key consideration with test automation is considering the total cost of ownership, not just the build and run. Many areas can be covered here, but I want to highlight a few for this article. 

Approach 

In our experience, having robust automation frameworks, allied to the coding languages used within your business is a good starter. It will mean a level of consistency, a technology platform that your business is familiar with and you can probably leverage tools/licenses in place already. All these have a longer-term impact on the cost of ownership, so think the bigger picture with your decision making. 

You will want to consider the construct of the team delivering this. We would always recommend a good mix of seniority in a team. You want to create scalability, and that will not happen easily if the headcount costs are too high. Think long term. Use talented, junior resources that can code from day one, but will learn and develop under the right leadership. Also, remember, you are still testing, not just automating, so get some good testing brainpower in the team – consider application knowledge and/or domain knowledge, but an inquisitive mind is a must. Whilst there is no secret formula, my advice is that a good blend will serve the long-term benefit of your organisation and provide a sustainable cost-effective model. 

When to automate. The million-dollar question. Unit testing through to UAT. The world is your oyster. However, each application needs to be considered independently. What suits the level of business change, the structure of the development team, the skills available, the appetite to risk, the application type etc. Identify a way of answering these questions quickly, and apply the rigour, and review the decision-making criteria often. Keep it alive with your business priorities. This is about getting the balance of cost versus value every time. 

Hidden Costs 

And then we come to hidden costs. The ones that might come under the radar a little bit but can explode unless they are managed well. Things to consider here are infrastructure costs, such as supporting toolsets (that support test environments (database, devices etc)), cloud environments that are quick to spin up, but nobody tears them down, and passing of data through test phases – rate limits costs, more so in performance testing, can quickly spiral if intense automation is executed. Keep some guidance and controls in place to manage this. 

Then comes the dreaded technical debt. No surprise here, that as a development piece, test automation will naturally carry some technical debt. It is how you deal with this that matters from a cost perspective. I have personally seen a lot of “fail-fast” approaches to test automation – and whilst I endorse innovation and driving change, I often see organisations with pockets of completely conflicting test automation frameworks, techniques and artefacts. And in most cases, a high proportion is of no value, some are questionable and with some investment, some can be rescued and re-purposed. It’s like shadow IT within the testing department. But it all costs money. It costs money to build and innovate and it costs money to re-do. My advice is to try and develop some standards or policy in which your teams can operate, innovate, yet not be too wasteful. 

Overall Business Value 

One of the other things that are is very expensive, but often overlooked when considering the overall cost of ownership, is automation not delivering business value. For example, we recently worked on an SAP project, and we looked at the existing automation pack. It was taking 10 days to execute and 25 days to maintain. Automation, right? Well. We spoke to the business teams and tried to understand what value it was adding, and what was required. Within 3 months, it is now down to 1 day to execute and 0.5 days to maintain. No additional risk and 30+ days saved per release, on one application. 

The example above is not unique. We often see high coverage rates of test automation – usually to hit a contractual SLA’s or KPI’s – but when you dig deeper, on average 50%+ is of absolutely no business value, out of date, and so the client has still got a huge manual team and an automation team maintaining a load of rubbish. All waste, all cost, and no value. Adding rigour around continually assessing the value your test automation is providing will reduce costs. 

Another area in business value is how to financially account for test automation. There are mixed views on doing this, but you can, in theory, capitalise it as an asset, so rather than an Opex cost, it can be added to the balance sheet, and apportion the depreciation over up to 10 years. Whilst a small regression pack will not be worth it, strategically, this could have some significant cost savings to on-going project costs. It’s something that is becoming more widespread, but each organisation will have financial prudence to adhere to. 

Within this piece, I have tried to give a balanced view of what we often see going wrong, along with some ideas and suggestions on how to avoid it or at least manage it. It would be easy to evangelise about how test automation can save you money – in a more traditional way – but I wanted to provide a more strategic perspective on the subject and drive new thinking in what will become a key way of working for a long time to come. 

As always, happy to discuss over Zoom or a Teams call.