Test Automation: Five key considerations

By Justin Brown, Client Services Director - Roq

Test Automation has been a part of software testing and Quality Engineering since the late 90’s.  However, it is now firmly on the radar of almost all organisations – from mature enterprises to start-ups.  The benefits are widely recognised and are now being discussed at the highest levels.

As a result of this, it has now become a breeding ground for over-hyped expectations, where the budgets very rarely match.  As an ‘Automation-first’ company that provides services to clients in this space, this is particularly dangerous and one area we tread with caution.  We want to set the right expectations and help clients maximise their investment.

Test Automation is a great building block to faster delivery.  It reduces risk and increases productivity.  With the right sponsorship, the right approach, and the right investment, all of these benefits are achievable, so why do organisations rarely provide these three things if the outcome is so positive to the business?

We’ve pulled together five key points for you to consider if you want to make Test Automation Acceleration success a reality:

1. Maximise the potential within your teams

Test Automation should be complementary to the work your development, test and release teams do.  You could even use it within production if you have the correct setup and appetite for this approach.  It should be a productivity enhancer to every one of them – a way to increase their throughput daily. 

One of the myths of Test Automation is that only testers should be responsible for automated testing – and therefore, if the team are lacking in the required specialist skills, it can’t be considered. The truth is that testers at all levels can contribute.  Their primary value lies in the test analysis skills they apply to identify and define the testing required.  The testers can rapidly learn a low code/no code solution, or if that is not appropriate, developers can help build the test implementation.

There is a UK-wide (if not global) digital skills shortage but the greatest asset you have is your people and their knowledge and experience.  Upskill them, set their goals higher.  It can be a catalyst for retention and recruitment and at the same time increase your speed of delivery.

2. Focus on the Goal

One of the biggest red flags we see when we start to work with organisations is that they refer to the SLA’s that they have in place with their larger suppliers, for example:

• volumes of test

• % coverage

• regression packs run etc. 

The truth is that these are not totally relevant. 

The relevance is in the “value” that the automation brings to your release, business, and team.  The value is determined by the level of risk you are mitigating against.  It’s about ensuring there is enough time for your exploratory and functional testing teams to spend on the areas that need that inquisitive validation to investigate and explore – assessing business risk.  It’s about ensuring your developers are picking things up early and that coding standards are high.  SLA’s are designed as contractual targets and to show pretty graphs, they are generally not used to measure against business risk – the only reason why you are testing in the first place! 

Look at these carefully and make sure you are comfortable with the value that you are getting from your investment. Identifying Return on Investment can be key when selling the benefits of automation to stakeholders.

3. Think differently regarding how you finance it

Roq CEO, Stephen Johnson comments,

“Having worked in testing for over 20 years, there is no doubt testing has become more prominent in discussions with Senior IT leaders.  It is on the agenda – and that’s great. However, a key issue is that Test Automation, in my view, is budgeted for incorrectly and it is one that can be easily fixed.  It is generally seen as part of the traditional testing spend, and it therefore fits within the level of investment that testing usually does – 20-25% of project spend.   

However, where I have seen automation work incredibly well, is when organisations invest in it like a development project.  In truth, the investment must match the ambition – in one circumstance we have been involved in, the result was game-changing – monthly releases went to daily releases across 7 brands in 18 months.  The initial investment was based on a move to CI/CD set-up and then to mobilise and integrate that for every product owner/brand.”

The subtlety of this may surprise people, but if you have the mindset of this as a project, particularly if you are looking at legacy application/regression builds, then the approach will get more buy-in, more budget and deliver better results.

It is also worth speaking to your CFO and see if you can add “Regression Packs” to your Balance Sheet as an Asset – it’s more complex than you think, but as a reusable asset it can mean different financial accounting that makes the investment more attractive.  It’s certainly worth exploring.

4. Pick the right Budget Owners

When a project is allocated funds to deliver, the project manager is nearly always focused on the immediate delivery milestones – getting the solution over the line.  They are less focused on the true lifecycle of the application – i.e. managing the true risk and cost to the business over time.  As such, they allocate their budgets accordingly – they spend the money fixing today’s issues – whether that be in development, test, environments etc.  That is expected and understandable.  

However, in this approach, the need for a sound regression pack, a well-thought-out automation suite is not at the top of their agenda.  The cost incurred now, will only be realised in a BAU context when they have moved on from the project.  The value to the business is critical, but the immediate value to the project manager is limited.  It’s a constraint that is self-inflicted by the set-up of projects and the associated finances.

Although a lot of organisations are moving towards Agile / Dev Ops this point may be less relevant, but as there are still so many organisations who are doing iterative development or dealing with legacy applications in a project world, then this is still a major blocker to Test Automation.  I have seen examples of an “Automation Tax” being introduced, but the language can be emotive – however, the principle remains – a set allocation of budget/finance dedicated to building a reusable asset.

5. Think Strategically

One of the things Roq often gets asked to review is the Test Automation strategy of an organisation.  The main driver for the conversation is usually that an organisation has pockets of automation in projects, sprint teams, third parties etc.  There is a plethora of tools being used – some open source, some paid-for.  There are mini-successes and there is a lot of shelf-ware.

Whilst experimentation and the idea of innovation is great, the truth of the matter is that most of these things fail because it is not part of a co-ordinated plan.  It’s not that the tools are wrong or that the intentions are negative, it’s because there is no identifiable end goal.  There is very little mapping of the technical landscape before tools are chosen.  There is seldom a strategic view considered of the technology or platforms to be used going forward.

All these things should feed into the test automation approach and therefore it will help you leverage the skills you have internally and remove wasted effort on unfruitful experiments.

At Roq we recognise the need to drive more efficiencies throughout the lifecycle, so always look to introduce automation where it makes sense. We can work with your organisation to review all of the points mentioned above and ensure that they are covered to ensure the best outcomes are delivered when you are looking to implement automation into your organisation.  Get in touch at ask@roq.co.uk.