Buttons & Bottles - Research to Track earlier testing and it's benefits.

By Dan Barrow

Before we start 19-June

I really love the study of testing, to observe how testing is done, suggest tweaks and observe the results. In a previous role i radically changed the SDLC in search of quality and a long standing regret was that i didn't diarise the process for better self reflection and understanding. This time i will, and here is the where. 

The experiment is simple, we have a number of teams in the organisation, each team will receive 2 empty jars and a pile of buttons. Each time an issue is found a button will be placed into a jar, depending on the time at which it was found. If an issue is discovered before it is deployed to our testing environment then a button goes in the first jar (W) and in the second jar (QA) if it is found afterwards. To begin with we'll review the buttons weekly, and see how we go. 

 What counts as an issue? Bugs, definitely. Previously unconsidered use-cases that may result in a change, yes. Questions that need an answer, yup. UX & design issues, them too! Pretty much anything that makes you go "OooH" or "Erm". 

Why buttons? This is not a metric, at least not one for management to measure, or to communicated outside of the QA Guild. It's not designed to be precise or competitive, it's a finger in the wind to get a feel for how things work in the different teams and how they like to operate. Sure we have bug tracking, but a lot of the issues as we define them never make it to Jira and that would be too tedious to get participation, i know i wouldn't write up every little thing. 

Why? The benefits of testing earlier through WOMMing are a continuation of the agile mantra "test early, test often", by testing before deploying to QA we save 30 minutes per deploy alone, then there is context switching, ticket writing and a whole bunch other time consuming moments that add up. I'm not saying WOMMing adds more quality, great QA find issues anywhere, and not every issue can be found in dev because of how the system is configured but it makes the boat go faster and we all know how the cost of a bug rises the longer it exists.

The Results  18-August

We've been running the research project for 2 months now, and it's been really interesting. It's been fascinating to see how the different team members and roles have taken on the project, some early skeptics converted and others gaining a clearer understanding of the value of their testing. 

One piece of feedback i received was how aware it made people of the additional time needed to fix a bug that is found in QA, if all went smoothly this would still take ~1 hour compared to the average <5mins taken when an issue was found on the developers environment. 

Even so the balance of issues found in QA vs on the Developers machine was still pretty even, WOMMing having a slight advantage. Naturally inquisitive i needed to ask why? Why, when the benefit is clear are we still finding so many issues after landing in QA? There were a number of reasons to this;

Test data, the current standard of test data on developer environments is old and does not contain a broad set of scenarios needed for testing, this adds overheads to local testing time. This is something that can be resolved, our approach is to build up a library of API calls that will be re-useable and generate required test data on demand. I think i'll elaborate more on this in another blog sometime. 

Acceptance Tests, When a acceptance test will cover test scenarios, it can feel redundant to still perform these manually, however until recently it wasn't possible to run these on a local dev environment. 

Integrations, in any system there are always integrations that are required for end to end testing, these can sometimes be expensive or difficult to implement on dev machines. We use a series of test commands to mock this experience, and planning and building these tolls has become a key part to our implementation plans. 

Dependancies, during development it's often hard to test functionality until the next stage is completed, as you can't go end-to-end. We've started looking to create this thin slice experience as soon as possible, enabling testers to complete the end-to-end process and ensure that new functionality integrates with the whole project.


We'll always find issues after landing in QA, but with a clearer understanding as the the cost benefits, we now know that testing earlier and on developers machines can really boost quality and productivity, however you will need to identify are remove as many of the blockers as possible within your own teams in order to maximise this bonus. 

Going Forward

The team have loved playing with buttons, so we're going to continue with the buttons, continue to learn and refine our practices.