30 Days of Agile Testing! Day fifteen.

Day 15:
How can you make your testing processes (more) lean?

Hi, I’m back!
Wow, a tough question like this really brings you back to reality after a holiday weekend! ^_^

I find this tough to answer, because I feel like our testing processes are already pretty lean.

Here’s a diagram I drew:


An explanation:
All our features go behind a feature flag. ‘Flag on’ means the feature is available, ‘flag off’ means it’s hidden.

In step one, we test our feature thoroughly on a developers local environment, in both flagged on and flagged off states. This is where the heavy lifting happens.
In step two, we test in our QA environment, flagged off. Also, all our automated tests (checks) run.
If it looks good, we ship to production!
Then flagged on testing can continue in the QA environment.

The thing missing from this diagram is what happens when we flip the flag from ‘off’ to ‘on’, but that’s a story for another day.

In thinking about what could make this process more lean, the answer I came up with was:

Get rid of the QA environment altogether (the bits marked in red)

Shocking! What would that mean?

  • We would need an easy way of flipping a flag per user in our production environment - so it can be on for the tester, but off for everyone else.
  • All our automated tests would run against production instead of QA. We’d need to account for any performance impact this might have, and ensure there were test users in production we could test against.
  • We deal in payments. How can we test a payment in production? Company credit cards, or prezzy cards, are often a sensible solution to this.

We would also need to answer the question - what happens in the case of a bad release. Feature flags are usually pretty safe, but what happens if something slips through? How quickly could we revert, and what would the damage be?

Something I’ve realised in investigating this, is that we’re not that far from being able to make this a reality - if we wanted to. It would help us move faster, but, we already move pretty fast - and, the risk involved may not be worth the effort. It’s an interesting question though!

- JE