Shifting Towards Omni Testing: Part 2

This is part 2 of a series originaly written for "Testing in DevOps" community, from Lisa Crispin et al.

In Part 1 I've introduced the idea of "Omni Testing" as a term that better expresses the concept of continually testing throughout the DevOps infinite loop of discovery and delivery. In this article I'll talk about shifting testing efforts versuys what Continuous Testing really means and how Omni Testing can be used to better describe how we look at testing nowadays.

We all heard about Shift-Left Testing and Shift-Right Testing in recent years. Probably you may have got the idea that it was the thing, that single idea that was going to massively improve your testing. Due to this misconception, for example, Janet Gregory prefers to think about testing as something always happening and holistic (i.e. considering all parts of the whole and their interconnection).

Let's be clear: you cannot shift something that is always there and happens all the time; however, what you’re focused on at a given moment in time can, and probably should, change. Within a team, testing happens continuously and in parallel, so different people may be performing quite different testing activities and from different angles.

If we have pure waterfall development in mind, testing happens as a phase after implementation. However, in real-life, testing is not constrained anymore. Of course, some teams perform more intensive testing before releasing; in that sense, people may refer to shift-left testing efforts as a way to clarify requirements altogether from the start and introduce more unit kind of tests. Similarly, with DevOps, you may "shift-right" testing efforts to production and use as a feedback loop to product development.

However, testing itself is not shifted-left nor right; testing efforts can shift though. Having "Agile Testing Quadrants" (by Lisa Crispin & Janet Gregory) and previous work from Brian Marick (i.e. testing matrix), we can see that there are different types of tests, with different goals. As Lisa and Janet point in their book Agile Testing – A Practical Guide for Testers and Agile Teams, risks are an inherent and important part of projects where testing is used as a mitigation strategy.

As you plan each epic, release, or iteration, work with your customer team to understand the business priorities and analyze risks. Use the quadrants to help identify all the different types of testing that will be needed and when they should be performed.

Thus, the type and level of testing performed, where and how it is performed, are dynamic.

Testing is not a specific task assignable to one person and delimited in time; everyone is involved in testing, it happens in different ways, at the same time and continuously; and this leads us to Continuous Testing.

From Continuous Testing to Omni Testing

Continuous Testing has been around for some time; the “Continuous” wording is not new. Unfortunately, it is being pushed by some tool vendors and some people to sell it as a way to continuously run “automated tests”.

Well, first there aren’t automated tests in a strict sense; you have automated checks but testing frameworks having been calling them as such from the start. Second, “continuous” in testing is not related to automated scripts/checks being continuously run or not. Even if it was, it wouldn’t make sense, as in CI “automated tests” are triggered on code commits, pull-requests, merges or due to some predefined schedule.

So what does Continuous Testing mean? I advise you to start by this post from Lisa Crispin and complement it with this course from Elisabeth Hocke.

We know that automation plays a key role in DevOps and also, though to a minor scale, on Continuous Integration (CI) practices which are closely related to Continuous Testing.

The CI concept was depicted by Grady Booch and largely widespread with Extreme Programming as one of the 12 core XP practices (Kent Beck, Ron Jeffries, Ward Cunningham, et al.

During Continuous Integration, "automated tests" (i.e. automated checks to be more precise: code that runs and performs checks to validate expected behavior) are run. These will help assure, to a point, that code that has been integrated isn’t breaking anything and previous expectations are still met.

As teams grow and code is committed more often, CI provides the means for immediate feedback so the team knows right away when a possible “bug has been added”. It can quickly be fixed before anyone builds on top of it. These automated scripts perform unit and integration tests but can also cover other types of tests such as security tests and more.

As builds are continuously being started by the CI process, these kinds of tests are also constantly being run.

However, “Continuous Testing” is not limited to these “tests”. Lisa Crispin and Janet Gregory put it like this instead:

Collaborative testing practices that occur continuously, from inception to delivery and beyond, supporting frequent delivery of value for our customers. Testing activities focus on building quality into the product, using fast feedback loops to validate our understanding. The practices strengthen and support the idea of whole team responsibility for quality. -- Lisa Crispin and Janet Gregory

In my opinion, this is the real reason behind the "Continuous" wording before "Testing".

Well, at first sight, “Continuous ” makes sense whenever everything else is using the “Continuous” prefix. But you don’t hear about “Continuous Coding”, do you?

Thus, a better adjective should be used instead: I think Omni (Testing) may describe Testing in a more complete way. Do you agree?

Continuously evolve our way of understanding the world around us

We moved from testing in waterfall, to testing in Agile, to testing in a DevOps context. Testing, similarly to other processes, evolved along the way; its essence hasn’t.

We still use testing for the same reasons, because we aim to understand what we test and use that information to make decisions.

Whenever we go to a shop and buy something, whenever we download an app and try it out, whenever we meet or go out with someone: we’re trying to somehow understand what’s in front of us. We’re information seekers because ultimately we want to take actions leveraged on data, i.e. informed decisions.

In software development, which may also include hardware development, by the way, testing is done as a means to ensure that the deliverables produced on our products/solutions iterations have a certain level of quality. And this includes evaluating not only expected behavior but also that “customer” needs are covered in the best way, thus we look at certain quality attributes.

Testing is centered around humans and it requires them. However, machines, tools, algorithms, and AI can leverage the unique power we have within us to provide better testing. That’s around 3.5 billion years of continuous evolution and accumulated learning about life on Earth.

I know what you may be thinking: maybe there is no Agile Life, no Continuous Life, and we also don’t call it Omni Life right? Well, true; similarly to what happened with life, testing evolved, it’s widespread, it lives from the outskirts of software development. But, life is restricted in many ways: to certain conditions, to our planet (as far as we know).

Testing, on the other hand, is a broader concept; no limits attached... and being Omni is part of its real essence.