Shifting Towards Omni Testing: Part 1

This is a revised article of the one originaly written for "Testing in DevOps" community, from Lisa Crispin et al.

Testing, as we foreseek, is something not limited to time, not limited to space, not limited to a specific team member, not limited to a given type, technique or approach; it is a process we use to understand something, to obtain more information about it so that we can take actions on it.

Testing is not just about continuously "running” (checks); testing is always there, it is part of the whole (e.g. product development), it encompasses different kinds of activities and involves everyone.

That’s why Omni Testing can, perhaps, be a better way of describing Testing as we aim it to be.

Testing is universal. It’s one thing every single human has performed in their life.

For many of us working in software or hardware product development, testing is even more "natural". However, we sometimes misinterpret it or see it just from a narrow perspective.

Lately, we have been hearing about Continuous Testing, Agile Testing, and DevTestOps/DevTestSecOps. Before that, Shift-Left and Shift-Right Testing and "Test Automation” were also around.

This tells us that Testing is:

• broad
• important
• always present
• intermingled with what we do
• used interchangeably, in different contexts

Because of all of this, I think that Testing is a process unbound, in time, location, people and context; yet, it always has a purpose – actionable information. No matter if you’re using it to verify if you retained knowledge after attending a course, to evaluate if you like some food or a perfume, or to understand how a product works and if it meets some customer needs. Even before software was invented, testing was already part of our lives and helping us shape our decisions.

Omni Testing

Why Omni? Is it a "new thing” or a "new kind” of testing?

What types of tests and approaches does it embrace? Who is involved in it?

"Omni” comes from the Latin word omnis and means "all,” "the whole of, "all things,” "everything.”

More and more, teams everywhere are aiming to have a holistic approach to testing. We’re aiming to have ongoing conversations, ongoing experiments and investigations, ongoing checks, ongoing information, and ongoing feedback. The whole team is involved in testing and has quality in mind, from the start. For that, the team starts by assessing all risks, performs all sorts of tests, using all sorts of techniques and approaches.

(Software) Testing is multi-faceted; it can be used in so many ways, although in Omni Testing we try to perform it in a way where:

• it produces relevant information ASAP (i.e. accordingly with the agreed test strategy and the overall agreed meaning of quality)
• it focuses on what matters first
• it is efficient and quickly adapts

Thus, it is not dependent/limited to:

• specific roles
• specific assets
• specific test levels, types, techniques and approaches
• specific methodologies
• specific phases
• specific tools

Defining Testing, again

Defining "testing” although may seem simple at first sight, it isn’t. Why? Because it is a broad concept and * sometimes used incorrectly or in a very restricted sense.

Recently, I had an opportunity of attending a workshop on exploratory testing with Nancy Kelln at TestBash 2019.

To the question "why do we perform testing?" the group came to these results.

• Spend time/resources
• Understanding the system
• Uncover gaps
• Provide valuable info to team members
• Give everyone feedback
• Finding risks
• Make the end user happy
• Translate information to others
• Ready to ship?

Probably, others would come to different conclusions. For some, testing is just about checking and "finding bugs". However, it’s much more than that.

I do agree with Lena Pejgan on DevOps that pictures such as the following one won’t help, as they depict "test” as a phase (which besides not being true is also not very agile).

There’s an in-depth explanation about Software Testing from Alan Richardson, where he goes over the definition from Cem Kaner (The Ongoing Revolution in Software Testing, 2007):

Testing is an empirical, technical investigation conducted to provide quality-related information about a software product to a stakeholder. -- Cem Kaner

What the previous definition subtly highlights is this investigative and exploratory nature of testing. This is why we cannot neglect this crucial aspect. Maaret Pyhäjärvi emphasizes Exploratory Testing, where we as curious, smart, and context-aware humans are driven and challenged by it, even when we do test automation, to provide valuable information that can drive decisions and fix problems ASAP.

Exploratory Testing is an approach to testing that centers the person doing testing by emphasizing intertwined test design and execution with continuous learning where next test is influenced by lessons on previous tests. -- Maaret Pyhäjärvi

In testing, as Alan Richardson correctly points, we build models and we compare them with reality. Anne-Marie Charrett has a short, clear video on Models & Software Testing which may be useful to you in this context.

When focusing on software testing, you’ll probably find some terms: "Traditional", Waterfall, Agile, Continuous, Exploratory and Modern Testing (you may want to look at Alan Page and Brent Jensen "Modern Testing Principles"; there’s a course on MoT).

Let’s look at this testing definition:

Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes: questioning, study, modeling, observation and inference, output checking, etc. -- James Bach & Michael Bolton

I would rephrase this slightly, adding my perspective.

Testing is a targeted process of challenging something (e.g. a product) and our understanding about it, using a set of activities leveraged by specific human skills.

Where,

• "targeted process" – a set of activities, having a common goal (i.e. quality attributes in mind), that will produce value-related information
• "challenging" – includes trial/experimentation/exploration, checking, ...
• "activities" – build models, create test ideas, execute actions in the SUT, investigation, ask questions, provide feedback, ...
• "human skills" – creativity, curiosity, perseverance, analytical thinking, communication, courage, continuous learning mindset, ...

Testing is exploratory by nature; we try things and test ideas, we explore to seek understanding, to obtain information as Dan Ashby states in his adapted model of James Bach’s Heuristic Test Strategy Model. We also use other approaches, such as script-based tests (i.e. checks) which sometimes are automated to verify "compliance” with predefined rules. We use all the outputs we collect as data that we process. This data becomes information, which is then used to improve our learning of the system/product and provide feedback to "stakeholders" (e.g. business, product team).

Whenever testing something we have in mind certain quality attributes (e.g. correctness, usability, scalability) – "requirements" we demand or expect from it.

To perform it, we test at different levels and use a bunch of different techniques that help us shape what we call "tests".

We test everything, software, hardware, specifications, established ideas, documents, and specifications.

Thus, testing is a complete, omni process that uses best-of-breed human skills to answer tailored questions to clarify our understanding about something and that can help shape it before, during and after it is built.

That being said, how does it relate with Shift-Left/Shift-Right and Continuous Testing? We’ll dive further into this topic on the next blog post of this series related to Omni Testing.