Is AI Bringing Judgement Day for Testers?

I thought this was going to be the usual type of testing-related webinar. You know, the ones where the topic is on a vehicle for pitching a testing tool? This time was different though, not in the format or the presentation, but in the claims the presenter was making. At one point she said “This tool doesn’t just automate your manual test cases, it writes them for you.” I have heard a hundred times about test automation tools that claimed to eliminate manual testing, but never had someone said that their tool could terminate the tester too! Well, maybe that’s an exaggeration, the elimination of human beings was never claimed specifically, but it sure sounded like it.

The Terminator series of movies, comics, and one ill-fated TV show has had its ups and downs, but my favorite part of the series has to be the second film, T2 Judgement Day. In this film we learn about the event called Judgement Day, when the artificial intelligence (AI) developed by a US defense contractor called Cyberdyne attempted to eradicate the human race via a nuclear attack. Today, in real life, AI is powering tools like Chat GPT, Bard, and Bing. In the testing world, it is showing up as the latest feature for many, if not all of the major testing tools, or it is powering new tools. It has many in the software testing field wondering if we might see a real Judgement Day for our careers.

In my 20 plus years in the testing field, it seems to me that our kind has often been forced to fight for our place at the software development table or even for our reason to exist. Every so often, there comes another novel attack on our usefulness. These usually come with reasons like “The customer can test the features.”, “The developers can test their own code.”, or “Testing is too expensive.” that have been around for decades. Somewhat more recently it’s been statements like “We’re Agile, just automate all of the testing.”, or “The product owners can do it”. Today, it is this specter of AI, with its eerie ability to mimic (or appear to mimic) human thought that has the value of software testers being questioned anew.

Given this uncertainty, as testers we need to understand how we can stay relevant in the coming years as AI infiltrates our world. The good news is that we should be used to this by now, considering all of the challenges to our work that we’ve already faced. The bad news is that we haven’t always been good at answering these challenges.

Out-Testing The Machines

Ultimately, the survival of our profession will depend on our ability to do our jobs better than these AI constructs can, but how do we demonstrate our superiority? I think there are a few concrete ways we can improve our profession as a whole.

Stop relying on scripted tests. I am amazed that scripted testing, aka writing and executing test cases, still holds such a stranglehold on the testing industry, especially among the toolmakers. How many test management tools are designed around exploratory testing (ET) vs. those designed for scripted testing? Somewhat reassuringly, at least ET is no longer being ignored. The number of ET tools has grown over the past few years, and some of the bigger players like Tricentis (qTest) and X-Ray feature robust exploratory testing tools as an alternative, even if the bulk of their tools’ functionality is still centered on test cases. We can’t expect the tools to drive our behavior though. Scripted test cases will continue to hold a dominant position as long as we continue preaching the message of “repeatable” tests. Why would a tester want to make their job repeatable? The more of us that embrace approaches like ET that rely on the tester’s knowledge and skill, the more the tools will follow and support us.
Write better bugs. Have you ever used an automated test tool that also logged the bugs it found? I’ve used a few, and the results were laughable. But I’ve seen examples and hear accounts of laughable bugs written by human testers too. “It broke” is not a helpful description of a software bug. One the most valuable tools taught in the BBST series of courses is the RIMGEN set of steps to use when writing a bug report. RIMGEN stands for Replicate, Isolate, Maximize, Generalize, Externalize, and maintain a Neutral tone. Check out this article on RIMGEN for more explanation on what these mean and then start getting your developers so accustomed to reading great bug reports that they won’t stand for the gibberish that an AI tool might log.
Know your stuff. An important part of a tester’s job is to know more about the product, technically and functionally, than anyone else on the team. Can a machine test without requirements? Can it log a bug that details how the application is detracting from its purpose? How about if it’s not being consistent with a user’s expectation? These are just two of the consistency heuristics at any tester’s disposal. What about the technical side? AI should have the advantage there right? Not when testers take the time and effort to understand the inner workings of the applications they are testing. Things like where you should look when you see an SQL error, How to check the logging level, what specific error messages mean. These are still tasks that are done best by a knowledgeable human.