Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.^[1][2] Continuous testing was originally proposed as a way of reducing waiting time for feedback to developers by introducing development environment-triggered tests as well as more traditional developer/tester-triggered tests.^[3]

For Continuous testing, the scope of testing extends from validating bottom-up requirements or user stories to assessing the system requirements associated with overarching business goals.^[4]

Adoption drivers

In the 2010s, software has become a key business differentiator.^[5] As a result, organizations now expect software development teams to deliver more, and more innovative, software within shorter delivery cycles.^[6][7] To meet these demands, teams have turned to lean approaches, such as Agile, DevOps, and Continuous Delivery, to try to speed up the systems development life cycle (SDLC).^[8] After accelerating other aspects of the delivery pipeline, teams typically find that their testing process is preventing them from achieving the expected benefits of their SDLC acceleration initiative.^[9] Testing and the overall quality process remain problematic for several key reasons.^[10]

• Traditional testing processes are too slow. Iteration length has changed from months to weeks or days with the rising popularity of Agile, DevOps, and Continuous Delivery. Traditional methods of testing, which rely heavily on manual testing and automated GUI tests that require frequent updating, cannot keep pace.^[9][11] At this point, organizations tend to recognize the need to extend their test automation efforts.^[1][12]
• Even after more automation is added to the existing test process, managers still lack adequate insight into the level of risk associated with an application at any given point in time.^[2] Understanding these risks is critical for making the rapid go/no go decisions involved in Continuous Delivery processes.^[13] If tests are developed without an understanding of what the business considers to be an acceptable level of risk, it is possible to have a release candidate that passes all the available tests, but which the business leaders would not consider to be ready for release.^[14] For the test results to accurately indicate whether each release candidate meets business expectations, the approach to designing tests must be based on the business's tolerance for risks related to security, performance, reliability, and compliance.^[5] In addition to having unit tests that check code at a very granular bottom-up level, there is a need for a broader suite of tests to provide a top-down assessment of the release candidate's business risk.^[4]
• Even if testing is automated and tests effectively measure the level of business risk, teams without a coordinated end-to-end quality process tend to have trouble satisfying the business expectations within today's compressed delivery cycles.^[4] Trying to remove risks at the end of each iteration has been shown to be significantly slower and more resource-intensive than building quality into the product through defect prevention strategies such as development testing.^[15][16]

Organizations adopt Continuous Testing because they recognize that these problems are preventing them from delivering quality software at the desired speed. They recognize the growing importance of software as well as the rising cost of software failure, and they are no longer willing to make a tradeoff between time, scope, and quality.^[2][17][18]

Goals and benefits

The goal of continuous testing is to provide fast and continuous feedback regarding the level of business risk in the latest build or release candidate.^[2] This information can then be used to determine if the software is ready to progress through the delivery pipeline at any given time.^{[1][5][13][19]}

Since testing begins early and is executed continuously, application risks are exposed soon after they are introduced.^[6] Development teams can then prevent those problems from progressing to the next stage of the SDLC. This reduces the time and effort that need to be spent finding and fixing defects. As a result, it is possible to increase the speed and frequency at which quality software (software that meets expectations for an acceptable level of risk) is delivered, as well as decrease technical debt.^[4][10][20]

Moreover, when software quality efforts and testing are aligned with business expectations, test execution produces a prioritized list of actionable tasks (rather than a potentially overwhelming number of findings that require manual review). This helps teams focus their efforts on the quality tasks that will have the greatest impact, based on their organization's goals and priorities.^[2]

Additionally, when teams are continuously executing a broad set of continuous tests throughout the SDLC, they amass metrics regarding the quality of the process as well as the state of the software. The resulting metrics can be used to re-examine and optimize the process itself, including the effectiveness of those tests. This information can be used to establish a feedback loop that helps teams incrementally improve the process.^[4][10] Frequent measurement, tight feedback loops, and continuous improvement are key principles of DevOps.^[21]

Scope of testing

Continuous testing includes the validation of both functional requirements and non-functional requirements.

For testing functional requirements (functional testing), Continuous Testing often involves unit tests, API testing, integration testing, and system testing. For testing non-functional requirements (non-functional testing - to determine if the application meets expectations around performance, security, compliance, etc.), it involves practices such as static code analysis, security testing, performance testing, etc.^[9][20] Tests should be designed to provide the earliest possible detection (or prevention) of the risks that are most critical for the business or organization that is releasing the software.^[6]

Teams often find that in order to ensure that test suite can run continuously and effectively assesses the level of risk, it's necessary to shift focus from GUI testing to API testing because 1) APIs (the "transaction layer") are considered the most stable interface to the system under test, and 2) GUI tests require considerable rework to keep pace with the frequent changes typical of accelerated release processes; tests at the API layer are less brittle and easier to maintain.^[11][22][23]

Tests are executed during or alongside continuous integration—at least daily.^[24] For teams practicing continuous delivery, tests are commonly executed many times a day, every time that the application is updated in to the version control system.^[9]

Ideally, all tests are executed across all non-production test environments. To ensure accuracy and consistency, testing should be performed in the most complete, production-like environment possible. Strategies for increasing test environment stability include virtualization software (for dependencies your organization can control and image) service virtualization (for dependencies beyond your scope of control or unsuitable for imaging), and test data management.^{[1][4][10][25]}

Common practices

• Testing should be a collaboration of Development, QA, and Operations—aligned with the priorities of the line of business—within a coordinated, end-to-end quality process.^{[1][4][10][17][26]}
• Tests should be logically-componentized, incremental, and repeatable; results must be deterministic and meaningful.^[1][4]
• All tests need to be run at some point in the build pipeline, but not all tests need to be run all the time since some tests are more resource expensive (integration tests) than other (unit tests).^[1][9]
• Eliminate test data and environment constraints so that tests can run constantly and consistently in production-like environments.^[1][4][9]
• To minimize false positives, minimize test maintenance, and more effectively validate use cases across modern systems with multitier architectures, teams should emphasize API testing over GUI testing.^[4][11][12]

Challenges/roadblocks

Since modern applications are highly distributed, test suites that exercise them typically require access to dependencies that are not readily available for testing (e.g., third-party services, mainframes that are available for testing only in limited capacity or at inconvenient times, etc.) Moreover, with the growing adoption of Agile and parallel development processes, it is common for end-to-end functional tests to require access to dependencies that are still evolving or not yet implemented. This problem can be addressed by using service virtualization to simulate the application under test's (AUT's) interactions with the missing or unavailable dependencies. It can also be used to ensure that data, performance, and behavior is consistent across the various test runs.^[1][7][10]

One reason teams avoid continuous testing is that their infrastructure is not scalable enough to continuously execute the test suite. This problem can be addressed by focusing the tests on the business's priorities, splitting the test base, and parallelizing the testing with application release automation tools.^[24]

Continuous testing vs automated testing

The goal of Continuous Testing is to apply "extreme automation" to stable, production-like test environments. Automation is essential for Continuous Testing.^[27] But automated testing is not the same as Continuous Testing.^[4]

Automated testing involves automated, CI-driven execution of whatever set of tests the team has accumulated.^{[clarification needed]} Moving from automated testing to continuous testing involves executing a set of tests that is specifically designed to assess the business risks associated with a release candidate, and to regularly execute these tests in the context of stable, production-like test environments. Some differences between automated and continuous testing:

• With automated testing, a test failure may indicate anything from a critical issue to a violation of a trivial naming standard. With continuous testing, a test failure always indicates a critical business risk.
• With continuous testing, a test failure is addressed via a clear workflow for prioritizing defects vs. business risks and addressing the most critical ones first.
• With continuous testing, each time a risk is identified, there is a process for exposing all similar defects that might already have been introduced, as well as preventing this same problem from recurring in the future.^[2][5]

Predecessors

Since the 1990s, Continuous test-driven development has been used to provide programmers rapid feedback on whether the code they added a) functioned properly and b) unintentionally changed or broke existing functionality. This testing, which was a key component of Extreme Programming, involves automatically executing unit tests (and sometimes acceptance tests or smoke tests) as part of the automated build, often many times a day. These tests are written prior to implementation; passing tests indicate that implementation is successful.^[13][28]

Continuous testing tools

Research firms Forrester Research and Gartner made Continuous Testing a primary consideration in their annual evaluations of test automation tools. Gartner published an updated version of the research in 2019.

Gartner evaluated 10 tools that met their criteria for enterprise-grade test automation tools. The evaluation involved inquiries with Gartner clients, surveys of tool users, vendor responses to Gartner questions, vendor product demonstrations. Gartner required tools to support native Windows desktop application testing and Android or iOS testing support as well as support 3 of the following: responsive web applications, mobile applications, package applications, API/web services. The results of the 2019 Magic Quadrant research are:^[29]

• Leaders: Eggplant, SmartBear Software, Tricentis
• Challengers: IBM, Micro Focus
• Visionaries: Broadcom, Parasoft
• Niche players: froglogic, Ranorex, Worksoft

In 2020, Forrester Research evaluated 15 tools that met their criteria for enterprise-grade test functional automation tools.^[30] Forrester determined 26 criteria based on past research, user needs, and expert interviews, then evaluated products versus that criteria based on vendor responses to Forrester questions, vendor product demonstrations, and customer interviews. Forrester required tools to have cross-browser, mobile, UI, and API testing capabilities. The results of the 2020 Forrester wave are:^[30]

• Leaders: ACCELQ, Eggplant, Parasoft, Tricentis
• Strong performers: Broadcom, IBM, Mabl, Micro Focus, Perforce, Sauce Labs, SmartBear Software
• Contenders: Cyara, Expiretest, Worksoft
• Challengers: Ranorex