Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.[1][2] Continuous testing was originally proposed as a way of reducing waiting time for feedback to developers by introducing development environment-triggered tests as well as more traditional developer/tester-triggered tests.[3]

For Continuous testing, the scope of testing extends from validating bottom-up requirements or user stories to assessing the system requirements associated with overarching business goals.[4]

Adoption drivers

In the 2010s, software has become a key business differentiator.[5] As a result, organizations now expect software development teams to deliver more, and more innovative, software within shorter delivery cycles.[6][7] To meet these demands, teams have turned to lean approaches, such as Agile, DevOps, and Continuous Delivery, to try to speed up the systems development life cycle (SDLC).[8] After accelerating other aspects of the delivery pipeline, teams typically find that their testing process is preventing them from achieving the expected benefits of their SDLC acceleration initiative.[9] Testing and the overall quality process remain problematic for several key reasons.[10]

Organizations adopt Continuous Testing because they recognize that these problems are preventing them from delivering quality software at the desired speed. They recognize the growing importance of software as well as the rising cost of software failure, and they are no longer willing to make a tradeoff between time, scope, and quality.[2][17][18]

Goals and benefits

The goal of continuous testing is to provide fast and continuous feedback regarding the level of business risk in the latest build or release candidate.[2] This information can then be used to determine if the software is ready to progress through the delivery pipeline at any given time.[1][5][13][19]

Since testing begins early and is executed continuously, application risks are exposed soon after they are introduced.[6] Development teams can then prevent those problems from progressing to the next stage of the SDLC. This reduces the time and effort that need to be spent finding and fixing defects. As a result, it is possible to increase the speed and frequency at which quality software (software that meets expectations for an acceptable level of risk) is delivered, as well as decrease technical debt.[4][10][20]

Moreover, when software quality efforts and testing are aligned with business expectations, test execution produces a prioritized list of actionable tasks (rather than a potentially overwhelming number of findings that require manual review). This helps teams focus their efforts on the quality tasks that will have the greatest impact, based on their organization's goals and priorities.[2]

Additionally, when teams are continuously executing a broad set of continuous tests throughout the SDLC, they amass metrics regarding the quality of the process as well as the state of the software. The resulting metrics can be used to re-examine and optimize the process itself, including the effectiveness of those tests. This information can be used to establish a feedback loop that helps teams incrementally improve the process.[4][10] Frequent measurement, tight feedback loops, and continuous improvement are key principles of DevOps.[21]

Scope of testing

Continuous testing includes the validation of both functional requirements and non-functional requirements.

For testing functional requirements (functional testing), Continuous Testing often involves unit tests, API testing, integration testing, and system testing. For testing non-functional requirements (non-functional testing - to determine if the application meets expectations around performance, security, compliance, etc.), it involves practices such as static code analysis, security testing, performance testing, etc.[9][20] Tests should be designed to provide the earliest possible detection (or prevention) of the risks that are most critical for the business or organization that is releasing the software.[6]

Teams often find that in order to ensure that test suite can run continuously and effectively assesses the level of risk, it's necessary to shift focus from GUI testing to API testing because 1) APIs (the "transaction layer") are considered the most stable interface to the system under test, and 2) GUI tests require considerable rework to keep pace with the frequent changes typical of accelerated release processes; tests at the API layer are less brittle and easier to maintain.[11][22][23]

Tests are executed during or alongside continuous integration—at least daily.[24] For teams practicing continuous delivery, tests are commonly executed many times a day, every time that the application is updated in to the version control system.[9]

Ideally, all tests are executed across all non-production test environments. To ensure accuracy and consistency, testing should be performed in the most complete, production-like environment possible. Strategies for increasing test environment stability include virtualization software (for dependencies your organization can control and image) service virtualization (for dependencies beyond your scope of control or unsuitable for imaging), and test data management.[1][4][10][25]

Common practices


Since modern applications are highly distributed, test suites that exercise them typically require access to dependencies that are not readily available for testing (e.g., third-party services, mainframes that are available for testing only in limited capacity or at inconvenient times, etc.) Moreover, with the growing adoption of Agile and parallel development processes, it is common for end-to-end functional tests to require access to dependencies that are still evolving or not yet implemented. This problem can be addressed by using service virtualization to simulate the application under test's (AUT's) interactions with the missing or unavailable dependencies. It can also be used to ensure that data, performance, and behavior is consistent across the various test runs.[1][7][10]

One reason teams avoid continuous testing is that their infrastructure is not scalable enough to continuously execute the test suite. This problem can be addressed by focusing the tests on the business's priorities, splitting the test base, and parallelizing the testing with application release automation tools.[24]

Continuous testing vs automated testing

The goal of Continuous Testing is to apply "extreme automation" to stable, production-like test environments. Automation is essential for Continuous Testing.[27] But automated testing is not the same as Continuous Testing.[4]

Automated testing involves automated, CI-driven execution of whatever set of tests the team has accumulated.[clarification needed] Moving from automated testing to continuous testing involves executing a set of tests that is specifically designed to assess the business risks associated with a release candidate, and to regularly execute these tests in the context of stable, production-like test environments. Some differences between automated and continuous testing:


Since the 1990s, Continuous test-driven development has been used to provide programmers rapid feedback on whether the code they added a) functioned properly and b) unintentionally changed or broke existing functionality. This testing, which was a key component of Extreme Programming, involves automatically executing unit tests (and sometimes acceptance tests or smoke tests) as part of the automated build, often many times a day. These tests are written prior to implementation; passing tests indicate that implementation is successful.[13][28]

Continuous testing tools

Research firms Forrester Research and Gartner made Continuous Testing a primary consideration in their annual evaluations of test automation tools. Gartner published an updated version of the research in 2019.

Gartner evaluated 10 tools that met their criteria for enterprise-grade test automation tools. The evaluation involved inquiries with Gartner clients, surveys of tool users, vendor responses to Gartner questions, vendor product demonstrations. Gartner required tools to support native Windows desktop application testing and Android or iOS testing support as well as support 3 of the following: responsive web applications, mobile applications, package applications, API/web services. The results of the 2019 Magic Quadrant research are:[29]

In 2020, Forrester Research evaluated 15 tools that met their criteria for enterprise-grade test functional automation tools.[30] Forrester determined 26 criteria based on past research, user needs, and expert interviews, then evaluated products versus that criteria based on vendor responses to Forrester questions, vendor product demonstrations, and customer interviews. Forrester required tools to have cross-browser, mobile, UI, and API testing capabilities. The results of the 2020 Forrester wave are:[30]

See also

Further reading


  1. ^ a b c d e f g h i Part of the Pipeline: Why Continuous Testing Is Essential, by Adam Auerbach, TechWell Insights August 2015
  2. ^ a b c d e f The Relationship between Risk and Continuous Testing: An Interview with Wayne Ariola, by Cameron Philipp-Edmonds, Stickyminds December 2015
  3. ^ Saff, D.; Ernst, M.D. (20 Nov 2003). Reducing wasted development time via continuous testing. 14th International Symposium on Software Reliability Engineering, 2003. Denver, CO, USA: IEEE. pp. 281–292. ISBN 0-7695-2007-3. ISSRE 2003. Archived from the original on 1 August 2016. doi:10.1109/ISSRE.2003.1251050
  4. ^ a b c d e f g h i j k DevOps: Are You Pushing Bugs to Clients Faster, by Wayne Ariola and Cynthia Dunlop, PNSQC October 2015
  5. ^ a b c d DevOps and QA: What’s the real cost of quality?, by Ericka Chickowski, June 2015
  6. ^ a b c The Importance of Shifting Right in DevOps, by Bob Aiello, CM Crossroads December 2014
  7. ^ a b Kinks persist in Continuous Workflows, by Lisa Morgan, SD Times September 2014
  8. ^ Continuous Testing: Think Different, by Ian Davis, Visual Studio Magazine September 2011
  9. ^ a b c d e f Testing in a Continuous Delivery World, by Rob Marvin, SD Times June 2014
  10. ^ a b c d e f Shift Left and Put Quality First, by Adam Auerbach, TechWell Insights October 2014
  11. ^ a b c The Forrester Wave™ Evaluation Of Functional Test Automation (FTA) Is Out And It's All About Going Beyond GUI Testing, by Diego Lo Giudice, Forrester Research April 23, 2015
  12. ^ a b Continuous Development Brings Changes for Software Testers, by Amy Reichert, SearchSoftwareQuality September 2014
  13. ^ a b c Zeichick’s Take: Forget 'Continuous Integration'—the Buzzword is now 'Continuous Testing', by Alan Zeichick, SD Times February 2014
  14. ^ Buy the Wrong Software? A Fix Can Cost $700,000 A Conversation with voke’s Theresa Lanowitz, by Dom Nicastro , CMS Wire October 2014
  15. ^ Jones, Capers; Bonsignour, Olivier (2011). The Economics of Software Quality. Addison-Wesley Professional. ISBN 978-0132582209.
  16. ^ Kolawa, Adam; Huizinga, Dorota (2007). Automated Defect Prevention: Best Practices in Software Management. Wiley-IEEE Computer Society Press. p. 73. ISBN 978-0-470-04212-0.
  17. ^ a b Theresa Lanowitz Talks Extreme Test Automation at STAREAST 2014, by Beth Romanik, TechWell Insights May 2014
  18. ^ Guest View: What’s keeping you from Continuous?, by Noel Wurst, SD Times November 2015
  19. ^ Manage the Business Risks of Application Development with Continuous Testing, by Wayne Ariola, CM Crossroads September 2014
  20. ^ a b The Power of Continuous Performance Testing, by Don Prather, Stickyminds August 2015
  21. ^ Practices for DevOps and Continuous Delivery, by Ben Linders, InfoQ July 2015
  22. ^ Produce Better Software by Using a Layered Testing Strategy[dead link], by Sean Kenefick, Gartner January 7, 2014
  23. ^ Cohn, Mike (2009). Succeeding with Agile: Software Development Using Scrum. Addison-Wesley Professional. p. 312. ISBN 978-0321579362.
  24. ^ a b Experiences from Continuous Testing at Siemens Healthcare, by Ben Linders, InfoQ February 2015
  25. ^ DevOps- Not a Market, but a Tool-Centric Philosophy That Supports a Continuous Delivery Value Chain, by Laurie F. Wurster, Ronni J. Colville, Jim Duggan, Gartner February, 2015
  26. ^ Keep your Software Healthy During Agile Development, by Adrian Bridgwater, ComputerWeekly November 2013
  27. ^ Extreme automation, meet the pre-production life cycle, by Alexandra Weber Morales, SD Times January 2014
  28. ^ Continuous Integration (original version), by Martin Fowler, September 2000
  29. ^ Magic Quadrant for Software Test Automation, Gartner, November 25, 2019
  30. ^ a b "The Forrester Wave: Continuous Functional Test Automation Suites, Q2 2020". Forrester. 2020-06-18. Retrieved 2020-10-16.