Analysis of more than one million production test runs finds 59% of failures can be resolved at a fraction of the cost with AI-assisted maintenance

SAN JOSE, Calif., Sept. 24, 2026 (GLOBE NEWSWIRE) -- (WeAreDevelopers World Congress North America) -- Checksum, the continuous quality platform for engineering teams, today published "The 2026 State of Code Verification," an analysis of more than one million end-to-end test runs across hundreds of production web applications. The report estimates that maintaining a 500-test suite costs $4.3 million a year and requires about 14 full-time engineers, with most costly failures tied to known issues that have mechanical fixes. Engineering teams tend to describe unreliable tests as flaky.

Across 18,000 sampled failures, selector changes accounted for 32% and flow changes for 27%, together nearly three in five. Environment instability, the category engineers reach for first, accounted for 22%. Teams without AI-assisted maintenance saw a median of 14.8 failures per 100 test runs. Teams using Checksum saw 2.7.

"Teams have been told for years that broken tests are the cost of doing business, so they triage, disable, and stop trusting the suite," said Gal Vered, Founder and CEO of Checksum. "What the data shows is that the expensive part was never the fix. It was the hour someone spent working out what broke."

The report breaks down where that hour goes. For failures needing human intervention, investigation and reproduction consumed 41% of total resolution time, ahead of writing the fix at 29%. A selector change took a median of 45 minutes to resolve by hand, while a flow change took 2.1 hours. Longer tests fail more often, with end-to-end journeys of 30 or more actions failing at 31.4 per 100 runs against 9.3 for tests of five actions or fewer.

Checksum's own data shows what changes when the repairs are automated. About 80% of failures were recovered in real time inside CI, and around 70% of test issues were fixed with no humans involved. Of the failures that reached an engineer, roughly 98% were fixed in under 10 minutes. Autonomous repair rates varied by cause, from 91% for selector changes to 52% for flow changes, where the agent has to judge whether the product changed on purpose or broke.

The report also argues that current web agent benchmarks understate what AI can do in production. At the 85% per-action accuracy that those benchmarks report, a routine 10-step workflow would succeed less than 20% of the time, yet companies are automating those workflows every day. This contradiction exists because standard benchmarks measure model behavior in isolation, while production automation depends on resilience over time.

The findings draw on Playwright and Cypress test runs executed in CI, pre-deploy pipelines, and synthetic production monitoring across Checksum's customer base. Cost figures model failure volume against a blended $195 per failure, based on a fully loaded US engineer rate of $150 an hour.

Separately today, Checksum announced updates to its Continuous Quality Loop: sessions that automatically recover from failures and resume where they left off, and memory that preserves proven fixes so the same problems don’t have to be solved twice.

Checksum will be at WeAreDevelopers World Congress North America, booth 657, September 24 to 25 in San Jose, California.

About Checksum

Checksum is the continuous quality platform for engineering teams shipping AI-generated code. Its AI agents automatically generate, run, and maintain end-to-end tests based on real user sessions, so teams can move fast without breaking things. Customers including Counterpart, Movable Ink, and Lyra Health use Checksum to keep end-to-end test coverage current as their products change, at a fraction of the cost of manual QA. Checksum AI, Inc. is headquartered in San Francisco and is available on Google Cloud Marketplace. Learn more at checksum.ai.

Media Contact
JMAC PR for Checksum
checksum@jmacpr.com