Why test suites decay — and what it actually costs

Ask any engineering lead whether their team values automated tests and the answer is yes. Ask when the end-to-end suite last caught a real bug before a customer did, and the answer gets vaguer. Test suites rarely die from a decision. They decay — one skipped test, one muted alert, one untested feature at a time — until the suite still runs but no longer protects anything.

This decay is not a discipline problem. It has specific, well-documented causes, and once you see them it becomes clear why “we just need to write more tests” rarely fixes it.

1. Tests break for reasons that have nothing to do with bugs

A browser test has to find things on the page: a button, a field, a heading. It usually does that with a locator — an ID, a CSS class, an XPath. When the interface changes, the locator stops matching, and the test fails even though the product works perfectly.

This is not a minor effect. Hammoudi and colleagues studied record/replay tests — browser tests recorded with Selenium IDE — across 300 versions of five open-source web applications. Of the 722 test breakages they catalogued, locators caused over 73%.¹ Most failures were not the test catching a regression. They were the test failing to keep up with a redesign.

Every one of those breakages costs someone time: open the test, work out what changed, update the selector, re-run. Multiply that by every UI change and the suite becomes a second product that needs maintaining — one that nobody is measured on.

2. Flaky tests teach teams to ignore red

A flaky test is one that passes and fails on the same code. At scale, flakiness is not an edge case. Google reported that almost 16% of its tests showed some level of flakiness, and that about 84% of the transitions it observed from pass to fail involved a flaky test.² In other words, most of the time a test went red, the cause was the test, not the code.

The damage is psychological as much as technical. In a survey of 121 developers, Eck and colleagues found that 58% encounter flaky tests at least monthly, and 73% said that once a test is flaky it is “no longer fully reliable”.³ The authors note where that leads: developers start disregarding test results, and eventually ignore an actual failure.

That is the real cost of flakiness. Not the minutes spent re-running a job, but the habit it builds. A team that has learned to click “retry” has quietly turned its test suite from an alarm into background noise.

3. Coverage falls behind the product

Products grow faster than suites. A new onboarding step, a pricing page variant, a settings screen behind a feature flag — each ships with good intentions and, often, a ticket to “add tests later”. Later competes with the next feature and loses.

Meanwhile the tests that do exist describe the product as it was when they were written. The checkout test still checks last year’s checkout. Coverage numbers can even look healthy, because they measure the code that has tests, not the journeys your users actually take today.

4. Nobody owns the suite

Many smaller teams have no dedicated QA function. Tests belong to everyone, which in practice means they belong to whoever last touched them. When a test breaks during a deadline, the rational choice for that person is to skip it and move on. Each decision is reasonable. Together they hollow the suite out.

What it actually costs

The cost of a decaying suite is hard to put on a dashboard, which is part of why it persists. It shows up as:

• False confidence. The build is green, so the release goes out — but green only means the tests that still exist still pass.
• Slower debugging. When something does fail, the evidence is scattered: a stack trace in CI, a screenshot if you are lucky, a console error nobody captured.
• Users as the test suite. Without a recurring check against the running product, the first report of a broken signup flow is a support ticket — or a customer who quietly leaves.

What a suite that lasts looks like

None of these causes is solved by writing tests faster. They are solved by changing what tests depend on and who carries the maintenance. A few principles hold up well:

Test what users see, not how the page is built. Locators based on roles and visible names — “the button called Sign up” — survive refactors that break CSS selectors. Modern tools such as Playwright recommend exactly this.⁴ It does not remove breakage, but it removes the kind caused by renaming a class.

Treat flakiness as a defect. A flaky test should be quarantined, given an owner and fixed — not retried until it passes. Retries hide the signal you built the suite to produce.

Check the running product, on a schedule. Tests in CI check a build. Users meet a deployed environment, with real configuration, real third-party scripts and real data. A recurring check against staging and production catches what CI cannot see.

Make maintenance someone’s job — or something’s. If keeping tests current depends on spare time, it will not happen. Either a person owns it, or the system proposes updated tests when the interface changes, and a person approves them.

Attach evidence to every failure. A failure should arrive with the screenshot, the console output and the failed requests that explain it. The faster someone can read a failure, the less tempting it is to dismiss it.

The point

A test suite is only as good as the last time someone maintained it. Most teams already know what should be tested. What they lack is the hours to keep it tested as the product changes. Any approach that depends on finding those hours will decay the same way the last one did. The approaches that last are the ones that remove the maintenance from the critical path — and keep checking after people stop.

Next articleThe tool that wrote the code shouldn’t be the only one checking it