The Computing Series

Introduction

A team has 3,000 end-to-end tests and 200 unit tests. The test suite takes 45 minutes to run. Flakiness is high: 15% of runs fail for reasons unrelated to code changes. Developers run tests locally only before pushing, and even then only sometimes. The test suite catches regressions rarely — not because the tests are wrong, but because they are so expensive to run that engineers avoid them.

A different team has 8,000 unit tests, 400 integration tests, and 50 end-to-end tests. Their suite runs in 4 minutes. Flakiness is below 1%. Developers run the full suite before every push. The first team ships with more regressions despite having more tests. Testing strategy, not test count, determines effectiveness.

Read in the book →