Identify flaky tests

What is a Flaky test?

Flaky tests are tests that fail or succeed seemingly randomly when executed on the exact same code.

Why Flaky tests are painful?

The frustration is obvious. In fact, a run of flaky tests can shatter your team’s confidence in the test suite as a whole and create the temptation to bypass certain tests.

Why detecting them is important?

Flaky tests slow down the pipeline and interfere with productivity—not just because you can’t release, but because you’re wasting precious time fixing the tests. What happened to all those productivity gains your test suite was supposed to offer?

How Foresight can help you about detecting flaky tests?

After you integrate Foresight into your CI pipeline which runs your tests, you will be able to see the performance history of your single test execution. As shown below, you can see the latest 10 execution of the near_cache_size_equals_map_size_when_eviction_policy_is_none test. 5 out of 10 executions are failed even though you know that you’ve never changed your code. You can easily identify that this is a flaky test and it blocks your CI pipeline.