Skip to main content

Flaky tests

What is a Flaky test?​

Flaky tests are tests that fail to produce a consistent result, despite no changes to the code or the test itself. Learn more from our blog.

How Foresight detects flaky tests?​

PREREQUISITE

You need to integrate Foresight's Test Kit action into your CI pipeline. Then, Foresight automatically captures your test report and shows you the details of your tests and test suite performance.

In order Foresight to detect a failed test, a test case needs to be re-ran until it succeeds, with no code change. For instance:

  • ❌ Test failed, re-run
  • ❌ Test failed, re-run
  • ✅ Test succeed. Foresight marks this test case as flaky on that test run.

You can find Flakiness Severity inside the test suite details.

  • Urgent: test marked as flaky more than 75% of the test runs
  • High: test marked as flaky more than 50% of the test runs
  • Mid: test marked as flaky more than 25% of the test runs
  • Low: test marked as flaky at least 1 time.

Detect flaky tests in a minute!​

Language & framework support​

info

A test needs to be a re-ran in case a failure to marked as flaky. You will find some retry & rerun plugin examples for different language options. However, you can use other rerun plugins too.

Cypress​

Foresight Test Kit Configuration​

You need to use CYPRESS with JSON format. In order Cypress to generate JSON file, you can achieve this by adding |& tee results.json after your test command.

## Example Foresight Test Kit Configuration for JEST / JSON
...
    test_framework: CYPRESS
    test_format: JSON
...

Example test command for Cypress​

cypress run --record false --reporter json |& tee results.json

Jest​

Foresight Test Kit Configuration​

You need to use JEST with JSON format.

## Example Foresight Test Kit Configuration for JEST / JSON
...
    test_framework: JEST
    test_format: JSON
...

You can set a global retry mechanism:

  • Create a file named setup.jest.js and add the following:
// you can change 3 as your retry preference
jest.retryTimes(3)
  • Go to jest.config.js
  • Add setupFilesAfterEnv: [‘./setup-jest.js’]

Pytest​

Foresight Test Kit Configuration​

You need to use PYTEST with JSON format.

## Example Foresight Test Kit Configuration for PYTEST / JSON
...
    test_framework: PYTEST
    test_format: JSON
...

pytest-rerunfailures​

pytest-rerunfailures is a plugin for pytest that re-runs tests to eliminate intermittent failures.

pip install pytest-rerunfailures

After installing pytest-rerunfailures you can set its rerun count as a variable:

pytest --reruns 5 --json-report --json-report-file=report.json

Mocha​

Golang​