How to Auto-Quarantine Flaky Cypress Tests

1. Define Quarantine Thresholds & Triggers #

Establish clear failure metrics before automation. A test should enter quarantine after exceeding a configurable threshold (e.g., 3 failures in 10 CI runs) while maintaining a minimum retry success rate. This logic forms the foundation of Building Auto-Quarantine Workflows and ensures only genuinely unstable tests are isolated. Avoid single-failure triggers to prevent false positives.

2. Parse Cypress Results & Generate Quarantine List #

Cypress writes results to the path configured via --reporter-options. Use a lightweight Node.js post-run script to parse that JSON output. Extract test titles, file paths, and failure counts. Map failing tests to a version-controlled quarantine.json array. Ensure the script handles concurrent CI jobs by using atomic file writes or a centralized artifact store.

// scripts/quarantine-parser.js
const fs = require('fs');

// Cypress JSON reporter output path (configured via --reporter-options output=...)
const results = JSON.parse(fs.readFileSync('cypress/results/output.json', 'utf8'));
const quarantine = fs.existsSync('cypress/quarantine.json')
  ? JSON.parse(fs.readFileSync('cypress/quarantine.json', 'utf8'))
  : [];

// Cypress JSON reporter: each suite has a `tests` array with `attempts`
results.runs.forEach(run => {
  run.tests.forEach(test => {
    const failed = test.attempts.filter(a => a.state === 'failed').length;
    const passed = test.attempts.filter(a => a.state === 'passed').length;
    if (failed > 3 && passed < 7) {
      const id = `${run.spec.relative}::${test.title.join(' > ')}`;
      if (!quarantine.includes(id)) quarantine.push(id);
    }
  });
});

fs.writeFileSync('cypress/quarantine.json', JSON.stringify(quarantine, null, 2));

3. Dynamically Skip Quarantined Tests in Cypress #

Read the quarantine list at runtime via a Cypress support file. Use the before() hook to skip matching tests before they run. This prevents pipeline failures while preserving test execution logs for debugging.

// cypress/support/e2e.ts
import { existsSync, readFileSync } from 'fs';

const QUARANTINE_PATH = 'cypress/quarantine.json';

const quarantineSet: Set<string> = existsSync(QUARANTINE_PATH)
  ? new Set(JSON.parse(readFileSync(QUARANTINE_PATH, 'utf-8')))
  : new Set();

// Use before() to skip quarantined tests before execution.
// Cypress does not expose a test.skip() API in event hooks;
// the standard pattern is to call this.skip() inside before()/beforeEach().
beforeEach(function () {
  const spec = Cypress.spec.relative;
  const title = this.currentTest?.fullTitle() ?? '';
  const id = `${spec}::${title}`;
  if (quarantineSet.has(id)) {
    cy.log(`Quarantined: ${id}`);
    this.skip();
  }
});

4. Automate PR Checks & Notification Routing #

Integrate the quarantine script into GitHub Actions or GitLab CI. Trigger Slack/Teams alerts when a test enters or exits quarantine. Require a reliability engineer sign-off or automated root-cause analysis before re-enabling quarantined tests. Block merges if quarantine rate exceeds defined SLOs.

# .github/workflows/cypress-ci.yml
- name: Run Cypress Tests
  run: npx cypress run --reporter json --reporter-options "output=cypress/results/output.json"

- name: Update Quarantine List
  if: always()
  run: node scripts/quarantine-parser.js

- name: Commit Updated Quarantine
  if: always()
  run: |
    git diff --quiet cypress/quarantine.json || (
      git config user.name 'ci-bot'
      git config user.email 'ci@company.com'
      git add cypress/quarantine.json
      git commit -m 'chore: update quarantine list [skip ci]'
      git push origin HEAD
    )

Common Pitfalls #

• Quarantining tests based on single CI failures instead of statistical flakiness thresholds
• Failing to version-control the quarantine list, causing CI race conditions across parallel runners
• Skipping tests without logging the quarantine event, leading to silent coverage degradation
• Not implementing an automated unquarantine validation step to verify fixes before re-enabling

FAQ #

How do I safely unquarantine a Cypress test? Run the quarantined test in isolation against a staging environment with 10+ consecutive passes. Update the quarantine script to remove it from the list only after CI validation confirms stability.

Does auto-quarantine affect overall test coverage metrics? Yes, skipped tests reduce execution coverage. Track quarantine rate as a separate reliability metric and require root-cause resolution within a defined SLA to restore full coverage.

Can this workflow integrate with Cypress Cloud? Yes. Use Cypress Cloud’s --record flag to capture flakiness metadata, then cross-reference the cloud API with your quarantine parser for centralized tracking.

Reliability Metrics #

• Quarantine Hit Rate (%)
• Mean Time to Quarantine (MTTQ)
• False Positive Quarantine Rate
• CI Pipeline Pass Rate Delta (Pre/Post Quarantine)
• Test Recovery SLA Compliance