Standard : Automated Test Pass Rate

Description

Automated Test Pass Rate measures the percentage of automated test executions that pass successfully during a build or deployment cycle. It is a fundamental quality signal that reflects the health, reliability, and coverage of your automated testing strategy.

A high, stable test pass rate helps teams detect regressions early, deliver confidently, and avoid deployment delays due to failing pipelines.

How to Use

What to Measure

  • Number of passed automated tests divided by total executed tests during CI/CD runs.
  • Can be scoped to unit, integration, system, or end-to-end test suites.

Formula

Test Pass Rate = (Number of Passed Tests / Total Tests Run) x 100

Instrumentation Tips

  • Use CI tools (e.g. GitHub Actions, Jenkins, GitLab) to collect pass/fail results.
  • Break down by test type or scope to identify patterns.
  • Track both per-run and trend over time.

Why It Matters

  • Delivery confidence: Fewer failing tests = fewer unknown risks at release time.
  • Early feedback: Surfaces issues quickly so teams can fix and move on.
  • Flow stability: Keeps pipelines green and developers unblocked.
  • Engineering health: Reveals test quality, flakiness, or neglected areas.

Best Practices

  • Separate flaky tests and track them independently.
  • Tag and group tests to filter and report more precisely.
  • Maintain high-quality assertions and clear failure messages.
  • Monitor not just pass rate, but test duration and coverage.
  • Use pre-merge gates to ensure broken tests never hit mainline.

Common Pitfalls

  • Ignoring persistent failures and normalising broken pipelines.
  • Treating pass rate as a vanity metric—without understanding why tests fail.
  • Letting tests age without maintenance or clear ownership.
  • Relying on pass rate alone without measuring actual coverage or value.

Signals of Success

  • Tests consistently pass on first run with minimal reruns.
  • Failures trigger fast feedback and fixes, not workarounds.
  • Developers trust CI and use it as a safety net.
  • Test failures correlate with real regressions—not false positives.
  • [[Test Coverage]]
  • [[Change Failure Rate]]
  • [[CoE/Measures/Delivery Performance/Deployment Frequency]]
  • [[Lead Time for Change]]
  • [[Code Quality Score]]