Test Automation

Test Automation refers to the practice of writing and executing automated tests at multiple levels (unit, integration, end-to-end) to validate system behaviour continuously and reliably.
It's a foundational enabler of continuous integration, delivery, and quality at speed.

Level 1 – Initial (Ad Hoc)

Testing is primarily manual and inconsistent.
Automated tests, if they exist, are minimal or only cover narrow use cases.

  • Most validation happens after development or during UAT
  • No clear test strategy or ownership
  • Defects are often discovered late in the lifecycle
  • Test results are not reproducible or reliable

Level 2 – Managed (Emerging Practice)

Some automated tests exist (typically unit tests), but coverage is patchy and practices vary across teams.

  • Individual developers or teams write tests inconsistently
  • Test automation may only be run locally or on demand
  • No standard frameworks or reusable test libraries
  • Flaky or failing tests are often ignored

Level 3 – Defined (Standardised)

Test automation is a standard practice across the organisation and embedded in the development lifecycle.
Multiple test levels (unit, integration, E2E) are applied consistently.

  • Teams follow a defined test strategy aligned with the test pyramid
  • Automated tests run as part of every CI pipeline
  • Tests are version-controlled, maintained, and reviewed alongside code
  • Quality gates are enforced (e.g. no deployment with failing tests)
  • Engineers understand the balance between speed and depth of tests

Level 4 – Quantitatively Managed (Measured & Controlled)

Test quality and coverage are measured, monitored, and improved through feedback loops.

  • Metrics include test coverage, test duration, pass/fail rate, and flakiness
  • Automated tests are fast, reliable, and provide actionable feedback
  • Flaky tests are systematically triaged and improved
  • Coverage thresholds are enforced and reported
  • Tests run in parallel and integrate with observability systems

Level 5 – Optimising (Continuous Improvement)

Test automation is continuously refined for efficiency, maintainability, and speed.
It enables fast, safe delivery with minimal manual intervention.

  • Test strategies evolve through experimentation and system feedback
  • Teams use techniques like contract testing, mutation testing, and property-based testing
  • Intelligent test selection (e.g. test impact analysis) accelerates feedback
  • Tests are self-healing and resilient to environmental variability
  • Test quality informs architectural design and development practices