Practice : Immutable Infrastructure

Purpose and Strategic Importance

Immutable infrastructure is the practice of replacing components rather than modifying them after deployment. Once a server or container is provisioned, it is never changed - any updates or fixes result in a new version being deployed in its place.

This approach reduces configuration drift, enhances reliability, and improves traceability across environments. Immutable infrastructure enables repeatable, auditable, and secure operations, significantly reducing the risks associated with manual intervention and “snowflake” systems.

Description of the Practice

  • Infrastructure components are not patched or reconfigured in-place once deployed.
  • Changes are made by deploying a new version and replacing the existing instance.
  • Infrastructure is defined as code and built through automated CI/CD pipelines.
  • Older versions are retired or decommissioned systematically.
  • This pattern is commonly used with containers, VM images, and serverless functions.

How to Practise It (Playbook)

1. Getting Started

  • Identify suitable targets (e.g. containers, virtual machines, cloud services) for immutable deployment.
  • Define infrastructure and configurations in code and bake them into deployment artefacts (e.g. AMIs, Docker images).
  • Avoid SSH or shell access to production systems to discourage ad hoc fixes.
  • Use versioned artefacts and enforce deployment via automation tools only.

2. Scaling and Maturing

  • Implement image pipelines to automate building, scanning, and testing of artefacts.
  • Use orchestration tools (e.g. Kubernetes, Nomad, EC2 Auto Scaling) to replace instances seamlessly.
  • Store artefact metadata and history for rollback and audit.
  • Integrate security scanning and compliance validation into the build stage.
  • Treat infrastructure lifecycles like application lifecycles - with clear versioning and rollout strategies.

3. Team Behaviours to Encourage

  • Avoid logging into production systems - resolve issues through image rebuilds.
  • Validate all changes in pre-production environments before replacing infrastructure.
  • Routinely review image pipeline health and update base images for security.
  • Collaborate with platform teams to maintain clean, shared templates and golden images.

4. Watch Out For…

  • Manual patching or interventions on live systems that break the immutability model.
  • Hidden state or runtime configuration changes that don’t exist in the source artefact.
  • Long-lived instances that are never cycled, accumulating risk over time.
  • Overlooking test automation - if images aren't validated, immutability doesn't guarantee quality.

5. Signals of Success

  • New versions of infrastructure are built, tested, and deployed without manual steps.
  • Teams roll forward with a new image rather than fix old ones.
  • Infrastructure is consistent across all environments, with no drift.
  • Recovery from failure involves replacement, not repair.
  • Audit trails and compliance checks reflect infrastructure version lineage clearly.