Patch Once, Replace Everywhere: how our Image Builder pipeline and Jenkins keep the fleet patched Our deployment cycles can take weeks, and required security patches can't wait that long. This write-up shows how we separated building a hardened image from replacing the servers that run it. Build side: Terraform maintains the EC2 Image Builder infrastructure (components, recipe, build instance settings, distribution, pipeline). GitHub Actions plans and applies the Terraform, runs the build, and publishes the new AMI ID to Parameter Store, with an approval step before promotion. Handoff: Parameter Store holds the AMI ID for each site and environment. Deploy side: Jenkins either picks up the AMI during a regular deployment or runs the standalone AMI refresh job. That job creates a new launch configuration or template, associates it with the ASG, and starts an instance replacement with no other build steps. The refresh job is what lets a required patch reach production without waiting for the next release.
Terraform maintains the image factory. Jenkins replaces the servers. Parameter Store connects the two.
A required security patch lands on a Tuesday morning. How long until every server in production is running it?
For a lot of teams the answer is "whenever the next release goes out." That's a real problem when release cycles take weeks. In a federal environment, required patches come with deadlines, and those deadlines don't move just because a release isn't ready.
On the federal health IT platform our team supports at RIVA Solutions, we solved this by separating two jobs that usually get tangled together. One job is building a hardened server image. The other is replacing running servers with that image. Once those are separate, a patch doesn't have to wait for anything else.
Stop fixing servers, start replacing them
The pattern is usually called immutable infrastructure. Instead of logging in to a running instance and applying updates, you bake everything the server needs into a machine image, which on AWS is an AMI. When something needs to change, you build a new image and replace the servers. Old instances get terminated, not repaired.
This matters most for security. With long-lived servers, every patch cycle is a chance for drift. Two instances that were identical in January quietly diverge by June, and nobody can say exactly how. When every instance launches from the same AMI, they are identical by construction, and you can trace that AMI back to the code that produced it.
Two halves, one handoff
Our setup has a build side and a deploy side, and they meet in exactly one place.
The build side is an image factory. Terraform maintains the AWS infrastructure that EC2 Image Builder uses, and GitHub Actions runs the Terraform changes and kicks off builds. The output is a hardened, encrypted AMI.
The deploy side is Jenkins. Dedicated Jenkins jobs take an AMI and put it into service by creating a new launch configuration or launch template, associating it with the Auto Scaling group, and starting an instance replacement.
The handoff is AWS Systems Manager Parameter Store. When a build finishes, its AMI ID is written to a parameter for that site and environment. Jenkins reads the AMI ID from there. The build side never touches running servers, and Jenkins never needs to know how an image was built.
That separation is the whole trick. Let's look at each side.
Terraform maintains the image factory
Terraform's job here is to own the Image Builder infrastructure. That includes the build components, the recipe that orders them, the infrastructure configuration for the temporary build instance, the distribution settings that name and encrypt the AMI, the pipeline itself, and the encrypted S3 bucket that holds build logs. All of it lives in one repository and changes only through pull requests.
Configuration stays out of the code. The Terraform root reads its settings from Parameter Store, including the base image ID, network placement, KMS key, instance type, and the version of each component. The variables file holds just two values, the site and the environment.
The same code maintains a separate pipeline for each of our Drupal sites across DEV, STAGE, and PROD. Each site and environment gets its own Terraform workspace, pipeline, and encryption key. A Terraform precondition fails the run if the selected workspace doesn't match the target site and environment, and the CI workflows check it again. A STAGE change can't land in PROD by accident.
Inside the bake
Every recipe starts from a STIG-hardened Ubuntu base image that meets the agency's compliance baseline. On top of that, Image Builder runs four small YAML components.
Common packages installs the baseline tooling our operations team relies on. Docker installs Docker CE with buildx and the compose plugin, with Docker's data stored on a dedicated application volume. Telegraf installs the InfluxData monitoring agent and verifies the repository's GPG key against a pinned fingerprint before trusting it. Session Manager user sets up an automation user and sudoers rules so operators connect through AWS Systems Manager instead of SSH keys.
The build runs on a short-lived EC2 instance with IMDSv2 required and encrypted EBS volumes. When it finishes, Image Builder publishes a KMS-encrypted, tagged AMI named with its build date and version, and the build instance is terminated.
Each component and the recipe carry their own semantic version, so updating Telegraf means bumping one version number, and the change history stays easy to read.
One lesson from hardened images is worth passing along. STIG baselines lock down /tmp, which is where Image Builder normally stages its work. Our components run from a dedicated build directory instead. It's a small detail that can eat a week if nobody warns you.
Recommended by LinkedIn
GitHub Actions runs the build side
Three GitHub Actions workflows move changes through the image factory. All of them authenticate to AWS with OpenID Connect, so no long-lived AWS access keys are stored in GitHub.
The plan workflow runs terraform plan on every pull request that touches Terraform, so reviewers see exactly what will change before anything merges.
The apply workflow runs on merge to main and can also be started on demand for a specific site and environment. It verifies the workspace, applies the Terraform plan, and then starts the Image Builder pipeline. A build usually takes 30 to 60 minutes, so the workflow checks the status every minute until the image is available or the build fails. On success it writes the new AMI ID to Parameter Store as the candidate for that environment, along with the workflow run number, the commit SHA, and a timestamp.
The promotion workflow is the gate. An engineer starts it with a site, an environment, and a reason for the audit log. It confirms the candidate AMI exists and is available, then waits on a GitHub environment protection rule until a designated reviewer approves
After approval, the workflow saves the current approved AMI as a previous parameter, so there is always a rollback point. Then it re-reads the candidate and aborts if anything changed while approval was pending. What the reviewer approved is exactly what gets promoted.
Jenkins replaces the servers
Everything so far produces an image. Nothing so far has changed a running server. That's Jenkins's job, and it has two ways to do it.
Path A is the regular deployment cycle. Our normal release jobs read the AMI ID from Parameter Store as part of the deployment. Every release picks up the newest approved image automatically, alongside the application changes going out with it.
Path B is a standalone AMI refresh job. It does one thing. It reads the AMI ID, creates a new launch configuration or launch template, associates it with the Auto Scaling group, and starts an instance replacement. There's no application build, no release packaging, and no other steps.
Both paths end the same way. The difference is timing, and that difference is the reason this design exists.
Why the refresh job matters
A full deployment cycle can take weeks. Features need to be finished, tested, and signed off, and releases follow a schedule. That's appropriate for application changes. It's a poor fit for a required security patch with a compliance deadline.
If the only way to get a new AMI into production were to wait for the next release, every patch would inherit the release schedule. The standalone refresh job breaks that dependency. It's the only way for us to respond to required security patches in a timely fashion, because it lets a patched image reach production on the security team's timeline instead of the release calendar.
Here's what that looks like when a required patch comes in.
Point the pipeline at the updated base image or bump the affected component's version. GitHub Actions applies the change, runs the Image Builder build, and publishes the candidate AMI. A reviewer approves the promotion. The Jenkins AMI refresh job rolls the new image onto the Auto Scaling groups.
No release gets rushed, and no patch waits for a release.
What we get out of it
Patching on our own timeline. Required patches move independently of multi-week release cycles. No drift. Every instance launched from a given AMI is identical, so "are all our servers patched?" is answered with one AMI ID. An audit trail by default. Infrastructure changes are pull requests, each image traces back to a commit and a workflow run, and each promotion records who asked, who approved, and why. Security built into the image. A STIG-hardened base, IMDSv2, KMS encryption, keyless access through Session Manager, and OIDC instead of stored keys come standard with every build. Clean separation of duties. Terraform owns the factory, Jenkins owns the fleet, and Parameter Store is the contract between them.
What's next
Today, builds run on merge or on demand. The pipeline already has a twice-monthly schedule defined that only fires when upstream updates are available, and we'll switch it on once scheduled builds are approved under the agency's security policy. We're also working toward a broader promotion gate, where each candidate image is tested against every Drupal project that depends on it before it can be approved.
The takeaway
The biggest lesson for us wasn't about any single tool. It was about separating building an image from replacing servers. Once those are two different jobs with a simple handoff between them, security patching stops being hostage to the release calendar.
If your team is still waiting on releases to ship patches, start small. Put one image build in Terraform, publish the AMI ID somewhere your deploy tooling can read it, and give your deploy tool a job that does nothing but swap the image.
I'd love to hear how other teams handle golden images, especially in regulated or STIG-hardened environments. What has worked for you?
Comments