We switched our Drupal deploy pipeline from SSH over to AWS Systems Manager Session Manager. I figured it would mostly be a connection setting change. That is not how it went. Our restore step syncs 12,000-plus files from S3, and the output alone was enough to break things. The async result file went past 13 MB and reading it back over SSM just stalled. Then we found out the aws_ssm Ansible plugin won't do poll: 0, our Run Command helper mangled anything with a backtick in it, and a fresh instance sits 2 to 5 minutes before SSM even knows it exists. The one that cost us real time was stranger. The restore would finish on the box in 10 to 20 minutes, Jenkins kept waiting anyway, and around the 40 minute mark the job died with a 403 on ssm:TerminateSession. Next run, same code, it passed. I wrote up what explained it, the wrong turns we took getting there, and a checklist if you're planning the same move. Still glad we switched. SSM just breaks in quieter ways than SSH did.
We moved our deployment pipeline from SSH to AWS Systems Manager (SSM) Session Manager expecting it to get simpler. It got more complicated first. Most of that complexity showed up in places we hadn't thought to look, and the worst of it looked random.
Our team runs blue-green deployments for several Drupal websites on AWS for a federal health agency. Jenkins drives a five-stage pipeline for each site. It runs pre-deploy checks, launches a new "blue" instance through an Auto Scaling group, restores a production backup onto blue, runs pre-promote checks, and promotes blue to live.
The restore stage is where this story lives. Ansible runs composer site-restore inside a Docker container on the blue instance. That loads the database and syncs site files from S3. On our largest site, the sync covers more than 12,000 files.
Why we moved to SSM
We switched Ansible from SSH to the aws_ssm connection plugin, and switched our ad hoc host commands to SSM Run Command. The reasons were security and audit, not speed.
No inbound SSH port on the instances. No SSH keys to distribute and rotate. Access controlled through IAM. Session logging for audit.
We'd make the same decision again. The rest of this post is about what the move cost us, not whether it was worth it.
What we expected to get simpler
We expected fewer moving parts. No key distribution and no inbound port 22. In our heads, the change was a connection setting in Ansible and a different CLI call for one-off commands.
What we learned is that SSM has different failure modes than SSH. The pipeline logic didn't change, but the transport underneath it did, and the transport had opinions.
The surprises
1. Output volume. The restore printed thousands of S3 sync lines. We ran it with Ansible async, so that output was captured into the async result file, which grew past 13 MB. Reading it back over SSM stalled. We now redirect the output to a log file on the instance and print only the tail.
2. No fire-and-forget async. The aws_ssm plugin doesn't support Ansible's poll: 0 pattern, so the usual trick for long tasks (start it, move on, check back later) was off the table.
3. Quoting. Our Run Command helper passed the script as a single-line parameter. Multi-line scripts, single quotes and backticks all corrupted it. The fix was a helper that base64-encodes the script and decodes it on the instance.
4. Permissions. On new instances, the deploy user wasn't in the docker group. We now run a root-level chmod on the Docker socket through Run Command before Ansible starts.
5. Startup timing. New instances take 2 to 5 minutes to register with SSM, so the pipeline needs an explicit wait for the agent to show up before the first command.
6. Plumbing. The aws_ssm plugin needs an S3 bucket for file transfer in each account. We also had to pass cross-account AWS profiles correctly into a Docker-based Ansible runner. None of it is hard, but all of it has to exist before the first run works.
The ghost wait
On our largest site, the restore finished on the instance in 10 to 20 minutes. Jenkins kept waiting anyway. About 40 minutes after the task started, the job failed with a 403 on ssm:TerminateSession.
The next run, with the same code and the same database, passed. That made it feel random, and random failures invite guesses.
Our wrong turns
One of them is worth admitting.
An early diagnosis rested on a misread Unix timestamp, and we committed a "fix" based on it. That diagnosis came from an AI coding assistant, and we accepted it without converting the timestamp ourselves. The tool wasn't the problem. Skipping the check was.
What the evidence showed
When we lined up the instance timing against the Jenkins job, the pieces started to fit.
The Ansible container received temporary AWS credentials once, as environment variables. They could never refresh. Our deploy roles allow one-hour sessions, and the AWS CLI can hand out cached credentials that are already partly used. So a container could start its work with well under an hour left.
The open SSM session kept working after those credentials expired. Only the API call that closes the session needed fresh credentials. That's why the 403 arrived at the very end, on TerminateSession, instead of anywhere in the middle.
That explains the error, but not why the session was still open long after the work finished. Two more facts helped there.
There's an open issue in the community-maintained amazon.aws Ansible collection (#3070) describing the SSM plugin blocking indefinitely on a read, even after the remote work is done, in a way its timeout can't interrupt. This is a bug in a plugin, not an AWS outage, and it's reported openly with a proposed fix.
Session Manager also closes sessions after 20 minutes of inactivity by default, and only client-side input resets that timer. Output from the instance doesn't count. A long, quiet task can trip it.
Put together, a run could pass or fail depending on how much time was left on cached credentials and how the session behaved near the end. The same code and data could land on either side.
What we changed, and what's still open
Redirect long output to a log file on the instance and print the tail. (Done) Base64-wrapped Run Command helper. (Done) Root-level chmod on the Docker socket through Run Command before Ansible starts. (Done) Explicit waits for SSM agent registration. Move the long-running restore off the interactive Ansible SSM session and onto Run Command, using send-command and polling get-command-invocation. That path has no idle timeout, and the host CLI refreshes credentials on each call. Pin Ansible and the amazon.aws collection versions in the runner image. Timestamp every task and poll so the logs show real timing.
Here's the shape of the Run Command pattern, scrubbed and simplified:
As of now, the latest stage build passed its restore in about 10 minutes. The underlying plugin hang isn't fixed upstream, so we're treating the pass as a good sign, not a closed case.
A checklist for your own move
Keep long task output off the session. Write it to a file on the instance and print the tail. Run long jobs with Run Command and poll for status instead of holding an interactive session open. Don't hand a container static credentials for a job that can outlast them. Log how much time is left on them when the job starts. Compare the Session Manager idle timeout with your longest quiet task. Encode any script you send through Run Command. Wait explicitly for SSM registration before the first command on a new instance. Create the plugin's S3 bucket and cross-account profiles in every account before migration day. Pin Ansible and collection versions in your runner image. Timestamp every step, and convert timestamps yourself before acting on them.
SSM was still the right move for us. It just fails differently than SSH, and the failures are quieter.
If you've made the same move, we'd like to hear what surprised you. What broke first, and what did you change?
Comments