Field Notes: Four Container Fixes and the Gotcha Behind Each
A round-up of recent fixes across these images — a 24× smaller OpenResty, healthchecks that actually run, and a PostgreSQL extension toolchain that stops drifting — with the root cause behind each.
Read articleEarlier posts
25 posts
Jun 20
Narrowed is not deadAfter migrating the fleet to a new build engine, I went to delete the old path's now-obsolete code. The problem: a partial migration doesn't kill the old code. It narrows its caller list — and from where the migration leaves you standing, narrowed looks exactly like dead.
Jun 18
The fallback I didn't buildA build died because a base image wasn't mirrored into our registry — and it refused, by design, to go fetch it from Docker Hub. The fallback would have worked. That's exactly why it isn't there.
Jun 16
A tag is not a releaseA routine dependency bump turned every Terraform build red with a 404 — for a version that existed. The tag existed; the release didn't. The build had been asking Git for the latest tag when it should have been asking GitHub for the latest release.
Jun 10
When your tools lie to youIn one CI cleanup, three tools told me things that weren't true: a run status that contradicted itself, a cancel that did nothing, and a review gate that refused code I hadn't written. None were broken. Each was answering a slightly different question than the one I thought I'd asked.
Jun 8
The 284-byte file that rebuilt thirteen containersA transient network reset on a tiny artifact upload made my CI rebuild the entire fleet. The safety mechanism that did it was working exactly as designed — which is the whole point. This is about calibrating a fail-closed gate so it stops over-firing on noise without dulling the alarm.
Jun 6
The dependency graph I forgot to plug inMy images build on each other, but changing a base only rebuilds the base — never the things on top of it. The fix turned out to be a dependency graph I'd already written, tested, and wired into exactly one of the two places that needed it.
Jun 4
Multi-arch is a distributed-systems problemA single-arch build from my laptop quietly overwrote a multi-arch manifest in my registry. Nothing broke for months — until an arm64 build went looking for a base that no longer had an arm64 in it. This is about what a tag actually is, why a registry is shared mutable state, and who should be allowed to write to one.
Jun 2
The cache that cached nothing — how a $(...) ate my memoizationI added a one-line memoization to skip redundant registry probes. It compiled, it would have passed every test I had — and it did absolutely nothing, because bash command substitution runs in a subshell. It got caught by reading the call site, not by a test. This is about the test that would have lied, and the one I wrote so it couldn't happen quietly again.
Jun 1
Turning on arm64 found three bugs. None of them were about arm64.Shipping a Postgres extension feature and enabling arm64 surfaced three latent bugs — a workflow that wouldn't compile, a build that only worked from cache, and an amd64-only compiler flag — that a clean review, green tests, and CodeQL all missed.
May 29
Nine bytes of bash hid a 30-minute hang (part 3 of 3)After five PRs, the dashboard's fast path was active, all unit tests were green, and the network profile had dropped 92%. The step still took 70 minutes. This is the story of how a bash parameter expansion you've probably used hundreds of times — ${var//pattern/replacement} — turned into an O(n) memory copy that ate 30 minutes per Windows-dev variant, and the nine-byte change that fixed it.
May 27
Six PRs to deliver one perf win (part 2 of 3)The architectural shift in part 1 was the right answer. It still took six PRs and a trace-bisection round to make it land. The story of how each fix passed its unit tests, then masked the next bug — and the integration smoke test that finally broke the chain.
May 25
The dashboard that did 540 network calls per regen (part 1 of 3)Our containers dashboard step was taking 70+ minutes per refresh, almost entirely on per-variant GHCR fetches. This is the first of three posts on the fix: the architectural shift from per-regen network calls to build-time lineage enrichment — and why getting that shift right is harder than it looks.
May 19
'Frozen' is not a state, it's a deferred decisionopenresty:1.29.2.4-alpine was broken for six days — not a regression: the OpenSSL 1.1.1w tarball we had pinned simply stopped existing. A post-mortem of a deliberate freeze that rotted, the migration to OpenSSL 3.5 LTS, and the design flaw behind 'frozen'.
May 18
The fix that worked by coincidence: a CI stall that was never in the codeWe shipped a tidy root cause for a 67-minute CI stall. Six measurement rounds — including regenerating the real SBOM and instrumenting the original script — proved the code was never the cause and the celebrated fix was post hoc ergo propter hoc.
May 14
Anatomy of a Docker Image Catalog: SBOM, Sigstore, Multi-Arch, Hub-429-Proof CIHow ten Docker images are built, signed, scanned, multi-arch'd, and monitored daily — with shell scripts, GitHub Actions, and a Jekyll dashboard. Every trust signal verifiable from your terminal.
May 12
When Docker Hub Starts Rate-Limiting: Anatomy of a Two-Month-Old Latent CI BugA regression that slept for 56 days, surfaced by Docker Hub HTTP 429s. The story of why our base image cache was silently bypassed, what `|| true` cost us, and how we structured the cache to actually survive rate limits.
May 10
From Bare Metal to Container: The export.sh Tool for Host-to-Docker MigrationA Debian base image with a companion script that converts an existing Linux host — packages, /etc, /home — into a reproducible container. 80 MB and genuinely useful.
May 8
Immutable WordPress in Docker: DISALLOW_FILE_MODS + SQLite for Single-Container DeploymentsA 237 MB WordPress container that runs SQLite (no MariaDB), blocks in-dashboard plugin installs, and fits the immutable-infrastructure pattern.
May 6
Vendor-Free Observability: Vector + PostgreSQL as a Datadog AlternativeA 52 MB Vector container paired with PostgreSQL (pgvector, TimescaleDB) gives you logs, metrics, and traces without the SaaS bill. Here's how.
May 4
A Hardened OpenVPN Server in Docker: 15 MB Alpine Image, PKCS11-Ready, Zero-PrivilegeRun OpenVPN in a minimal Alpine container with Easy-RSA, PKCS11 hardware token support, and dropped capabilities except NET_ADMIN. 15 MB compressed.
May 2
Zero-Touch Docker Image Maintenance: SBOMs, Sigstore, and Automated Upstream Tracking for 13 ContainersHow we keep 13 container images in sync with upstream releases, signed with Sigstore, and shipped with SPDX SBOMs — without anyone manually running docker build.
Apr 30
Self-Hosted GitHub Actions Runners in Docker: Ubuntu, Debian Trixie, and Windows Server 2022Multi-distro self-hosted runner family — Ubuntu 24.04, Debian Trixie, Windows ltsc2022, each in base (289 MB) and dev (574+ MB with build tools) flavors.
Apr 28
One Terraform Image for AWS, Azure, and GCP CI/CD: Flavors, SBOMs, and Reproducible PipelinesCurated Terraform CLI image with flavors per cloud — 205 MB base, 593 MB full. Bundles tflint, terragrunt, trivy, infracost, and the cloud SDKs you actually use.
Apr 26
PostgreSQL 18 with pgvector, pg_search, and TimescaleDB on Alpine: A RAG-Ready Database in 214 MBRun vectors, full-text BM25 search, time-series, and GIS in one PostgreSQL container — 214 MB for the vector flavor, 242 MB with the full extension set.
Apr 24
Running SSH, HTTPS, and OpenVPN on the Same Port with SSLH in DockerA practical guide to protocol multiplexing with sslh — how to serve SSH, HTTPS, and OpenVPN on port 443 using a 2 MB scratch container.