Update 2026-11-11 17:30: image tags are now the build time, yyyymmddTHHMMSS (e.g. airflow:20261111T172041), so two builds on the same day no longer collide, and ./run.sh with no tag lists what’s available. The outputs below are from the first build, which still used a plain date tag, 20261111. The downloadable scripts are the current versions.

The outage postmortem had one recurring theme: every time someone rebuilt the Airflow container by hand, it came back missing something. A mount, the concurrency settings, the plugin, the registry login. The image was whatever someone last typed, and the settings that mattered lived only in a running container.

So the same afternoon I wrote three small scripts. build.airflow.sh turns the running container into an image tagged with its build time (airflow:20261111T172041), pushes it, and saves a tarball. run.sh recreates the container from that tag and checks it’s actually healthy, or puts the old one back. restart.sh runs inside the container and keeps the scheduler and webserver up. From the 170 entries in docker diff, the image keeps 13 files (172K). A redeploy is healthy about 15 s after the new container starts, and a failed one rolls back in 1 s.

Two pages I look at every day had the same problem: monospace text that was too thin and pale to read comfortably. One was the log viewer in our Airflow 2.7 web UI, the other the dark theme of an internal Tabler dashboard. In both, Chrome was drawing the text in Courier New, a hairline font, even though both pages list four or five better fonts before it.

The cause was a fontconfig file from the fonts-powerline package, installed in 2019. It puts the symbols-only “PowerlineSymbols” font in front of most monospace families, and Chrome then refuses every one of them. A one-rule fontconfig file fixed it.

Update 2026-11-11: the “a restart is a procedure, not a command” advice below is now three scripts: a dated image built from the running container, a redeploy that refuses while tasks run and rolls back on failure, and a supervisor inside the container. See Rebuildable Airflow.

At 19:56 one evening, a 21.8 TB data disk on our Airflow host started throwing I/O errors. By 21:37 someone had pointed Docker at a fresh, empty data directory to get things running again. That one change, made under pressure and never written down, turned into five separate incidents by 09:30 the next morning: a message broker that vanished, a missing mount, concurrency limits back at their defaults, live-trading processes Airflow no longer knew about, and two restarts that orphaned every live trade.

Everything was back by 10:08. This post is the timeline, the root cause of each problem, how each was fixed (including the 1.5-minute outage I caused myself while fixing it), and what I would change so this can’t happen the same way again.

My Codex CLI sometimes paused for about five seconds when it started. Other launches were almost immediate. After checking the proxy setup, I looked at the startup logs. They showed a model-list request consuming five seconds before timing out.

Three Codex versions were sharing the same model cache. A cache written by one version could be rejected by another, which forced another network request. I aligned the CLI with the shared server and changed the desktop launcher to use the same CLI on its next launch. Five subsequent terminal checks reached the model and prompt in a median of 0.80 seconds.

Update 2026-11-06: a later startup investigation found a separate delay. Different Codex versions rejected each other’s shared model cache and triggered model-list refreshes that timed out after five seconds. See Codex startup and the shared model cache.

My XPS uses lfang.idc as a proxy gateway. v2rayA runs on the IDC host, and applications on XPS connect to the gateway over the private network. Codex needs one extra check: its shared app-server daemon keeps the environment it received when it started.

I keep Claude Code in a tmux pane above a zsh pane, so its status line sits just above my zsh prompt, and the two almost matched. Both used the same powerlevel10k colours and separators, but Claude’s line started 2 columns further in and stopped wherever its text ran out. The zsh line ran the full width of the terminal, so the pair looked misaligned.

The fix was a dotted gap computed from $COLUMNS and "padding": -2, a value the docs don’t mention. Now both prompts start at column 0 and close on the same column, 134 of 135.