* fix(docker): stage2 API_SERVER_KEY bootstrap no longer depends on .env existing (OOF-285)
Fleet sweep found 144/351 started hosted instances (41%) on v2026.8.13+
with no API_SERVER_KEY: the loopback gateway api_server (which serves
/api/cron/fire on :8642) never started, so every scheduled cron fire was
silently lost until the NAS retry budget exhausted.
Root cause chain:
- .dockerignore excludes .env.example (image-size optimization), so
/opt/hermes/.env.example does not exist in shipped images
- stage2's first-boot seed `seed_one ".env" ".env.example"` is a silent
no-op when the source is missing -> fresh volumes never get a .env
- the API_SERVER_KEY generation added in #84339 was gated on
`[ -f "$HERMES_HOME/.env" ]` -> never ran on those instances
Fixes:
- stage2-hook.sh: keygen now creates an owner-only .env when missing
instead of requiring it to exist; still append-only w.r.t. operator
keys, still refuses symlinked paths
- .dockerignore: re-include .env.example (negation after the .env.*
exclusion) so the first-boot template seed works again
- tests: new tests/tools/test_stage2_hook_api_server_keygen.py covers
create-when-missing, append-without-clobber, operator-key preservation,
symlink refusal, and a .dockerignore contract test for .env.example
* fix(docker): container-provided API_SERVER_KEY wins over stage2 keygen (review)
The bootstrap generated a key whenever .env lacked one, without checking
the inherited container environment. That broke the documented
`docker run -e API_SERVER_KEY=...` flow: Hermes loads $HERMES_HOME/.env
with override=True (hermes_cli/env_loader.py), so the generated key
silently shadowed the operator's env key and 401'd existing clients.
- stage2-hook.sh: skip generation when API_SERVER_KEY is present in the
container environment; if BOTH the env and .env carry keys, warn that
the .env value wins at runtime and touch nothing
- tests: regression tests for the env-provided path (skip + no .env
write; env+file conflict warns without clobbering); sandbox runner now
pins/unsets API_SERVER_KEY explicitly so results don't depend on the
host environment
* fix(docker): drop stale empty API_SERVER_KEY= line when container env provides the key
A leftover empty 'API_SERVER_KEY=' assignment in .env clobbers a
container-provided key at runtime (.env loads with override=True and
python-dotenv sets the empty string), so the api_server startup guard
fails and every scheduled cron fire is silently lost — the exact
symptom class this PR fixes, reintroduced in the env-key branch.
Remove the stale empty line (behind the existing symlink guard) before
skipping generation, so the operator's env key actually wins. Addresses
the IMPORTANT finding both reviewers converged on.
Test: env-key + stale-empty-line combination now covered; strict
removal assertion gated on GNU sed (BSD sed on macOS dev hosts skips
the -i invocation, same caveat as the append test).
* fix(docker): warn at boot when a container-provided API_SERVER_KEY is too weak to start the api_server
The startup guard refuses keys under 16 chars. Now that a
container-provided key suppresses stage2 generation, a weak
`docker run -e API_SERVER_KEY=...` value means the api_server stays
down (cron fires unavailable) instead of clients getting 401s against
a generated key. Say so in the boot log, where the operator will look.
* fix(docker): create .env under umask 077 instead of touch+chmod
touch created the file with the inherited umask (typically 0644), then
a silenced chmod tightened it to 0600 — a brief group/world-readable
window, and no warning if the chmod failed. Creating under umask 077
makes the file owner-only from the first instant with no dependence on
a second command succeeding. Covered by the existing 0600 mode
assertion in test_keygen_creates_env_when_missing.
* fix(docker): guard the API_SERVER_KEY append so a read-only .env degrades to a warning, not a failed boot
stage2 runs under set -eu; the unguarded printf append meant a keyless
.env on a read-only volume (or full disk) aborted the whole cont-init
phase and the container boot. Guard it and emit the same loud warning
the create-failure path uses.
Test harness now runs the extracted block under set -eu to match
production (it ran set -u only, so it could not see this defect class);
new read-only regression test verified RED against the unguarded
append via mutation.
* fix(docker): only warn about a weak container API_SERVER_KEY when it is actually the effective key
The <16-chars warning fired before the .env inspection, so a weak
container key alongside a strong .env key produced a false boot-log
claim that the api_server 'will refuse to start' — immediately followed
by the both-keys warning saying the .env value wins, and the server in
fact starts. Move the check into the branch where the env key really is
the effective key on this boot (round-2 review finding, verified by
execution against python-dotenv last-wins semantics).
---------
Co-authored-by: Ben Barclay <ben@nousresearch.com>
7a17a1b8a6