Before, a worker killed by a revoked credential or a missing model was booked as an ordinary crash and re-spawned into the identical failure until kanban.failure_limit / max_retries was spent — burning worker slots and the retry budget on something a retry cannot fix (#114587). Now KANBAN_TERMINAL_PROVIDER_EXIT_CODE (78) is its own exit kind, `terminal_provider`: `_classify_dead_worker_exit` books the run crashed with the provider's words appended, and `_account_crashes` force-trips the breaker on that first death (sticky, so recompute_ready does not resume it before the operator fixes the provider). Same booking in the implementation and the review lane — the review worker dies through the same sweep. Transient failures (429 / 5xx / timeout) keep the existing rate_limited requeue and the consecutive_failures budget. `hermes kanban show` / the dashboard diagnostics now fire for that trip below the repeated-failure threshold ("Provider rejected this profile's credential or model — blocked after one attempt") with the fix path. No new columns, no separate review-lane counter, no new config: option B of #114587. The terminal-vs-transient split was proposed in #114589 by @TFOjojo; its regex-on-error-text classifier is replaced by the worker's own FailoverReason verdict. Part of #114587 Co-authored-by: TFOjojo <279183633+TFOjojo@users.noreply.github.com>
8.4 KiB
cron/ (+ kanban) — scheduled jobs and the multi-agent work queue
Applies on top of the root AGENTS.md. Long-form: website/docs/developer-guide/cron-internals.md;
user docs website/docs/user-guide/features/cron.md, kanban.md.
Cron
cron/jobs.py (job store) + cron/scheduler.py (tick loop; scheduler_*.py siblings). Agents
schedule via the cronjob tool; users via hermes cron list|add|edit|pause|resume|run|remove or
/cron. Schedules: duration ("30m", "2h", "1d"), "every" phrase ("every 2h", "every monday 9am"), 5-field cron ("0 9 * * *"), ISO one-shot ("2026-06-01T09:00:00Z"). Per-job fields:
skills, model/provider overrides, script (pre-run data-collection script whose stdout is
injected into the prompt; no_agent=True makes the script the whole job), context_from (chain job
A's last output into job B's prompt), workdir (run with that directory's AGENTS.md/CLAUDE.md
loaded), multi-platform delivery.
Hardening invariants — each guards a real failure; don't weaken without answering for it:
- Inactivity watchdog on cron agent sessions (
_cron_inactivity_seconds()): default 600s idle,HERMES_CRON_TIMEOUToverrides,0= unlimited. It is idle time, not wall-clock — a stalled session is hard-interrupted so it cannot monopolise the scheduler, while a long-but-active job is never cut off. Attached scripts (pre-run orno_agent) are bounded separately by the script timeout (_DEFAULT_SCRIPT_TIMEOUT, 3600s). - Catch-up window = half the period, clamped to 120s–2h; 120s grace for missed one-shots.
- Every recurring occurrence is accounted for:
tick()advancesnext_run_atBEFORE dispatch (at-most-once across a mid-run crash) and stampspending_slotin the same save; a scan that finds the stamp with a dead owner restores the instant ONCE (cron/occurrences.py), the executions ledger'sscheduled_instantblocks a second fire,cron.catch_up_missed: falseskips past-grace misses with a logged reason. Never drop a slot silently (#107485). - Per-home tick lock
<home>/cron/.tick.lockprevents duplicate ticks across processes for that profile's store; never a~/.hermes/...literal. - The ticker binds each served profile's scope for the whole tick, including pre-loop code.
scheduler_provider.py::_start_multiplexis ONE ticker iteratingprofiles_to_serve()sequentially under_profile_cron_scope(home)(home + secret scope + terminal scope) — never N threads (module globals race). Everything a tick touches lives inside that guard: store open, lock path, backoff/failure counters (_note_tick_failure), job env construction, and theon_session_endflush of a finished job. Supervision (scheduler_thread.py:: SupervisedTickerThread; start on gateway boot, stand down for homes another gateway already serves, re-enumerate when a profile dir appears or is tombstoned) is per served home, not per process. Why: a store opened before the scope was entered wrote a secondary profile's run records into the launch profile'sjobs.json. - Cron sessions pass
skip_memory=True; memory providers intentionally do not run during cron. - Cron execution has its own session. Eligible continuable deliveries may mirror or seed the
reply-facing conversation: origin, origin-less home fallback, user-written bare-platform home,
or opted-in explicit targets.
allexpansions do not gain home mirror eligibility. Mirrored briefs are labelled user turns appended at a turn boundary, preserving role alternation. - The cron ticker runs in the desktop-spawned backend when
HERMES_DESKTOP=1— that env var means "spawned by the app", not "a GUI is watching" (root: capability is a property of the session). - Background
delegate_taskis process-local; work that must survive restarts is a cron job or aterminal(background=True, notify_on_complete=True)process.
Kanban (multi-agent work queue)
Durable SQLite-backed board letting multiple profiles/workers collaborate. Users: hermes kanban <verb>; dispatcher-spawned workers use a dedicated kanban_* toolset so their schema footprint is
zero outside a kanban task (footprint ladder rung 3).
- CLI:
hermes_cli/kanban.pyfacade + 14kanban_*.pysiblings (boards,db,db_connect,db_dispatch,db_notify,db_graph(task initialization and decomposition),workspace, ...). Verbs:init, create, list (ls), show, assign, link, unlink, comment, attach, attachments, attach-rm, complete, request-review, request-changes, reopen-review, block, unblock, archive, tail, pluswatch, stats, runs, log, assignees, heartbeat, notify-*, dispatch, daemon, gc. Argparse alias dispatch must accept bothlistandls(root). - Toolset:
tools/kanban_tools.py—kanban_show, kanban_complete, kanban_request_review, kanban_request_changes, kanban_block, kanban_heartbeat, kanban_comment, kanban_create, kanban_link, kanban_attach, kanban_attach_url, kanban_attachments; platforms whose saved selection enableskanban(hermes tools enable kanban --platform <p>; default-off, inCONFIGURABLE_TOOLSETS) get the full set pluskanban_list/kanban_unblockfor board routing. The check_fn reads the schema build's own selection (tools/kanban_toolset_context.py), never the legacy top-leveltoolsetskey alone. - Dispatcher: long-lived loop (default 60s) that reclaims stale claims, promotes ready tasks,
atomically claims, and spawns assigned profiles. Runs inside the gateway by default
(
kanban.dispatch_in_gateway: true). Standalone:plugins/kanban/systemd/hermes-kanban-dispatcher.service. - Plugin assets:
plugins/kanban/dashboard/(web UI) + systemd unit.kanban_db.connectis its own connection helper — do not alias it toprojects_db.connect(a path-proximity generator did).
Isolation: board is the hard boundary — workers get HERMES_KANBAN_BOARD pinned in their env and
cannot see other boards; tenant is a soft namespace within a board (workspace-path + memory-key
isolation, one fleet serving several businesses). After kanban.failure_limit consecutive
non-success attempts on a task (default 2) the dispatcher auto-blocks it to stop spin loops; a
worker exit of KANBAN_TERMINAL_PROVIDER_EXIT_CODE (78 — credential revoked, model gone; the
worker's own failure_reason classification via cli._TERMINAL_PROVIDER_REASONS) trips it on
the first attempt, sticky, because no retry can heal it (#114587).
Process-identity note: kanban --preserve-cache contains "serve" — never classify processes by argv
substring (root). Worker liveness is (worker_pid, worker_started_at) — the start-time fingerprint
(gateway.status.get_process_start_time) recorded at claim time — never bare PID existence, or a
recycled PID gets killed on reclaim.
- Notifications leave through the task's owning profile.
hermes_cli/kanban_db_notify.pysubscriptions carry the profile;gateway/kanban_watchers_notifier.pydelivers via THAT profile's adapter under its scope (_notify_profile_filter), never the multiplexer's launch adapter; a fail-closed skip logs once at WARNING with the remedy, never a barecontinue. Dispatched workers getHERMES_KANBAN_BOARDand the assignee'sHERMES_HOMEpinned in a scrubbed child env (build_subprocess_env+strip_launch_profile_env); they never inherit the default profile's.env. - Prompt injection sites gate on ownership, not tool access. Tool access (
kanban_showvisible via a profile's toolset) and an inheritedHERMES_KANBAN_TASK(delegate children, cron runs beside a worker) are not ownership. The kanban guidance (agent_init,system_promptfallback) and the stop nudge resolve the task viaagent/delegation_context.py::owned_kanban_task(); other readers pair their env read withis_dispatcher_owned_worker_context(). - Descendant fence is a path, not a flag. A delegated child's Kanban marker
(
agent/delegation_context.py::DELEGATED_CHILD_ENV_MARKER) carries the fenced board ROOT;kanban_path_is_fenced(path)denies mutations only on the dispatcher-pinnedHERMES_KANBAN_DBor under that root, so a child working against a scratchHERMES_HOMEkeeps a writable board.
Tests
tests/cron/, tests/hermes_cli/test_kanban*.py, tests/tools/test_kanban*.py. Schedule parsing
and catch-up windows are pure functions — test them as data. Never assert on the verb list or
toolset size (root: no change-detectors). Time-based tests use loose bounds (≥ 2s) and event sync.