A scoped or wedged daily-brief in profile A must not hide a degraded or
live daily-brief in profile B. Split the scheduler's in-flight keys before
counting and add awaitable cron runs independently of non-cron work.
Carry the per-run restart_safe flag to the observer and preserve the
existing bare-ID accessors for unrelated callers.
Thanks to @ehz0ah for diagnosing the multiplexed collision and proposing
the profile-keyed accounting and mixed scoped/degraded regression case.
Preserve @Superfly73's source commit separately; map their verified bare
noreply email for the contributor CI gate.
Refs #130987
Co-authored-by: z1gon <130889443+ehz0ah@users.noreply.github.com>
A cron run handed to an external worker in its own transient systemd scope
outlives the gateway: the shutdown drain deliberately leaves it unmarked
(``mark_running_jobs_interrupted``) and its final send rides the durable
delivery queue, drained by whichever gateway is live next. The after-turn
restart wait still counted it as active work, so ``/restart``, SIGUSR1 and
``hermes update`` held the gateway in ``draining`` — refusing every new turn —
for up to ``agent.restart_after_turn_timeout`` (30 min by default) while the
worker kept running regardless. Observed in the field: a 100-minute job turned
a restart into a ~30-minute lockout.
- ``cron.scheduler`` records whether an accepted handoff got its own scope
(``dispatch.mode == "scoped"``) and exposes
``get_restart_safe_external_job_ids()``; runs already reported as wedged are
excluded there too, because the gateway subtracts both counts and a run in
both sets would be subtracted twice.
- ``_awaitable_work_count()`` subtracts that count as a second, disjoint
exclusion; ``_describe_active_work()`` marks the unit (``restart_safe``).
- A degraded worker (external, but no reachable user bus, so it shares the
gateway cgroup) is NOT listed and is still awaited — the restart would kill
it mid-run.
- The wait and observation log lines now name both exclusion classes instead of
claiming everything still listed will be interrupted.
Tests: tests/gateway/test_restart_drain.py (skip + control),
tests/cron/test_restart_safe_worker.py (registry contract).
Docs: updating.md restart section, cli-config.yaml.example.
(cherry picked from commit 6a2f0b77cf07a60510430fc4ef8b75539ee422fb)
The router turned every failed SendResult into a bare RuntimeError, which
dropped raw_response, and the cron live lane falls back to the standalone
sender on any error. A split send that failed after delivering its head
(Telegram, Discord, and now SMS/WeCom) was therefore delivered again from
the start by the standalone lane.
The router raises PartialDeliveryError for a partial_overflow result; the
live lane records "delivered N of M chunks, then failed" on the run and
does not fall back.
_live_send_text used future.cancel() to tell a send that never started
(wedged loop: fall back to standalone) from one in flight (assume delivered:
a resend would duplicate). A run_coroutine_threadsafe future stays PENDING
until the coroutine finishes, so cancel() returned True mid-send: the
in-flight send was cancelled part-way and the standalone lane then sent the
whole payload again. Paced multi-chunk sends (WeCom's 30 msgs/min bucket)
outlast 60s routinely; a 55-chunk cron payload arrived as 25 chunks and was
reported failed.
The send now records its own start under a lock; on timeout it is
abandoned (and the standalone fallback runs) only when it never began.
The existing in-flight test faked the state with a mocked cancel()
returning False, the exact assumption that does not hold; it is rewritten
against a real event loop, with the never-started (wedged loop) case it
never covered alongside.
The inactivity watchdog polls every 5s, so the same stall reports
"idle for 601s" on one run and "idle for 604s" on the next. The incident
signature hashed that raw number, so each run minted a new incident: an
acked timeout pinged again and the repeat-alert cooldown never applied.
Mask duration tokens before hashing; status codes and other numbers still
split incidents.
Fixes#129587
e33181e883 stopped caching every None from _cron_interval_minutes, so a
malformed cron expression (croniter loaded, expression invalid) was
re-parsed on every in-flight-guard check. Track whether croniter actually
loaded and cache unless the None came from a failed import: a bad-expr
None is stable, an ImportError None is transient. The kept #127182
regression test now asserts the import-failure None stays uncached.
Also drop the redundant recovery_kind operand in
_recover_missing_next_run (_is_recoverable_error_job already requires a
cron/interval schedule) and tighten two comments.
Not latching a failed croniter import fixed#127182, but with nothing
cached every _ensure_croniter() call re-ran the full finder walk when the
package is genuinely absent - several times per cron job on every
due-scan tick, under _jobs_lock. Remember a monotonic retry-at 60s out
instead: a transient failure still recovers within one tick, explicit
True/False overrides still win, and a missing package costs one probe a
minute.
Also tidy the error->scheduled re-arm: reuse _is_recoverable_error_job()
for the condition and drop the redundant next_run_at assignment that
job.update(fields) already performs; fix the test comment (the patched
import raises for as long as it is patched in, not once).
Once croniter imports again, due-scan recovery re-arms a recurring job's
missing next_run_at, but the job kept state=error until it next fired, so
`hermes cron list` kept reporting a healthy, due job as broken. Flip it
back to scheduled in the same persist, and correct the docstring: recurring
jobs now also land here after a transient croniter ImportError (#127182),
not only after a hand-edited jobs.json.
Co-authored-by: itsflownium <157689911+itsflownium@users.noreply.github.com>
A single transient croniter ImportError (wrong interpreter, shadowed path)
was cached as HAS_CRONITER=False for the whole process lifetime. While
latched, compute_next_run() returns None for every recurring job, so jobs
stay enabled with next_run_at=None and state=error — never rescheduled
until a gateway restart, despite the due-scan recovery path existing
exactly for that case (#127182).
Keep HAS_CRONITER None on failure so the next call re-probes; an explicit
monkeypatched value still wins. Also log sys.executable with the warning so
a scheduler running under the wrong interpreter is diagnosable from
errors.log.
Fixes#127182
(cherry picked from commit bb48a7b48fb28d45d63c47b59a1cf0c5049efb87)
Replaces the utils-only rename test with one parametrized over a writable
target dir (publish by rename, no copy) and a read-only target dir (save still
succeeds via the copy fallback). Since all five writers stage through the
helper, this covers the shared rule instead of one call site. The cron test now
patches the utils seam (os.replace via utils, utils._copy_fallback) that
jobs.json actually publishes through.
_stage_jobs_payload is the only stager, so it owns the "temp beside the
resolved store" rule instead of its caller wrapping the path. If the link
target's directory is not writable to us (file writable, dir not), fall
back to staging beside the link: saves keep working via atomic_replace's
copy fallback, exactly the pre-fix behaviour, instead of failing outright.
Move the symlink regression test next to the other save-path tests in
test_jobs_shrink_merge.py with module-level imports.
When jobs.json is a symlink into another directory (e.g. a different
filesystem), the temp file was created next to the symlink. The rename
onto the real target then failed with EXDEV and atomic_replace fell back
to copying in place, so a crash mid-copy left a torn, unparseable store.
Stage the temp beside os.path.realpath(jobs_file) so the publish stays a
same-directory atomic rename. atomic_replace, its EXDEV/EBUSY fallback
(still needed for single-file bind mounts) and the #60703 lock-timeout
degrade are unchanged.
Salvaged from #126237 (narrow hunk only).
The context_from reader in cron/scheduler_prompt.py validates archives
against _PROMPT_HEADING/_PROMPT_SEPARATOR/_RESPONSE_HEADING/
_RESPONSE_TERMINATOR, but the writer in cron/scheduler.py still spelled
the same bytes as literals. If one side changed, framed archives would
silently stop validating. The writer now composes the header and body
from those constants; output is byte-identical (checked by rendering
documents with the old and new writer and cmp-ing them).
Also collapse `not candidate or candidate.isspace()` to the equivalent
`not candidate.strip()`, and drop the two `assert final == answer`
lines in the extraction tests: they only checked that the stubbed agent
echoed its own mock response, not any cron behaviour.
The framed-archive change moved the caller's .strip() into
_archive_answer's framed/legacy branches only, so headingless
(script-mode / FAILED) archives reached _prepend_context_block with their
padding, putting stray blank lines inside the fence and spending clip
budget on whitespace. Return archive.strip() in that branch; the
script-mode test now asserts the exact trimmed fence.
Fold the review cleanups for the frame format so writer and reader
cannot drift: the stamp labels live once in scheduler_prompt.py (it
binds _sched only at import tail, so module-level patterns cannot be
built from scheduler attributes) and are imported by scheduler.py;
_PROMPT_FRAME_RE/_RESPONSE_FRAME_RE are compiled once; the +2/+1 magic
offsets are named separators; run_job normalizes newlines once via
_normalize_newlines and builds both stamps (header takes prompt_stamp
instead of a bool flag); the empty check uses isspace() instead of a
stripped copy; the docstring is rewrapped; both writer tests share a
_run_stub_job helper.
The salvaged stack added three tests for response framing. Keep the
writer->reader round trip with nested frames (red on main: rpartition
injects only the embedded subsection) and the truncated-outer-frame
case. Drop the test and the in-test block that exercise "framed archive
without a prompt length": that shape was only ever produced by the PR's
own intermediate commits, never by a writer on main, and the follow-up
removes the reader branch for it.
Preserve the optional-value disambiguation from #129282 while removing its flag-count cutoff. Make token and whitespace runs possessive, and retain standalone dash values. Bound regression probes in subprocesses so a GIL-holding regex cannot wedge the test runner.
Co-authored-by: Yuan Li <dskwelmcy@163.com>
Extend Finn763's dead-owner recovery with owner-fenced failure delivery based on PR #120391 and retain stderr until the parent consumes an escaping worker traceback. Preserve unknown side-effect state and do not retry automatically. Cover both shared entry paths, an exception through the payload lifecycle, abrupt death, and concurrent recovery.
Co-authored-by: Denis Dresvyanskiy <dresvyanskiydenis@gmail.com>
A restart-safe cron worker that dies AFTER adopting its attempt was reported as
"Scheduler restarted after this execution's owner exited" and treated as success.
recover_interrupted_executions() only knows that an owner is gone, so that is all it
can write -- but the waiter holding the worker's Popen knows the owner was that
worker and knows its exit status. Worse, the sweep leaves the row terminal, so the
waiter returned True, run_one_job short-circuited, and mark_job_run never ran: the
job's fire claim outlived its lost execution and refused the next manual fire with
"Job is already being fired" for the whole 300s lease, with an empty last_error.
- executions.terminalize_dead_owner(): terminalize ONE attempt with an observed
cause. Refuses a live owner, this process, and a handoff inside the adoption
grace, so it can never steal a running worker's attempt; returns False so the
caller keeps the generic sweep.
- The waiter records that cause and RAISES, routing the existing post-handoff
bookkeeping: mark_job_run records the run and retires the claim, no dispatch
incident and no false crash notice. Also folds the worker's captured stderr into
the error, which the pre-ack path already did (#112729) and this path dropped.
(cherry picked from commit d3f3571f8c6f4d2f188e90151d8f8393f049a573)
The dashboard's manual Run now binds the sibling profile via
web_server_cron._cron_store_scope, not the ticker's _profile_cron_scope,
so the marker-based routed_profile_fire() stayed False there: the routed
profile's .env was reloaded into the shared os.environ and the managed
external-worker handoff serialized multiplex_active=False.
routed_profile_fire() now compares the fire's active HERMES_HOME with
get_routing_process_hermes_home(), so every entry point (ticker,
dashboard Run now, CLI, gateway forward) and the external-worker handoff
share one answer. Regression test drives the real dashboard fire path
A -> B -> A with no_agent jobs.
Answers the P1 review on #111187: build_profile_secret_scope() held only
<profile>/.env plus that profile's external-source snapshot, never the
administrator-managed .env. The launch process applies that file LAST with
override (_apply_managed_env), so a managed key beats the user's own value in
os.environ. Under multiplex semantics get_secret() stops falling back to
os.environ on a scope miss, so inside a routed cron fire (and equally inside a
real multiplex gateway turn, which builds its scope through the same function
via gateway/run.py::_load_profile_secret_scope) a managed-only credential
resolved as absent and a managed-vs-user collision resolved to the USER value:
reversed precedence.
Fix at the source: build_profile_secret_scope() overlays load_managed_env()
last, after the profile .env and external sources, skipping process-global
names exactly as it does for the other two layers. Every multiplex-authoritative
scope (gateway turn, routed desktop fire, external worker env build) is built
here, so managed authority is composed once instead of restored per consumer.
No generic ambient-env fallback is reintroduced: only the managed file's own
keys enter the scope, and only with the managed file's values.
Regression (parametrized, two invariants): inside a routed fire a managed-only
key resolves through get_secret(); a managed-vs-user collision yields the
managed value.
(cherry picked from commit cedf4a3d78)
(cherry picked from commit 4a750b4022ff321863d6746ec39ec5ca67885c16)
The earlier trim dropped the only test of the `routed_profile_fire() and
not is_multiplex_active()` branch in the external-worker handoff: the
process flag is OFF (a desktop tick is not a multiplexer) yet a fire routed
to a sibling profile must serialize `multiplex_active=True` and hand the
worker an env without the launch profile's residue, and the context must
not outlive the handoff span. Without it that branch could regress to the
pre-fix behaviour unnoticed.
(cherry picked from commit 5850a50a81)
(cherry picked from commit 440bee6ccf9d48eecf91769787f6daa3fd3ea00c)
The PR shipped nineteen tests across six files, most of them variations of
one boundary. Keep the three that pin distinct behaviour:
- a routed desktop-ticker fire runs under multiplex semantics for exactly
its scope: a scope miss returns None instead of the launch credential and
the parent os.environ is byte-identical afterwards;
- a routed no_agent child never sees a launch-only name, whether the launch
.env defined it or a launch external source supplied it (applied or lost
to a pre-existing process value), while its own values come through;
- administrator-managed keys keep policy precedence over the routed
profile's own value.
Everything else was either a positive control of the same seam, a
set-membership check on a module-level constant, or a re-statement through
a different entry point.
(cherry picked from commit 042b9830b4)
(cherry picked from commit 0b82557fbdbfa6968b2b795e4e63d484ffb582b1)
Review findings on f5f88d5058. Three are defects the previous round introduced.
Managed keys were stripped as launch residue. Recording every dotenv load as
residue swept in the administrator-managed `.env`, which `_apply_managed_env`
applies LAST with override precisely so it beats the user's own `.env`. A
routed child then lost `ORG_POLICY_FLAG=managed-value` to the routed user's
`user-value`. Managed keys are now recorded separately, never enter the
residue set, and are re-applied over the routed scope in both child builders
(`scheduler_script`, the restart-safe handoff) so the child sees the same
precedence the launch process does. `kanban_db_dispatch` and
`scheduler_delivery` strip without any overlay, so for them the exclusion
alone is the guarantee; the test pins the case that exercises it — the same
key defined in both the user and the managed file.
Private hydration did not record supplied names. `_hydrate_profile_secret_sources`
now feeds `provenance` plus `skipped_existing` into the same ownership set the
process-global path uses; the provenance label map stays applied-only.
Removal cleanup cleared its marker before the fallible work. A raising
reload left the removed plugin's credential active with no retry, because the
next no-source discovery saw the flag already false. The marker is cleared
only after reset, reload and installed-scope refresh succeed.
Routed fire not multiplexed at the handoff. `run_one_job` enables the
context in `_install_fire_secret_scope`, which runs AFTER
`_launch_external_cron_worker`, so a routed desktop fire on the managed path
serialized `multiplex_active=False` and built the worker env with launch
residue and no scrub. The handoff now treats `routed_profile_fire()` as
multiplexed for exactly its own span; the worker re-establishes the state from
the payload as before.
Each fix was checked by reverting it and confirming its regression fails,
including the overlay half and the exclusion half of the managed fix
separately.
(cherry picked from commit 329cbd8963d68c45b425e95a5b11ade59f513960)
(cherry picked from commit 62b4488cb5)
(cherry picked from commit d3319102a7e4dcce35ec2676fe22aed41613e893)
Review findings on d8c467f223, each reproduced through its production path.
Stale launch key. `strip_launch_profile_env` built its residue set from a
re-parse of the launch `.env`. A key removed or renamed in that file after
boot is still in `os.environ` with the old value (dotenv never unsets), and
the current file no longer names it, so it survived into the routed child.
`_load_dotenv_with_fallback` — the one chokepoint every dotenv load goes
through — now records the KEY names it put into the process env, additive for
the process lifetime (`launch_dotenv_keys()`), and the strip unions that record
with the current file.
Source name that lost to the process env. `_apply_external_secret_sources`
snapshots every name a source SUPPLIED (`provenance` + `skipped_existing`),
but `secret_source_names()` only exposed `_SECRET_SOURCES`, which is
provenance metadata and names applied values alone. A launch-profile source
that supplied `CUSTOM_VAULT_SECRET` while the process already had it was
therefore invisible to the scrub, and a routed child with an empty scope got
the launch value. Supplied names are tracked separately
(`_SOURCE_SUPPLIED_NAMES`) so the provenance labels stay honest, and
`secret_source_names()` returns the union.
Last plugin source removed. `_refresh_secret_sources_after_discovery`
returned before the cache reset and the installed-scope refresh whenever no
plugin source was enabled — and `discover_and_load(force=True)` unloads the
old registration first, so removing the final plugin source hit exactly that
return with the removed plugin's names still in the per-home snapshot and the
current scope. The manager now remembers that a discovery re-applied plugin
sources and, on the next discovery that finds none, reconciles once. A home
that never had a plugin source is still a no-op (pinned by the existing tests).
Regressions: the stale-key lifecycle and the skipped-existing case through
`_run_job_script` against a real routed child, and the removal case through
the manager. Each checked by reverting its fix and confirming the test fails.
(cherry picked from commit d464f5f6126a394cfb47937f683d3a5e2f141840)
(cherry picked from commit 9d7de6c140)
(cherry picked from commit b0beb731b18edd884e36a30ab91c91687bc7a965)
The source-name strip added alongside the external-source fix ran
unconditionally. Outside multiplexing there is no other profile to leak from --
os.environ IS this profile's environment -- so popping those names relied on the
routed scope overlay putting each one back, which in turn relies on the
per-home snapshot recorded at boot. Correct today, but it made a single-profile
child's credentials depend on bookkeeping that has nothing to do with isolation.
Guard it the way strip_launch_profile_env guards itself: no multiplexing, no
strip. A single-profile no_agent child now keeps a byte-identical env even if a
source's snapshot were ever missing. Pinned by a regression that runs a real
child with a source-owned name in os.environ and no multiplex context; making
the strip unconditional fails it.
(cherry picked from commit afa429b30a1c9c9b4c011f7d2426a9f099911596)
(cherry picked from commit 3dedff6a12)
(cherry picked from commit 1cb2a8e6eb9a1848fa758fcaef7dce531532258c)
Two credential-isolation gaps found in review of the previous head.
1. strip_launch_profile_env() only knows dotenv- and terminal-config-owned names,
but external secret sources (vault, 1Password, ...) also write their names into
the shared os.environ and are tracked in secret_source_names(). A name the LAUNCH
profile's source supplied therefore still reached a routed no_agent child. Drop
every non-global source-owned name from the base; the routed scope overlay that
follows puts back exactly the ones that profile's OWN sources supply, since
build_profile_secret_scope folds get_secret_source_values(home) in.
2. refresh_installed_secret_scope() merged the rebuild with dict.update(), so a
name a source had stopped supplying -- rotated, revoked, source removed -- kept
its old value for the rest of the fire. Replace the mapping contents instead: the
rebuild is the profile's current truth.
Regressions: a routed child sees <unset> for a launch-source name while its own
source value comes through, and a refresh whose rebuild omits a name drops it.
Both fail if the corresponding change is reverted.
(cherry picked from commit ecd51517c4828a75acb5f458ed99aea0bc3e5e9f)
(cherry picked from commit 0943e77136)
(cherry picked from commit 68ab0384e17e06e8eca684c4d8eac15543dc3943)
The routed no_agent child env started from all of os.environ and only overwrote the
names present in the installed scope. A name defined only by the LAUNCH profile's .env
and absent from the routed profile therefore reached the routed child with the launch
value instead of unset -- the secret scrub only knows classified names, so a custom or
unclassified secret crossed the profile boundary (review finding on the first head).
strip_launch_profile_env (main, 284d220ba4) is the primitive the external-worker path
already uses for exactly this: it drops the launch profile's dotenv-owned keys and the
bridged TERMINAL_* settings, and is a no-op outside multiplex or when the target IS the
launch profile. Apply it to the base BEFORE the scope overlay (so a shared name keeps its
routed value) and BEFORE the sanitizer (so routed values still pass the scrub and
passthrough rules). Pinned by a child-process negative control: the launch-only name
arrives <unset>, the shared name arrives routed, and the parent process is unchanged.
(cherry picked from commit 69349b527b5144c1f1540a4c10035d5ec798c1db)
(cherry picked from commit 023e4f997f)
(cherry picked from commit cace4f3f2013e9dc60356add245b2bad4b56ab08)
The desktop backend ticks EVERY local profile's cron store from one process — its own docstring
says "like a multiplex gateway" (hermes_cli/web_server.py) — but never sets the process-global
multiplex flag, and cannot: its own chat turns are unscoped and would fail closed. Every
isolation in the tree keys on that flag — the guard that keeps a routed `.env` out of the shared
`os.environ`, `get_secret`'s fail-closed miss, passthrough resolution, the MCP and kanban
subprocess scrubs — so all of it was inert for a sibling profile's fire. Verified: a secondary
profile's API keys replaced the launch profile's in `os.environ` with `override=True` and stayed
there after the tick, and a scope miss read the launch profile's tokens (#107692).
Give multiplex mode a context-local counterpart. `set_multiplex_context` (agent/secret_scope.py)
is OR'd into `is_multiplex_active()`. `_profile_cron_scope` only MARKS a fire whose home is not
the process's own (`routed_profile_fire`, decided against `get_process_hermes_home()`, the
override-immune resolver); `_install_fire_secret_scope` in cron/scheduler.py installs the
profile's hydrated secret scope and, for a marked fire, the multiplex context — for exactly that
span, dropped again before the scope by `_reset_fire_secret_scope`. Multiplex semantics are
therefore never active in cron without a scope to read: `run_one_job`'s restart-safe handoff runs
before the body's scope and keeps today's semantics (its own scope is #107413 / #106050's seam,
left untouched so this composes with whichever lands). Every existing multiplex-keyed isolation
applies inside the routed fire with no per-site patching; the launch profile's own fires and the
backend's turns keep single-profile semantics; marker and override both reach the pool worker via
`copy_context()`. `get_secret` read the raw global in its miss branch; it now goes through
`is_multiplex_active()`. The dotenv guard keeps its pinned flag-only form (#77970).
Two consequences of suppressing the write are handled rather than left as regressions:
- a `no_agent` script's env is `os.environ.copy()`, which no longer carries the routed `.env`;
the runner overlays the installed scope onto the base BEFORE sanitizing, so the same scrub /
passthrough rules apply to those values and the parent process is never mutated;
- plugin secret sources are discovered on the fire's first agent build, after the scope froze,
and the post-discovery reload is hydrate-only under multiplex semantics; the refresh now folds
the values into the installed scope in place (`refresh_installed_secret_scope`, the pattern
`_publish_env_value` already uses for `.env` writes under multiplex).
And the profile's external secret sources are hydrated before the scope is frozen, the order
gateway/run.py and the external cron worker already use.
Tests pin each direction: the marker without the semantics before the scope, the semantics on and
off exactly with it, the marker reaching a copy_context worker; the process's own profile staying
single-profile; the restart-safe handoff's child env building without raising under a routed tick
with a passthrough key registered; a real child process receiving the routed values while
`os.environ` keeps the launch value; a source registered after the freeze reaching the fire
through the real PluginManager refresh. Reverting any one direction fails a distinct test.
(cherry picked from commit 2f87677425d2cca19286ac83bc45cab23e546669)
(cherry picked from commit dbede34f6e)
(cherry picked from commit a7ac197ff12710e045e3f75d960b20be952960f3)
Each removed test either asserts nothing reachable or repeats another
test's exact body:
- auxiliary_client empty-base_url: body only writes config.yaml; a merge
(890bbbda1f) restored the header of a test b3d4f67b20 had purged. The
non-Anthropic-host test beside it drives the same default fallback.
- background_review no-session-db: _record_review_usage_to_parent swallows
every exception and there is no DB to observe, so removing the guard
still passes; the path has no observable effect to pin.
- code_execution Windows "would have failed" pair: every branch returns,
skips or passes; they exercise CPython defaults, not Hermes code.
- cron test_should_use_direct_api_call_*: byte-identical copy of the test in
tests/agent/test_cron_inline_api_call.py (0213b8b77b had to edit both).
- cron liveness no-multiplexer: identical body to test_no_lock_no_pids_is_false;
its multiplexer patch is never reached.
- browser console/get_images "private URLs allowed": identical to the
local-backend test in each file; both mock the predicate the name promises.
- hermes_state TestTitleUniqueness: the duplicate-title case is a strict
subset of TestSessionTitleLineage::test_unrelated_session_still_conflicts;
the NULL-title case never writes a title.
worker_bootstrap() logged a failed activate_dependencies() and let the worker
carry on through the inherited PYTHONPATH pin. That path is not leased: once
the gateway releases its generation the collector deletes it under the live
worker and the next dependency import dies mid-job (#122936 review, reproduced
with a committed generation that has no site-packages).
The failure now propagates, so the worker exits before its ownership ack and
_launch_external_cron_worker reports the dispatch failure with the worker's
stderr. PM's own no-op cases (venv, wheel, Nix, nothing committed under a venv)
still return normally.
Tests drive the real PM boot in a marked child: it imports a sentinel that
exists only in the committed generation, and when an update commits a newer
one mid-job the real collector keeps the worker's generation until the worker
exits. A damaged selection exits before cron.jobs is imported.
123456hyu made the same prelude move and fail-closed change on #122290
(dff16e12); this carries them without the argv gate and source-order test.
Co-authored-by: 123456hyu <123456hyu@users.noreply.github.com>
- Rename the boot marker HERMES_CRON_EXTERNAL_WORKER to _HERMES_CRON_WORKER_BOOT.
The old name differed by one leading underscore from
_HERMES_CRON_EXTERNAL_WORKER, the owning execution id read in the same worker.
- The subprocess probe now gets the marker name from WORKER_MARKER instead of
a literal, so a future rename cannot silently skip the consumed-marker check.
- Resolve the repo root inside worker_bootstrap(): only the marked worker uses it.
- Correct the docs to match the boot order: the boot re-selects and leases the
generation before any third-party import, and the PYTHONPATH pin is the
fallback for a failed boot. They are not two load-bearing halves, and the pin
does not resolve modules exactly as the gateway did.
`-m cron.scheduler` runs `cron/__init__.py` before the module itself. That
file's first import, `cron.jobs` -> `hermes_yaml` -> `ruamel`, is already a
third-party dependency. With the boot at the top of `cron/scheduler.py`, the
worker imported through the pinned PYTHONPATH alone:
- no lease on the generation was held yet;
- no `.pth` files had run;
- with the dependency pin removed, it still died with
"No module named 'ruamel'".
Run the marked-worker boot at the top of `cron/__init__.py`, under a private
alias, so the package attribute `cron.worker_bootstrap` stays the module. Drop
the now-redundant call in `scheduler.py`. Replace the in-process marker test
with a subprocess test: the boot runs before `cron.jobs` is imported, runs
exactly once, consumes the marker, and never fires for an unmarked importer.
Also add the missing trailing newlines on the two new files.
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes
entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its
interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the
subprocess sanitizer has just stripped, so the worker died at its first dependency import
(`No module named 'ruamel'`, #122222) before it could publish its ownership
acknowledgement -- every scheduled job landed `last_status: failed`.
Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency
site-packages and leased the generation at the worker entry. Reworked onto PM's own
machinery:
* `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` +
`site_packages()` for the generation, instead of deriving one from this process's
`sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active
home, so the child-spawn path no longer reads the real home -- that probe tripped
`tests/home_io_guard.py` when the suite runs from a self-managed install -- and the
uv-launcher, first-match-only and ordering edge cases of the old derivation go with it.
* `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease +
`site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all
`pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it
holds the generation's lease for the process's life. It is gated on the
`HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than
sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch
contract without argv identity inference.
Verified live on PM's store Python against a real generation (A/B, same child): tree-only
PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports,
`running_from_selected_environment` True, and the worker holds a real kernel lease while it
runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now
reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed.
Tests: the two invariants that carry the contract -- the pin restores the committed
generation (and invents nothing without one), and the boot runs only in the marked worker,
once. The six construction tests that drove the old derivation are dropped with it.
Duplicate-cluster credit: #122238 (@webtecnica, earliest), #122382 (@Sakujakira), #122685
(@etrisnanto), #122719 (@aglowinthefield) and #122738 (@GabboAlarcon) diagnosed and fixed
this same worker-dependency failure in parallel while this branch was rebuilt. None of their
diffs is contained here -- the co-author trailers below record direction and duplicate-cluster
credit, not lifted code; their roles are listed in the PR body.
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com>
Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com>
Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com>
Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>
(cherry picked from commit 7c55715c0478280adf4e492b24edf331342470f7)
The pin in scheduler_worker_env.py restores the site-packages *directory* into the
child's PYTHONPATH, but a directory name is not a retention permit. The PM collector
removes an unselected, lease-managed generation once it is old enough and no process
holds its lease, so between the gateway's exit and the worker's next import the very
generation the worker needs can be collected away.
New cron/worker_bootstrap.py closes that gap at the worker entry, before the first
application/dependency import:
- lease_dependencies(): acquire the committed generation's kernel lease (the same
hermes_cli.runtime_state.lease_generation primitive activate_dependencies uses),
so the collector's 'no process holds the lease' safety condition keeps the
generation alive for the worker's whole lifetime. Unconditional and idempotent.
- activate_worker_dependencies(): delegate to pm.environments.activate_dependencies
(the same resolution + site.addsitedir + sys.path reordering the gateway's boot
uses) so the worker's own imports resolve without waiting for the gateway's
handoff. Gated on --external-worker-file in argv, so the in-process gateway path
keeps its exact launch contract and the runner-owns-its-own-deps case stays a
no-op.
Hooked into the top of cron/scheduler.py, after the stdlib-only sys.path pin and
before any Hermes package import, so hermes_yaml -> ruamel.yaml succeeds. Both
degrade to a logged no-op when no dependency generation is committed.
Tests: test_worker_bootstrap_lease_and_activation, _leases_but_skips_activation_in_gateway,
_no_committed_venv_is_noop, and a scheduler-import smoke test. The two home_io_guard
failures in this suite are pre-existing on a clean checkout and unrelated.
Fixes the P2 lifetime finding in @andrexibiza's review of #122290.
(cherry picked from commit 429f0dd191dbf719d05cbbbe3cda6546893b97d0)
On a self-managed install the restart-safe cron worker is spawned with the store Python, which owns no third-party dependencies. The sanitizer drops the runtime site-packages and the tree pin only re-adds the checkout, so the worker dies at its first import (No module named 'ruamel') before publishing its ownership ack.
Restore the dependency environment this process actually activated, next to the tree, derived from sys.path (never a home-scoped read) and gated on a real venv so nothing is invented for installs that already own their dependencies.
Fixes#122222
(cherry picked from commit 8403cd5681b8da2a52061e3ae16c5533ef3b5ff3)
The gateway and cron ticker were blind spots in shared metrics: we could not
tell which messaging platforms fail to connect or drop, how often replies fail
to reach users, how long users wait for a first reply, or whether scheduled
jobs run, fail, get skipped or get missed while Hermes was down.
New opt-in counters (all through the existing enabled() / record_process_mark
gate, recorded fire-and-forget on one background worker that runs in a copy of
the caller's context so the owning profile gets the row):
- hermes.platform.health: platform, event (connect_ok/connect_failed/
reconnect/disconnect), error_class. One seam in the runner
(_connect_adapter_with_timeout, used by cold start, multiplex secondaries and
the reconnect watcher) plus the fatal-error handler; classified from
exception types, HTTP statuses and Hermes's own fatal codes, never text.
- hermes.platform.delivery: platform, outcome, failure_class. One row per
logical BasePlatformAdapter._send_with_retry call (retries and plain-text
fallback included), from SendResult's typed fields or the exception type.
- hermes.gateway.reply_latency: platform, first_response_bucket. Clock starts
when _handle_message accepts a non-internal turn; stops on the stream
consumer's first delivered text or send_final_ledgered. Busy acks and
command replies never stop it.
- hermes.cron.run: outcome (success/failed/missed/skipped), delivery_kind,
duration_bucket. Recorded at the write-once terminal execution row
(finish_execution) and where the due scan drops an occurrence (catch-up
disabled, expired one-shot). Job names, prompts and targets never leave.
Platform names: platforms Hermes ships under plugins/platforms/ now report by
name, and a plugin-catalog platform reports its catalog entry name only when
the installer-owned .install-metadata.json record proves a catalog install of
the plugin dir that defines the registered adapter factory (a URL install
cannot claim a catalog name via its tree or manifest). Everything else stays
"plugin". GATEWAY_PLATFORMS becomes catalog-backed (_CatalogValues), and the
v3 schema's platform fields accept published catalog identifiers.
The cron SQLite stores write hermes_time.now().isoformat(), an ISO string
that carries the local UTC offset. That offset changes at a DST transition
and after a timezone config change, so the text order of these columns is
not their time order. Across a fall-back, 01:10-05:00 is 20 minutes after
01:50-04:00 but sorts before it. latest_execution(), latest_executions()
(which fills job["latest_execution"] for `hermes cron list` and the
dashboard), list_executions() and its before_claimed_at cursor, the
delivery queue's pending-claim order, delivery and execution pruning, and
list_incidents() all returned the older row first.
Every ORDER BY and the cursor comparison on these columns now use
julianday(col), which SQLite resolves to the same instant for any offset.
The raw text stays as the next sort key, so rows within the same
millisecond (julianday's resolution) keep their microsecond order, and the
cursor compares the same (instant, text) key on both sides. Stored values
and the schema are unchanged, so existing rows with mixed offsets order
correctly without a migration.
latest_executions() moves from a correlated per-row subquery to one
ROW_NUMBER() window. With julianday() in the ORDER BY the index on
claimed_at no longer serves the per-row LIMIT 1, and the correlated form
took about 90 ms at 1000 rows; the window form takes about 1 ms.
Refs #86520
Thanks to #86889 (zhao0112) for the executions-ledger diagnosis.
Keeps #121950's one-line fix — `discover_plugins()` under the payload's home override before
`hydrate_profile_secret_sources()` in `_run_external_worker_payload` (L3 ticker -> L5 child
process; T2/T4) — and trims around it: the comment keeps only the WHY, and the two salvaged
tests collapse into one invariant that proves the owning-profile scoping the issue asks for:
the worker's launch HERMES_HOME has no plugins, the payload's profile home ships a
`register_secret_source` plugin, and the plugin's value reaches the secret scope seen by
`run_one_job`. The bare-profile control case was already covered by
`tests/cron/test_restart_safe_worker.py::test_external_worker_adopts_execution_and_runs_payload_once`.
Trim of #121595: instead of consulting the managed file first and the raw user file second
through two new helpers, `_primary_profile_routes_for_current_home` now reads the primary home
through `gateway.config_loader.read_yaml_layers` — the exact user-file + managed-overlay layering
`load_gateway_config()` uses, so the satellite side cannot drift from the gateway's own routing
view (L8: behavioural read through the surface's loader, never `read_user_config_raw`). The
absent-user-file early return goes too: a fleet host with only `/etc/hermes/config.yaml` must
still see its routes, exactly as the gateway loader does.
Both consumers ride the one helper — preflight rescue (`_delivery_platform_routed_from_primary_gateway`)
and the delivery-time `SharedRouteAdapters` fallback in `scheduler_provider.tick_adapters_for` and
the `gateway/run.py` delivery-queue drain — so managed-scope routes stop false-blocking and stop
failing delivery closed on every satellite (T2/T4).
Tests trimmed to two invariants on real files: managed-only routes reach both preflight and the
delivery adapter map; managed routes replace user-file routes (list precedence).
External cron workers start with the builtin secret-source registry alone —
plugin-registered sources (the documented path for third-party vaults,
developer-guide/secret-source-plugin) only exist after plugin discovery runs.
Since the restart-safe topology hands every scheduled agent-mode job to a
worker process, jobs on secondary profiles under a multiplexed gateway died
at runtime resolution with 'No usable credentials found for provider ...'
while identical direct runs in the gateway succeeded.
Run the idempotent discover_plugins() under the profile home override before
hydrating: discovery registers the profile's plugin sources and re-pulls
enabled ones (#64177 bootstrap parity), the same lazy-load pattern the
delivery path already uses for plugin platforms.
Fixes#121929
(cherry picked from commit 248d441559478978cbdae1cc2178856997bef5f3)
_primary_profile_routes_for_current_home() read only the primary home's
raw config.yaml, so routes pinned in the managed scope (e.g.
gateway.profile_routes in /etc/hermes/config.yaml) never reached it:
preflight false-blocked satellite agent jobs and routed delivery failed
closed. The managed overlay is now consulted first (it wins over the user
file, as in the merged config), with the raw user read as fallback.
Fixes#121212
(cherry picked from commit 7b05243220e98a4c8e22587d51ca786492cb148b)
Drop the four unit tests that pre-set `live_error` on the fixture (they never
exercised the population path the issue is about) and keep two invariants,
both driving the real DeliveryRouter so the rejection arrives on the EXCEPTION
arm of `_deliver_via_live_adapter` exactly as `send_path_degraded` does in
production (`DeliveryRouter._deliver_to_platform` raises the SendResult error):
- a reconnect-only rejection whose standalone fallback then fails on a
token-less satellite worker is retained as a failed ledger row that
`sweep_failed_for_runtime` claims for that adapter's profile;
- any other live failure still falls through to standalone as before and
queues nothing.
