Plugins could only see a cron run as the task_id string `cron:<job>:<execution>`,
with no way to tell a scheduled fire from a manual `hermes cron run`. A
security plugin that may only act inside a real scheduled occurrence had
nothing to check.
The scheduler now binds a frozen CronExecution (job id/name, execution id,
source, scheduled_instant, started_at, profile) in a ContextVar once the run
wins its claimed->running CAS (or a restart-safe worker adopted the row), and
resets it when the fire ends. ContextVar scoping keeps concurrent jobs and
profiles apart and follows the agent onto the threads the scheduler already
copies context into. Delegated children read None.
Also trims duplicated comments in _run_one_job_body to keep the facade within
its code-health line cap.
Closes#130722
_run_one_job_body's except branch sends _deliver_crash_failure when anything in
the body raises, but the firing profile's secret scope was only installed after
claim_dispatch and mark_execution_running. A raise from either sent the crash
notice with no scope: under multiplex get_secret fails closed
(UnscopedSecretError, "Hermes could not read this profile's
TELEGRAM_HOME_CHANNEL") and the notice never left. Same class as the
dispatch-failure branch fixed for #131585 (and called out in #130431's body).
Install the scope first thing in the try so every exit through the except is
scoped; the outer finally already resets it. The handoff path stays outside it
(#107692), and the external-worker payload path and the post-handoff notice
already run scoped.
Extends the #130431 test with a claim_dispatch-raising case.
Refs #131585
Co-authored-by: skyeyesec333 <skyeyesec333@users.noreply.github.com>
run_one_job's pre-handoff dispatch-failure branch (#124013) calls
_deliver_crash_failure before _run_one_job_body, which is where the
firing profile's secret scope is installed for delivery. Under a
multiplexed gateway the notice then resolves the home channel through
get_secret with no scope, raises UnscopedSecretError, and never leaves.
Wrap the notice in _install_fire_secret_scope/_reset_fire_secret_scope,
the pair _run_one_job_body and record_unknown_worker_outcome already use.
(cherry picked from commit 5e0735b17267a636337818505bae485d7e208837)
The external worker now calls GatewayStartupMixin._register_config_hooks (the
helper gateway startup and each multiplexed profile already use) instead of a
copied load_config/register_from_config sequence, and calls it after
set_secret_scope rather than before. Registered outside the scope, a
hooks.outbound target with secret_env hits get_secret with no scope while
multiplexing, which raises UnscopedSecretError; the never-raise guard swallowed
it, so that profile's outbound webhooks were silently dropped again.
Both registries are idempotent per (home, event, ...) so a worker running
several payloads registers each hook once. The kept test gains a secret_env
target on a multiplexed payload to pin the ordering; the broken-config test is
dropped (register_from_config never raises on a malformed block, and the
helper's guard is already covered by the gateway path).
Refs #131764
The restart-safe external worker (python -m cron.scheduler
--external-worker-file) called discover_plugins() under the payload's
home override but never agent.shell_hooks.register_from_config() nor
agent.outbound_webhooks.register_from_config(), so hooks: blocks in the
owning profile's config.yaml silently stopped firing for cron sessions
once gateway fires were handed to the worker instead of running
in-process. Plugin hooks kept working, which made the gap invisible.
Register both right after plugin discovery + secret hydration, under the
same home override, with the gateway's never-raise shape (#102504):
consent resolves non-interactively from the owning profile's allowlist /
HERMES_ACCEPT_HOOKS / hooks_auto_accept, and a broken hooks block logs a
warning instead of taking the cron execution down.
Fixes#131764
(cherry picked from commit d1971c698a4429868e9d43d058032930ec62d7b9)
Register direct workers with a live future and fence dispatcher release and future attachment with per-registration identity tokens. Preserve profile-scoped release keys.
Port the in-memory fix from 251b99332550be4dc5a7ba265109e367842e9081, without timestamp-based durable-state guards.
Fixes#132223. Refs #88688.
(cherry picked from commit dd834c874fef04a7d56658e0d05baa3bee5decba)
The scheduler persists last_delivery_unverified as a list of unconfirmed
platform:chat_id[:thread_id] targets and cron.manage passes it through, but
CronJobRow typed it bool | None, so contract validation rejected any job
with an unconfirmed delivery. Regenerate the TS/OpenRPC contract.
test_ladder_reruns_do_not_spend_extra_repeat_budget only exercised the
finish and crash tails, so the dead-worker recovery fix in
scheduler_worker_failure and the ownership-lost fix had no test that fails
without them. Add "worker" (record_unknown_worker_outcome on an unknown
execution) and "ownership_lost" (transport-cancelled rung recorded as
interrupted) rows; each asserts repeat.completed stays at 1. Both rows fail
against the pre-fix source.
The previous commit stopped a re-run from the unreachable-model ladder from
counting toward repeat, but only on the normal _finish_completed_run path.
The other places that record a failed fire on the same claimed snapshot
still counted it: the crash/interrupt tail of _run_one_job_body, the
pre-handoff dispatch failure in run_one_job, and the recovered unknown
worker outcome in scheduler_worker_failure. A re-run that crashed or failed
to dispatch therefore still bumped repeat.completed a second time for an
occurrence that had already counted.
All three now pass ladder_rung=True when is_retry_run(job). The
worker-failure job is the same run_one_job snapshot, which carries
_scheduled_instant (stamped by the due scan and copied by
_process_due_job), so is_retry_run applies there too. Manual runs carry no
_scheduled_instant and still count.
The kept test gains a "crash" row: a re-run whose run_job raises must leave
repeat.completed at 1.
A recurring job with a finite repeat count (for example repeat=3, every
24h) lost one repeat slot for each automatic re-run after a fire that
could not reach the model. The failed occurrence counted, and then each
5/15/30-minute re-run of that occurrence counted again, so a half-hour
outage finished a repeat=3 job with zero model calls. With
cron.retry_unreachable off, the same outage costs only the one failed
occurrence.
Cause: cron/jobs.py:_advance_after_run bumps repeat.completed on every
mark_job_run, and the re-runs parked by
cron/unreachable_retry.py:plan_retry go through it like scheduled
occurrences. cron/unreachable_retry.py:will_retry also treated a re-run
on the last slot as the job's final run.
Fix: cron/scheduler.py:_finish_completed_run passes ladder_rung=True
when the dispatched instant is the parked re-run instant (new
is_retry_run, which compares the dispatch snapshot's _scheduled_instant
with the ladder state). mark_job_run then calls _advance_after_run with
count_run=False, and will_retry leaves that run out of its final-run
check the same way. The first failed occurrence still counts, and manual
runs still count.
(cherry picked from commit 27038f7064bb611811efee8c527dddecd4fcb57b)
A failed cron run records last_status=error and bumps failure_streak
correctly, but the next successful run resets both in the same pass and
nothing durable records that a failure happened — a silent watchdog-style
script that fails every Nth tick collapses to a single tick of visible
failure, and monitors sampling jobs.json report green forever (#118354).
Stamp job[last_failure] = {at, detail} on every failed run, latest
failure wins, and do NOT clear it on success: last_status keeps
describing current health while last_failure preserves the recency
evidence for dashboards and cron list. cron list now prints a warning
line for the stamp (marked recovered when last_status is healthy).
Fixes#118354🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
Keep newly observed sleep credit until the following poll because the supplied idle duration can predate that sleep. Extend both contributor regression tests across the sampling boundary and qualify the platform guarantee.
Add AwakeIdleMeter, which reads the wall clock and time.monotonic() on
each watchdog check. monotonic pauses during sleep on macOS
(mach_absolute_time) and Linux (CLOCK_MONOTONIC), so the gap between the
two clocks is time spent asleep, and that sleep is not charged as idle
time until the agent records new activity. The cron inactivity watchdog
and the turn liveness watchdog both use it, because on main both guard a
cron run with a 600 s wall-clock bound. A job that is really idle for
600 s awake is still killed.
Fixes#131523
(cherry picked from commit 07d9ddc24736e243d72327ff90a24f95e86119d8)
A scoped or wedged daily-brief in profile A must not hide a degraded or
live daily-brief in profile B. Split the scheduler's in-flight keys before
counting and add awaitable cron runs independently of non-cron work.
Carry the per-run restart_safe flag to the observer and preserve the
existing bare-ID accessors for unrelated callers.
Thanks to @ehz0ah for diagnosing the multiplexed collision and proposing
the profile-keyed accounting and mixed scoped/degraded regression case.
Preserve @Superfly73's source commit separately; map their verified bare
noreply email for the contributor CI gate.
Refs #130987
Co-authored-by: z1gon <130889443+ehz0ah@users.noreply.github.com>
A cron run handed to an external worker in its own transient systemd scope
outlives the gateway: the shutdown drain deliberately leaves it unmarked
(``mark_running_jobs_interrupted``) and its final send rides the durable
delivery queue, drained by whichever gateway is live next. The after-turn
restart wait still counted it as active work, so ``/restart``, SIGUSR1 and
``hermes update`` held the gateway in ``draining`` — refusing every new turn —
for up to ``agent.restart_after_turn_timeout`` (30 min by default) while the
worker kept running regardless. Observed in the field: a 100-minute job turned
a restart into a ~30-minute lockout.
- ``cron.scheduler`` records whether an accepted handoff got its own scope
(``dispatch.mode == "scoped"``) and exposes
``get_restart_safe_external_job_ids()``; runs already reported as wedged are
excluded there too, because the gateway subtracts both counts and a run in
both sets would be subtracted twice.
- ``_awaitable_work_count()`` subtracts that count as a second, disjoint
exclusion; ``_describe_active_work()`` marks the unit (``restart_safe``).
- A degraded worker (external, but no reachable user bus, so it shares the
gateway cgroup) is NOT listed and is still awaited — the restart would kill
it mid-run.
- The wait and observation log lines now name both exclusion classes instead of
claiming everything still listed will be interrupted.
Tests: tests/gateway/test_restart_drain.py (skip + control),
tests/cron/test_restart_safe_worker.py (registry contract).
Docs: updating.md restart section, cli-config.yaml.example.
(cherry picked from commit 6a2f0b77cf07a60510430fc4ef8b75539ee422fb)
The router turned every failed SendResult into a bare RuntimeError, which
dropped raw_response, and the cron live lane falls back to the standalone
sender on any error. A split send that failed after delivering its head
(Telegram, Discord, and now SMS/WeCom) was therefore delivered again from
the start by the standalone lane.
The router raises PartialDeliveryError for a partial_overflow result; the
live lane records "delivered N of M chunks, then failed" on the run and
does not fall back.
_live_send_text used future.cancel() to tell a send that never started
(wedged loop: fall back to standalone) from one in flight (assume delivered:
a resend would duplicate). A run_coroutine_threadsafe future stays PENDING
until the coroutine finishes, so cancel() returned True mid-send: the
in-flight send was cancelled part-way and the standalone lane then sent the
whole payload again. Paced multi-chunk sends (WeCom's 30 msgs/min bucket)
outlast 60s routinely; a 55-chunk cron payload arrived as 25 chunks and was
reported failed.
The send now records its own start under a lock; on timeout it is
abandoned (and the standalone fallback runs) only when it never began.
The existing in-flight test faked the state with a mocked cancel()
returning False, the exact assumption that does not hold; it is rewritten
against a real event loop, with the never-started (wedged loop) case it
never covered alongside.
The inactivity watchdog polls every 5s, so the same stall reports
"idle for 601s" on one run and "idle for 604s" on the next. The incident
signature hashed that raw number, so each run minted a new incident: an
acked timeout pinged again and the repeat-alert cooldown never applied.
Mask duration tokens before hashing; status codes and other numbers still
split incidents.
Fixes#129587
e33181e883 stopped caching every None from _cron_interval_minutes, so a
malformed cron expression (croniter loaded, expression invalid) was
re-parsed on every in-flight-guard check. Track whether croniter actually
loaded and cache unless the None came from a failed import: a bad-expr
None is stable, an ImportError None is transient. The kept #127182
regression test now asserts the import-failure None stays uncached.
Also drop the redundant recovery_kind operand in
_recover_missing_next_run (_is_recoverable_error_job already requires a
cron/interval schedule) and tighten two comments.
Not latching a failed croniter import fixed#127182, but with nothing
cached every _ensure_croniter() call re-ran the full finder walk when the
package is genuinely absent - several times per cron job on every
due-scan tick, under _jobs_lock. Remember a monotonic retry-at 60s out
instead: a transient failure still recovers within one tick, explicit
True/False overrides still win, and a missing package costs one probe a
minute.
Also tidy the error->scheduled re-arm: reuse _is_recoverable_error_job()
for the condition and drop the redundant next_run_at assignment that
job.update(fields) already performs; fix the test comment (the patched
import raises for as long as it is patched in, not once).
Once croniter imports again, due-scan recovery re-arms a recurring job's
missing next_run_at, but the job kept state=error until it next fired, so
`hermes cron list` kept reporting a healthy, due job as broken. Flip it
back to scheduled in the same persist, and correct the docstring: recurring
jobs now also land here after a transient croniter ImportError (#127182),
not only after a hand-edited jobs.json.
Co-authored-by: itsflownium <157689911+itsflownium@users.noreply.github.com>
A single transient croniter ImportError (wrong interpreter, shadowed path)
was cached as HAS_CRONITER=False for the whole process lifetime. While
latched, compute_next_run() returns None for every recurring job, so jobs
stay enabled with next_run_at=None and state=error — never rescheduled
until a gateway restart, despite the due-scan recovery path existing
exactly for that case (#127182).
Keep HAS_CRONITER None on failure so the next call re-probes; an explicit
monkeypatched value still wins. Also log sys.executable with the warning so
a scheduler running under the wrong interpreter is diagnosable from
errors.log.
Fixes#127182
(cherry picked from commit bb48a7b48fb28d45d63c47b59a1cf0c5049efb87)
Replaces the utils-only rename test with one parametrized over a writable
target dir (publish by rename, no copy) and a read-only target dir (save still
succeeds via the copy fallback). Since all five writers stage through the
helper, this covers the shared rule instead of one call site. The cron test now
patches the utils seam (os.replace via utils, utils._copy_fallback) that
jobs.json actually publishes through.
_stage_jobs_payload is the only stager, so it owns the "temp beside the
resolved store" rule instead of its caller wrapping the path. If the link
target's directory is not writable to us (file writable, dir not), fall
back to staging beside the link: saves keep working via atomic_replace's
copy fallback, exactly the pre-fix behaviour, instead of failing outright.
Move the symlink regression test next to the other save-path tests in
test_jobs_shrink_merge.py with module-level imports.
When jobs.json is a symlink into another directory (e.g. a different
filesystem), the temp file was created next to the symlink. The rename
onto the real target then failed with EXDEV and atomic_replace fell back
to copying in place, so a crash mid-copy left a torn, unparseable store.
Stage the temp beside os.path.realpath(jobs_file) so the publish stays a
same-directory atomic rename. atomic_replace, its EXDEV/EBUSY fallback
(still needed for single-file bind mounts) and the #60703 lock-timeout
degrade are unchanged.
Salvaged from #126237 (narrow hunk only).
The context_from reader in cron/scheduler_prompt.py validates archives
against _PROMPT_HEADING/_PROMPT_SEPARATOR/_RESPONSE_HEADING/
_RESPONSE_TERMINATOR, but the writer in cron/scheduler.py still spelled
the same bytes as literals. If one side changed, framed archives would
silently stop validating. The writer now composes the header and body
from those constants; output is byte-identical (checked by rendering
documents with the old and new writer and cmp-ing them).
Also collapse `not candidate or candidate.isspace()` to the equivalent
`not candidate.strip()`, and drop the two `assert final == answer`
lines in the extraction tests: they only checked that the stubbed agent
echoed its own mock response, not any cron behaviour.
The framed-archive change moved the caller's .strip() into
_archive_answer's framed/legacy branches only, so headingless
(script-mode / FAILED) archives reached _prepend_context_block with their
padding, putting stray blank lines inside the fence and spending clip
budget on whitespace. Return archive.strip() in that branch; the
script-mode test now asserts the exact trimmed fence.
Fold the review cleanups for the frame format so writer and reader
cannot drift: the stamp labels live once in scheduler_prompt.py (it
binds _sched only at import tail, so module-level patterns cannot be
built from scheduler attributes) and are imported by scheduler.py;
_PROMPT_FRAME_RE/_RESPONSE_FRAME_RE are compiled once; the +2/+1 magic
offsets are named separators; run_job normalizes newlines once via
_normalize_newlines and builds both stamps (header takes prompt_stamp
instead of a bool flag); the empty check uses isspace() instead of a
stripped copy; the docstring is rewrapped; both writer tests share a
_run_stub_job helper.
The salvaged stack added three tests for response framing. Keep the
writer->reader round trip with nested frames (red on main: rpartition
injects only the embedded subsection) and the truncated-outer-frame
case. Drop the test and the in-test block that exercise "framed archive
without a prompt length": that shape was only ever produced by the PR's
own intermediate commits, never by a writer on main, and the follow-up
removes the reader branch for it.
Preserve the optional-value disambiguation from #129282 while removing its flag-count cutoff. Make token and whitespace runs possessive, and retain standalone dash values. Bound regression probes in subprocesses so a GIL-holding regex cannot wedge the test runner.
Co-authored-by: Yuan Li <dskwelmcy@163.com>
Extend Finn763's dead-owner recovery with owner-fenced failure delivery based on PR #120391 and retain stderr until the parent consumes an escaping worker traceback. Preserve unknown side-effect state and do not retry automatically. Cover both shared entry paths, an exception through the payload lifecycle, abrupt death, and concurrent recovery.
Co-authored-by: Denis Dresvyanskiy <dresvyanskiydenis@gmail.com>
A restart-safe cron worker that dies AFTER adopting its attempt was reported as
"Scheduler restarted after this execution's owner exited" and treated as success.
recover_interrupted_executions() only knows that an owner is gone, so that is all it
can write -- but the waiter holding the worker's Popen knows the owner was that
worker and knows its exit status. Worse, the sweep leaves the row terminal, so the
waiter returned True, run_one_job short-circuited, and mark_job_run never ran: the
job's fire claim outlived its lost execution and refused the next manual fire with
"Job is already being fired" for the whole 300s lease, with an empty last_error.
- executions.terminalize_dead_owner(): terminalize ONE attempt with an observed
cause. Refuses a live owner, this process, and a handoff inside the adoption
grace, so it can never steal a running worker's attempt; returns False so the
caller keeps the generic sweep.
- The waiter records that cause and RAISES, routing the existing post-handoff
bookkeeping: mark_job_run records the run and retires the claim, no dispatch
incident and no false crash notice. Also folds the worker's captured stderr into
the error, which the pre-ack path already did (#112729) and this path dropped.
(cherry picked from commit d3f3571f8c6f4d2f188e90151d8f8393f049a573)
The dashboard's manual Run now binds the sibling profile via
web_server_cron._cron_store_scope, not the ticker's _profile_cron_scope,
so the marker-based routed_profile_fire() stayed False there: the routed
profile's .env was reloaded into the shared os.environ and the managed
external-worker handoff serialized multiplex_active=False.
routed_profile_fire() now compares the fire's active HERMES_HOME with
get_routing_process_hermes_home(), so every entry point (ticker,
dashboard Run now, CLI, gateway forward) and the external-worker handoff
share one answer. Regression test drives the real dashboard fire path
A -> B -> A with no_agent jobs.
Answers the P1 review on #111187: build_profile_secret_scope() held only
<profile>/.env plus that profile's external-source snapshot, never the
administrator-managed .env. The launch process applies that file LAST with
override (_apply_managed_env), so a managed key beats the user's own value in
os.environ. Under multiplex semantics get_secret() stops falling back to
os.environ on a scope miss, so inside a routed cron fire (and equally inside a
real multiplex gateway turn, which builds its scope through the same function
via gateway/run.py::_load_profile_secret_scope) a managed-only credential
resolved as absent and a managed-vs-user collision resolved to the USER value:
reversed precedence.
Fix at the source: build_profile_secret_scope() overlays load_managed_env()
last, after the profile .env and external sources, skipping process-global
names exactly as it does for the other two layers. Every multiplex-authoritative
scope (gateway turn, routed desktop fire, external worker env build) is built
here, so managed authority is composed once instead of restored per consumer.
No generic ambient-env fallback is reintroduced: only the managed file's own
keys enter the scope, and only with the managed file's values.
Regression (parametrized, two invariants): inside a routed fire a managed-only
key resolves through get_secret(); a managed-vs-user collision yields the
managed value.
(cherry picked from commit cedf4a3d78)
(cherry picked from commit 4a750b4022ff321863d6746ec39ec5ca67885c16)
The earlier trim dropped the only test of the `routed_profile_fire() and
not is_multiplex_active()` branch in the external-worker handoff: the
process flag is OFF (a desktop tick is not a multiplexer) yet a fire routed
to a sibling profile must serialize `multiplex_active=True` and hand the
worker an env without the launch profile's residue, and the context must
not outlive the handoff span. Without it that branch could regress to the
pre-fix behaviour unnoticed.
(cherry picked from commit 5850a50a81)
(cherry picked from commit 440bee6ccf9d48eecf91769787f6daa3fd3ea00c)
The PR shipped nineteen tests across six files, most of them variations of
one boundary. Keep the three that pin distinct behaviour:
- a routed desktop-ticker fire runs under multiplex semantics for exactly
its scope: a scope miss returns None instead of the launch credential and
the parent os.environ is byte-identical afterwards;
- a routed no_agent child never sees a launch-only name, whether the launch
.env defined it or a launch external source supplied it (applied or lost
to a pre-existing process value), while its own values come through;
- administrator-managed keys keep policy precedence over the routed
profile's own value.
Everything else was either a positive control of the same seam, a
set-membership check on a module-level constant, or a re-statement through
a different entry point.
(cherry picked from commit 042b9830b4)
(cherry picked from commit 0b82557fbdbfa6968b2b795e4e63d484ffb582b1)
Review findings on f5f88d5058. Three are defects the previous round introduced.
Managed keys were stripped as launch residue. Recording every dotenv load as
residue swept in the administrator-managed `.env`, which `_apply_managed_env`
applies LAST with override precisely so it beats the user's own `.env`. A
routed child then lost `ORG_POLICY_FLAG=managed-value` to the routed user's
`user-value`. Managed keys are now recorded separately, never enter the
residue set, and are re-applied over the routed scope in both child builders
(`scheduler_script`, the restart-safe handoff) so the child sees the same
precedence the launch process does. `kanban_db_dispatch` and
`scheduler_delivery` strip without any overlay, so for them the exclusion
alone is the guarantee; the test pins the case that exercises it — the same
key defined in both the user and the managed file.
Private hydration did not record supplied names. `_hydrate_profile_secret_sources`
now feeds `provenance` plus `skipped_existing` into the same ownership set the
process-global path uses; the provenance label map stays applied-only.
Removal cleanup cleared its marker before the fallible work. A raising
reload left the removed plugin's credential active with no retry, because the
next no-source discovery saw the flag already false. The marker is cleared
only after reset, reload and installed-scope refresh succeed.
Routed fire not multiplexed at the handoff. `run_one_job` enables the
context in `_install_fire_secret_scope`, which runs AFTER
`_launch_external_cron_worker`, so a routed desktop fire on the managed path
serialized `multiplex_active=False` and built the worker env with launch
residue and no scrub. The handoff now treats `routed_profile_fire()` as
multiplexed for exactly its own span; the worker re-establishes the state from
the payload as before.
Each fix was checked by reverting it and confirming its regression fails,
including the overlay half and the exclusion half of the managed fix
separately.
(cherry picked from commit 329cbd8963d68c45b425e95a5b11ade59f513960)
(cherry picked from commit 62b4488cb5)
(cherry picked from commit d3319102a7e4dcce35ec2676fe22aed41613e893)
Review findings on d8c467f223, each reproduced through its production path.
Stale launch key. `strip_launch_profile_env` built its residue set from a
re-parse of the launch `.env`. A key removed or renamed in that file after
boot is still in `os.environ` with the old value (dotenv never unsets), and
the current file no longer names it, so it survived into the routed child.
`_load_dotenv_with_fallback` — the one chokepoint every dotenv load goes
through — now records the KEY names it put into the process env, additive for
the process lifetime (`launch_dotenv_keys()`), and the strip unions that record
with the current file.
Source name that lost to the process env. `_apply_external_secret_sources`
snapshots every name a source SUPPLIED (`provenance` + `skipped_existing`),
but `secret_source_names()` only exposed `_SECRET_SOURCES`, which is
provenance metadata and names applied values alone. A launch-profile source
that supplied `CUSTOM_VAULT_SECRET` while the process already had it was
therefore invisible to the scrub, and a routed child with an empty scope got
the launch value. Supplied names are tracked separately
(`_SOURCE_SUPPLIED_NAMES`) so the provenance labels stay honest, and
`secret_source_names()` returns the union.
Last plugin source removed. `_refresh_secret_sources_after_discovery`
returned before the cache reset and the installed-scope refresh whenever no
plugin source was enabled — and `discover_and_load(force=True)` unloads the
old registration first, so removing the final plugin source hit exactly that
return with the removed plugin's names still in the per-home snapshot and the
current scope. The manager now remembers that a discovery re-applied plugin
sources and, on the next discovery that finds none, reconciles once. A home
that never had a plugin source is still a no-op (pinned by the existing tests).
Regressions: the stale-key lifecycle and the skipped-existing case through
`_run_job_script` against a real routed child, and the removal case through
the manager. Each checked by reverting its fix and confirming the test fails.
(cherry picked from commit d464f5f6126a394cfb47937f683d3a5e2f141840)
(cherry picked from commit 9d7de6c140)
(cherry picked from commit b0beb731b18edd884e36a30ab91c91687bc7a965)
The source-name strip added alongside the external-source fix ran
unconditionally. Outside multiplexing there is no other profile to leak from --
os.environ IS this profile's environment -- so popping those names relied on the
routed scope overlay putting each one back, which in turn relies on the
per-home snapshot recorded at boot. Correct today, but it made a single-profile
child's credentials depend on bookkeeping that has nothing to do with isolation.
Guard it the way strip_launch_profile_env guards itself: no multiplexing, no
strip. A single-profile no_agent child now keeps a byte-identical env even if a
source's snapshot were ever missing. Pinned by a regression that runs a real
child with a source-owned name in os.environ and no multiplex context; making
the strip unconditional fails it.
(cherry picked from commit afa429b30a1c9c9b4c011f7d2426a9f099911596)
(cherry picked from commit 3dedff6a12)
(cherry picked from commit 1cb2a8e6eb9a1848fa758fcaef7dce531532258c)
Two credential-isolation gaps found in review of the previous head.
1. strip_launch_profile_env() only knows dotenv- and terminal-config-owned names,
but external secret sources (vault, 1Password, ...) also write their names into
the shared os.environ and are tracked in secret_source_names(). A name the LAUNCH
profile's source supplied therefore still reached a routed no_agent child. Drop
every non-global source-owned name from the base; the routed scope overlay that
follows puts back exactly the ones that profile's OWN sources supply, since
build_profile_secret_scope folds get_secret_source_values(home) in.
2. refresh_installed_secret_scope() merged the rebuild with dict.update(), so a
name a source had stopped supplying -- rotated, revoked, source removed -- kept
its old value for the rest of the fire. Replace the mapping contents instead: the
rebuild is the profile's current truth.
Regressions: a routed child sees <unset> for a launch-source name while its own
source value comes through, and a refresh whose rebuild omits a name drops it.
Both fail if the corresponding change is reverted.
(cherry picked from commit ecd51517c4828a75acb5f458ed99aea0bc3e5e9f)
(cherry picked from commit 0943e77136)
(cherry picked from commit 68ab0384e17e06e8eca684c4d8eac15543dc3943)
The routed no_agent child env started from all of os.environ and only overwrote the
names present in the installed scope. A name defined only by the LAUNCH profile's .env
and absent from the routed profile therefore reached the routed child with the launch
value instead of unset -- the secret scrub only knows classified names, so a custom or
unclassified secret crossed the profile boundary (review finding on the first head).
strip_launch_profile_env (main, 284d220ba4) is the primitive the external-worker path
already uses for exactly this: it drops the launch profile's dotenv-owned keys and the
bridged TERMINAL_* settings, and is a no-op outside multiplex or when the target IS the
launch profile. Apply it to the base BEFORE the scope overlay (so a shared name keeps its
routed value) and BEFORE the sanitizer (so routed values still pass the scrub and
passthrough rules). Pinned by a child-process negative control: the launch-only name
arrives <unset>, the shared name arrives routed, and the parent process is unchanged.
(cherry picked from commit 69349b527b5144c1f1540a4c10035d5ec798c1db)
(cherry picked from commit 023e4f997f)
(cherry picked from commit cace4f3f2013e9dc60356add245b2bad4b56ab08)
The desktop backend ticks EVERY local profile's cron store from one process — its own docstring
says "like a multiplex gateway" (hermes_cli/web_server.py) — but never sets the process-global
multiplex flag, and cannot: its own chat turns are unscoped and would fail closed. Every
isolation in the tree keys on that flag — the guard that keeps a routed `.env` out of the shared
`os.environ`, `get_secret`'s fail-closed miss, passthrough resolution, the MCP and kanban
subprocess scrubs — so all of it was inert for a sibling profile's fire. Verified: a secondary
profile's API keys replaced the launch profile's in `os.environ` with `override=True` and stayed
there after the tick, and a scope miss read the launch profile's tokens (#107692).
Give multiplex mode a context-local counterpart. `set_multiplex_context` (agent/secret_scope.py)
is OR'd into `is_multiplex_active()`. `_profile_cron_scope` only MARKS a fire whose home is not
the process's own (`routed_profile_fire`, decided against `get_process_hermes_home()`, the
override-immune resolver); `_install_fire_secret_scope` in cron/scheduler.py installs the
profile's hydrated secret scope and, for a marked fire, the multiplex context — for exactly that
span, dropped again before the scope by `_reset_fire_secret_scope`. Multiplex semantics are
therefore never active in cron without a scope to read: `run_one_job`'s restart-safe handoff runs
before the body's scope and keeps today's semantics (its own scope is #107413 / #106050's seam,
left untouched so this composes with whichever lands). Every existing multiplex-keyed isolation
applies inside the routed fire with no per-site patching; the launch profile's own fires and the
backend's turns keep single-profile semantics; marker and override both reach the pool worker via
`copy_context()`. `get_secret` read the raw global in its miss branch; it now goes through
`is_multiplex_active()`. The dotenv guard keeps its pinned flag-only form (#77970).
Two consequences of suppressing the write are handled rather than left as regressions:
- a `no_agent` script's env is `os.environ.copy()`, which no longer carries the routed `.env`;
the runner overlays the installed scope onto the base BEFORE sanitizing, so the same scrub /
passthrough rules apply to those values and the parent process is never mutated;
- plugin secret sources are discovered on the fire's first agent build, after the scope froze,
and the post-discovery reload is hydrate-only under multiplex semantics; the refresh now folds
the values into the installed scope in place (`refresh_installed_secret_scope`, the pattern
`_publish_env_value` already uses for `.env` writes under multiplex).
And the profile's external secret sources are hydrated before the scope is frozen, the order
gateway/run.py and the external cron worker already use.
Tests pin each direction: the marker without the semantics before the scope, the semantics on and
off exactly with it, the marker reaching a copy_context worker; the process's own profile staying
single-profile; the restart-safe handoff's child env building without raising under a routed tick
with a passthrough key registered; a real child process receiving the routed values while
`os.environ` keeps the launch value; a source registered after the freeze reaching the fire
through the real PluginManager refresh. Reverting any one direction fails a distinct test.
(cherry picked from commit 2f87677425d2cca19286ac83bc45cab23e546669)
(cherry picked from commit dbede34f6e)
(cherry picked from commit a7ac197ff12710e045e3f75d960b20be952960f3)
Each removed test either asserts nothing reachable or repeats another
test's exact body:
- auxiliary_client empty-base_url: body only writes config.yaml; a merge
(890bbbda1f) restored the header of a test b3d4f67b20 had purged. The
non-Anthropic-host test beside it drives the same default fallback.
- background_review no-session-db: _record_review_usage_to_parent swallows
every exception and there is no DB to observe, so removing the guard
still passes; the path has no observable effect to pin.
- code_execution Windows "would have failed" pair: every branch returns,
skips or passes; they exercise CPython defaults, not Hermes code.
- cron test_should_use_direct_api_call_*: byte-identical copy of the test in
tests/agent/test_cron_inline_api_call.py (0213b8b77b had to edit both).
- cron liveness no-multiplexer: identical body to test_no_lock_no_pids_is_false;
its multiplexer patch is never reached.
- browser console/get_images "private URLs allowed": identical to the
local-backend test in each file; both mock the predicate the name promises.
- hermes_state TestTitleUniqueness: the duplicate-title case is a strict
subset of TestSessionTitleLineage::test_unrelated_session_still_conflicts;
the NULL-title case never writes a title.
worker_bootstrap() logged a failed activate_dependencies() and let the worker
carry on through the inherited PYTHONPATH pin. That path is not leased: once
the gateway releases its generation the collector deletes it under the live
worker and the next dependency import dies mid-job (#122936 review, reproduced
with a committed generation that has no site-packages).
The failure now propagates, so the worker exits before its ownership ack and
_launch_external_cron_worker reports the dispatch failure with the worker's
stderr. PM's own no-op cases (venv, wheel, Nix, nothing committed under a venv)
still return normally.
Tests drive the real PM boot in a marked child: it imports a sentinel that
exists only in the committed generation, and when an update commits a newer
one mid-job the real collector keeps the worker's generation until the worker
exits. A damaged selection exits before cron.jobs is imported.
123456hyu made the same prelude move and fail-closed change on #122290
(dff16e12); this carries them without the argv gate and source-order test.
Co-authored-by: 123456hyu <123456hyu@users.noreply.github.com>
