Plugins can now call ctx.register_automation_blueprint(key, *, title,
description, schedule_template, prompt_template, category, slots,
deliver_default, skills, tags). The fields mirror the built-in
AutomationBlueprint, and slots are BlueprintSlot field dicts, so the call
also crosses the plugin-host boundary.
- One lookup, cron.blueprint_catalog.list_blueprints() / get_blueprint(),
merges the built-ins with the active profile's plugin blueprints at call
time. /blueprint (CLI/TUI/messengers), /api/cron/blueprints and the
instantiate route all use it. Each dashboard route resolves inside the
?profile= scope, off the event loop.
- Keys are namespaced <plugin>:<key>, so a plugin cannot shadow a built-in.
A ':' in the key, a duplicate key, an unknown placeholder, a bad slot type
or a schedule that does not parse is warned and ignored at registration.
- Catalog entries carry source/plugin. Desktop fetches the catalog for the
profile it creates jobs in, keyed per profile, and shows the plugin name
on the blueprint row and in the "Start from" select. The dashboard shows
a plugin badge.
- A created job is an ordinary cron job with the prompt and schedule already
filled in, so it keeps running if the plugin is later disabled.
- The ctx content registrars (register_skill, now joined by
register_automation_blueprint) move to hermes_cli/plugins_content.py.
The dashboard's blueprint types move to web/src/lib/automation-blueprints.ts.
Closes#132373. Requested by @enckequity.
Plugins could only see a cron run as the task_id string `cron:<job>:<execution>`,
with no way to tell a scheduled fire from a manual `hermes cron run`. A
security plugin that may only act inside a real scheduled occurrence had
nothing to check.
The scheduler now binds a frozen CronExecution (job id/name, execution id,
source, scheduled_instant, started_at, profile) in a ContextVar once the run
wins its claimed->running CAS (or a restart-safe worker adopted the row), and
resets it when the fire ends. ContextVar scoping keeps concurrent jobs and
profiles apart and follows the agent onto the threads the scheduler already
copies context into. Delegated children read None.
Also trims duplicated comments in _run_one_job_body to keep the facade within
its code-health line cap.
Closes#130722
The Home Assistant gateway platform (plugins/platforms/homeassistant) and the
homeassistant toolset (tools/homeassistant_tool.py) now ship as the official
catalog plugin `homeassistant` (NousResearch/hermes-homeassistant), with the
same platform name, toolset/tool names, config keys and env vars.
Migration (hermes_cli/left_core_migration.py, a generic "left core" table
reusing the memory-provider migration's helpers): every profile home that
used Home Assistant (HASS_TOKEN in its .env, platforms.homeassistant enabled
or holding a token, or the homeassistant toolset selected) gets the plugin
installed from the catalog at its reviewed pin by `hermes update`, and lazily
once per process at agent start / gateway start (honours
security.allow_lazy_installs). Every outcome reaches terminal, Desktop and
chat; a gateway-start outcome is handed to that home's first agent.
Core HA special cases become generic PlatformEntry seams the plugin uses:
trusted_inbound (allowlist bypass for event-bus platforms), display_tier
(built-in display defaults) and shared_env_prefixes (profile-clone ownership
rule for env shared with the plugin's tools). Cron delivery uses the existing
cron_deliver_env_var seam; HASS_TOKEN stays stripped from subprocesses via the
installed plugin's manifest secrets.
`hermes -s dup-demo` (CLI, oneshot, TUI/kanban worker) died with "Unknown
skill(s): dup-demo" and cron logged "skill not found, skipping" for a name that
exists twice. skill_view's refusal now carries load_names; preload labels the
failed identifier "Ambiguous skill name X: use one of <paths>" (the label is
computed only on the failure path) and format_missing_skills() keeps that
wording in the raised error; cron skips with the same message and lists it in
the job's skipped-skills notice.
CRONSCOPE r2 quality Lows: the dispatch-failure notice in run_one_job and
the env build in _launch_external_cron_worker each hand-rolled the same
install / try / finally / reset around _install_fire_secret_scope. Add
_fire_secret_scope() next to the helper pair and use it at both sites;
reset order is unchanged (it still delegates to _reset_fire_secret_scope).
Also collapse the five-line comment at the dispatch-failure site to one.
_run_one_job_body keeps its tokens (the reset sits in a larger finally) and
scheduler_worker_failure.py is untouched (mark_job_run shares its finally).
_run_one_job_body's except branch sends _deliver_crash_failure when anything in
the body raises, but the firing profile's secret scope was only installed after
claim_dispatch and mark_execution_running. A raise from either sent the crash
notice with no scope: under multiplex get_secret fails closed
(UnscopedSecretError, "Hermes could not read this profile's
TELEGRAM_HOME_CHANNEL") and the notice never left. Same class as the
dispatch-failure branch fixed for #131585 (and called out in #130431's body).
Install the scope first thing in the try so every exit through the except is
scoped; the outer finally already resets it. The handoff path stays outside it
(#107692), and the external-worker payload path and the post-handoff notice
already run scoped.
Extends the #130431 test with a claim_dispatch-raising case.
Refs #131585
Co-authored-by: skyeyesec333 <skyeyesec333@users.noreply.github.com>
run_one_job's pre-handoff dispatch-failure branch (#124013) calls
_deliver_crash_failure before _run_one_job_body, which is where the
firing profile's secret scope is installed for delivery. Under a
multiplexed gateway the notice then resolves the home channel through
get_secret with no scope, raises UnscopedSecretError, and never leaves.
Wrap the notice in _install_fire_secret_scope/_reset_fire_secret_scope,
the pair _run_one_job_body and record_unknown_worker_outcome already use.
(cherry picked from commit 5e0735b17267a636337818505bae485d7e208837)
The external worker now calls GatewayStartupMixin._register_config_hooks (the
helper gateway startup and each multiplexed profile already use) instead of a
copied load_config/register_from_config sequence, and calls it after
set_secret_scope rather than before. Registered outside the scope, a
hooks.outbound target with secret_env hits get_secret with no scope while
multiplexing, which raises UnscopedSecretError; the never-raise guard swallowed
it, so that profile's outbound webhooks were silently dropped again.
Both registries are idempotent per (home, event, ...) so a worker running
several payloads registers each hook once. The kept test gains a secret_env
target on a multiplexed payload to pin the ordering; the broken-config test is
dropped (register_from_config never raises on a malformed block, and the
helper's guard is already covered by the gateway path).
Refs #131764
The restart-safe external worker (python -m cron.scheduler
--external-worker-file) called discover_plugins() under the payload's
home override but never agent.shell_hooks.register_from_config() nor
agent.outbound_webhooks.register_from_config(), so hooks: blocks in the
owning profile's config.yaml silently stopped firing for cron sessions
once gateway fires were handed to the worker instead of running
in-process. Plugin hooks kept working, which made the gap invisible.
Register both right after plugin discovery + secret hydration, under the
same home override, with the gateway's never-raise shape (#102504):
consent resolves non-interactively from the owning profile's allowlist /
HERMES_ACCEPT_HOOKS / hooks_auto_accept, and a broken hooks block logs a
warning instead of taking the cron execution down.
Fixes#131764
(cherry picked from commit d1971c698a4429868e9d43d058032930ec62d7b9)
Register direct workers with a live future and fence dispatcher release and future attachment with per-registration identity tokens. Preserve profile-scoped release keys.
Port the in-memory fix from 251b99332550be4dc5a7ba265109e367842e9081, without timestamp-based durable-state guards.
Fixes#132223. Refs #88688.
(cherry picked from commit dd834c874fef04a7d56658e0d05baa3bee5decba)
All three ownership-lost call sites in _classify_delivery_outcome computed
ladder_rung=is_retry_run(job) themselves and passed job["id"]; the helper is
the only place that needs the flag, so pass the job and derive it inside.
That removes the duplicated expression (and a mis-indented continuation
line at the third call site) and drops the now-unused function-level
is_retry_run import in _classify_delivery_outcome. The conditional
ladder_rung splat is kept so mark_job_run fakes without the kwarg still
see the same call shape as before. No behaviour change.
_record_fire_ownership_lost is the last terminal write on the claimed
snapshot that still counted a ladder re-run toward repeat. When a re-run of
an unreachable-model occurrence is cancelled at the transport (dashboard
drain) while we still own the claim, it records the interruption through
mark_job_run, which bumped repeat.completed a second time for an occurrence
that had already counted -- the same bug the previous commits fixed for the
finish, crash, dispatch-failure and dead-worker tails.
The helper now takes ladder_rung and forwards it (conditional kwarg, same
idiom as the sibling paths), and its three callers in _run_one_job_body pass
is_retry_run(job). The shutdown/mark_interrupted path carries no snapshot
and is left as is.
The flag was count_run=True, and mark_job_run passed
count_run=not ladder_rung, so the same idea had two names with opposite
polarity. Use ladder_rung on _advance_after_run too. The docstring now
says when the bump is skipped, and the inline comment lists both reasons a
run is not counted again (a ladder re-run and a pre-claimed finite
one-shot). No behaviour change.
The previous commit stopped a re-run from the unreachable-model ladder from
counting toward repeat, but only on the normal _finish_completed_run path.
The other places that record a failed fire on the same claimed snapshot
still counted it: the crash/interrupt tail of _run_one_job_body, the
pre-handoff dispatch failure in run_one_job, and the recovered unknown
worker outcome in scheduler_worker_failure. A re-run that crashed or failed
to dispatch therefore still bumped repeat.completed a second time for an
occurrence that had already counted.
All three now pass ladder_rung=True when is_retry_run(job). The
worker-failure job is the same run_one_job snapshot, which carries
_scheduled_instant (stamped by the due scan and copied by
_process_due_job), so is_retry_run applies there too. Manual runs carry no
_scheduled_instant and still count.
The kept test gains a "crash" row: a re-run whose run_job raises must leave
repeat.completed at 1.
A recurring job with a finite repeat count (for example repeat=3, every
24h) lost one repeat slot for each automatic re-run after a fire that
could not reach the model. The failed occurrence counted, and then each
5/15/30-minute re-run of that occurrence counted again, so a half-hour
outage finished a repeat=3 job with zero model calls. With
cron.retry_unreachable off, the same outage costs only the one failed
occurrence.
Cause: cron/jobs.py:_advance_after_run bumps repeat.completed on every
mark_job_run, and the re-runs parked by
cron/unreachable_retry.py:plan_retry go through it like scheduled
occurrences. cron/unreachable_retry.py:will_retry also treated a re-run
on the last slot as the job's final run.
Fix: cron/scheduler.py:_finish_completed_run passes ladder_rung=True
when the dispatched instant is the parked re-run instant (new
is_retry_run, which compares the dispatch snapshot's _scheduled_instant
with the ladder state). mark_job_run then calls _advance_after_run with
count_run=False, and will_retry leaves that run out of its final-run
check the same way. The first failed occurrence still counts, and manual
runs still count.
(cherry picked from commit 27038f7064bb611811efee8c527dddecd4fcb57b)
When two cron scheduler instances run against the same jobs.json (e.g.,
a desktop Hermes InProcessCronScheduler and a NSSM-managed
HermesGateway), the cross-process file lock correctly prevents the
loser from firing jobs. However, the "Tick skipped" message was at
DEBUG level, making dual-scheduler suppression invisible in production
logs at the default INFO level.
Change to INFO so operators can see when a scheduler instance is being
suppressed by the lock — this is an expected control-flow outcome, not
diagnostic noise.
Closes#53720
A failed cron run records last_status=error and bumps failure_streak
correctly, but the next successful run resets both in the same pass and
nothing durable records that a failure happened — a silent watchdog-style
script that fails every Nth tick collapses to a single tick of visible
failure, and monitors sampling jobs.json report green forever (#118354).
Stamp job[last_failure] = {at, detail} on every failed run, latest
failure wins, and do NOT clear it on success: last_status keeps
describing current health while last_failure preserves the recency
evidence for dashboards and cron list. cron list now prints a warning
line for the stamp (marked recovered when last_status is healthy).
Fixes#118354🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
Add AwakeIdleMeter, which reads the wall clock and time.monotonic() on
each watchdog check. monotonic pauses during sleep on macOS
(mach_absolute_time) and Linux (CLOCK_MONOTONIC), so the gap between the
two clocks is time spent asleep, and that sleep is not charged as idle
time until the agent records new activity. The cron inactivity watchdog
and the turn liveness watchdog both use it, because on main both guard a
cron run with a 600 s wall-clock bound. A job that is really idle for
600 s awake is still killed.
Fixes#131523
(cherry picked from commit 07d9ddc24736e243d72327ff90a24f95e86119d8)
A scoped or wedged daily-brief in profile A must not hide a degraded or
live daily-brief in profile B. Split the scheduler's in-flight keys before
counting and add awaitable cron runs independently of non-cron work.
Carry the per-run restart_safe flag to the observer and preserve the
existing bare-ID accessors for unrelated callers.
Thanks to @ehz0ah for diagnosing the multiplexed collision and proposing
the profile-keyed accounting and mixed scoped/degraded regression case.
Preserve @Superfly73's source commit separately; map their verified bare
noreply email for the contributor CI gate.
Refs #130987
Co-authored-by: z1gon <130889443+ehz0ah@users.noreply.github.com>
A cron run handed to an external worker in its own transient systemd scope
outlives the gateway: the shutdown drain deliberately leaves it unmarked
(``mark_running_jobs_interrupted``) and its final send rides the durable
delivery queue, drained by whichever gateway is live next. The after-turn
restart wait still counted it as active work, so ``/restart``, SIGUSR1 and
``hermes update`` held the gateway in ``draining`` — refusing every new turn —
for up to ``agent.restart_after_turn_timeout`` (30 min by default) while the
worker kept running regardless. Observed in the field: a 100-minute job turned
a restart into a ~30-minute lockout.
- ``cron.scheduler`` records whether an accepted handoff got its own scope
(``dispatch.mode == "scoped"``) and exposes
``get_restart_safe_external_job_ids()``; runs already reported as wedged are
excluded there too, because the gateway subtracts both counts and a run in
both sets would be subtracted twice.
- ``_awaitable_work_count()`` subtracts that count as a second, disjoint
exclusion; ``_describe_active_work()`` marks the unit (``restart_safe``).
- A degraded worker (external, but no reachable user bus, so it shares the
gateway cgroup) is NOT listed and is still awaited — the restart would kill
it mid-run.
- The wait and observation log lines now name both exclusion classes instead of
claiming everything still listed will be interrupted.
Tests: tests/gateway/test_restart_drain.py (skip + control),
tests/cron/test_restart_safe_worker.py (registry contract).
Docs: updating.md restart section, cli-config.yaml.example.
(cherry picked from commit 6a2f0b77cf07a60510430fc4ef8b75539ee422fb)
The router turned every failed SendResult into a bare RuntimeError, which
dropped raw_response, and the cron live lane falls back to the standalone
sender on any error. A split send that failed after delivering its head
(Telegram, Discord, and now SMS/WeCom) was therefore delivered again from
the start by the standalone lane.
The router raises PartialDeliveryError for a partial_overflow result; the
live lane records "delivered N of M chunks, then failed" on the run and
does not fall back.
_live_send_text used future.cancel() to tell a send that never started
(wedged loop: fall back to standalone) from one in flight (assume delivered:
a resend would duplicate). A run_coroutine_threadsafe future stays PENDING
until the coroutine finishes, so cancel() returned True mid-send: the
in-flight send was cancelled part-way and the standalone lane then sent the
whole payload again. Paced multi-chunk sends (WeCom's 30 msgs/min bucket)
outlast 60s routinely; a 55-chunk cron payload arrived as 25 chunks and was
reported failed.
The send now records its own start under a lock; on timeout it is
abandoned (and the standalone fallback runs) only when it never began.
The existing in-flight test faked the state with a mocked cancel()
returning False, the exact assumption that does not hold; it is rewritten
against a real event loop, with the never-started (wedged loop) case it
never covered alongside.
The inactivity watchdog polls every 5s, so the same stall reports
"idle for 601s" on one run and "idle for 604s" on the next. The incident
signature hashed that raw number, so each run minted a new incident: an
acked timeout pinged again and the repeat-alert cooldown never applied.
Mask duration tokens before hashing; status codes and other numbers still
split incidents.
Fixes#129587
e33181e883 stopped caching every None from _cron_interval_minutes, so a
malformed cron expression (croniter loaded, expression invalid) was
re-parsed on every in-flight-guard check. Track whether croniter actually
loaded and cache unless the None came from a failed import: a bad-expr
None is stable, an ImportError None is transient. The kept #127182
regression test now asserts the import-failure None stays uncached.
Also drop the redundant recovery_kind operand in
_recover_missing_next_run (_is_recoverable_error_job already requires a
cron/interval schedule) and tighten two comments.
_cron_interval_minutes cached None when croniter failed to import, the
same latch class as #127182: after a transient ImportError that
expression kept the floor in-flight allowance for the process lifetime
even once croniter imported again. Cache only a real cadence (it never
changes); None is recomputed on the next call.
Not latching a failed croniter import fixed#127182, but with nothing
cached every _ensure_croniter() call re-ran the full finder walk when the
package is genuinely absent - several times per cron job on every
due-scan tick, under _jobs_lock. Remember a monotonic retry-at 60s out
instead: a transient failure still recovers within one tick, explicit
True/False overrides still win, and a missing package costs one probe a
minute.
Also tidy the error->scheduled re-arm: reuse _is_recoverable_error_job()
for the condition and drop the redundant next_run_at assignment that
job.update(fields) already performs; fix the test comment (the patched
import raises for as long as it is patched in, not once).
Once croniter imports again, due-scan recovery re-arms a recurring job's
missing next_run_at, but the job kept state=error until it next fired, so
`hermes cron list` kept reporting a healthy, due job as broken. Flip it
back to scheduled in the same persist, and correct the docstring: recurring
jobs now also land here after a transient croniter ImportError (#127182),
not only after a hand-edited jobs.json.
Co-authored-by: itsflownium <157689911+itsflownium@users.noreply.github.com>
A single transient croniter ImportError (wrong interpreter, shadowed path)
was cached as HAS_CRONITER=False for the whole process lifetime. While
latched, compute_next_run() returns None for every recurring job, so jobs
stay enabled with next_run_at=None and state=error — never rescheduled
until a gateway restart, despite the due-scan recovery path existing
exactly for that case (#127182).
Keep HAS_CRONITER None on failure so the next call re-probes; an explicit
monkeypatched value still wins. Also log sys.executable with the warning so
a scheduler running under the wrong interpreter is diagnosable from
errors.log.
Fixes#127182
(cherry picked from commit bb48a7b48fb28d45d63c47b59a1cf0c5049efb87)
Five writers (cron jobs store, utils._atomic_write, config._write_env_lines,
the env_loader sanitize rewrite, backup _extract_member_atomically) each
hand-rolled "stage beside the resolved symlink target", and only cron kept the
PermissionError fallback. A symlink into another filesystem whose real
directory is read-only to us therefore made utils/config/backup raise and the
.env sanitize silently skip (NUL bytes left on disk), where base saved by copy.
mkstemp_beside picks the same directory atomic_replace renames into (resolved
only when the target is a link) and, on PermissionError, retries beside the
link so the base EXDEV -> copy fallback still saves. One rule, one docstring,
one place to test.
_stage_jobs_payload is the only stager, so it owns the "temp beside the
resolved store" rule instead of its caller wrapping the path. If the link
target's directory is not writable to us (file writable, dir not), fall
back to staging beside the link: saves keep working via atomic_replace's
copy fallback, exactly the pre-fix behaviour, instead of failing outright.
Move the symlink regression test next to the other save-path tests in
test_jobs_shrink_merge.py with module-level imports.
When jobs.json is a symlink into another directory (e.g. a different
filesystem), the temp file was created next to the symlink. The rename
onto the real target then failed with EXDEV and atomic_replace fell back
to copying in place, so a crash mid-copy left a torn, unparseable store.
Stage the temp beside os.path.realpath(jobs_file) so the publish stays a
same-directory atomic rename. atomic_replace, its EXDEV/EBUSY fallback
(still needed for single-file bind mounts) and the #60703 lock-timeout
degrade are unchanged.
Salvaged from #126237 (narrow hunk only).
The context_from reader in cron/scheduler_prompt.py validates archives
against _PROMPT_HEADING/_PROMPT_SEPARATOR/_RESPONSE_HEADING/
_RESPONSE_TERMINATOR, but the writer in cron/scheduler.py still spelled
the same bytes as literals. If one side changed, framed archives would
silently stop validating. The writer now composes the header and body
from those constants; output is byte-identical (checked by rendering
documents with the old and new writer and cmp-ing them).
Also collapse `not candidate or candidate.isspace()` to the equivalent
`not candidate.strip()`, and drop the two `assert final == answer`
lines in the extraction tests: they only checked that the stubbed agent
echoed its own mock response, not any cron behaviour.
The framed-archive change moved the caller's .strip() into
_archive_answer's framed/legacy branches only, so headingless
(script-mode / FAILED) archives reached _prepend_context_block with their
padding, putting stray blank lines inside the fence and spending clip
budget on whitespace. Return archive.strip() in that branch; the
script-mode test now asserts the exact trimmed fence.
Fold the review cleanups for the frame format so writer and reader
cannot drift: the stamp labels live once in scheduler_prompt.py (it
binds _sched only at import tail, so module-level patterns cannot be
built from scheduler attributes) and are imported by scheduler.py;
_PROMPT_FRAME_RE/_RESPONSE_FRAME_RE are compiled once; the +2/+1 magic
offsets are named separators; run_job normalizes newlines once via
_normalize_newlines and builds both stamps (header takes prompt_stamp
instead of a bool flag); the empty check uses isspace() instead of a
stripped copy; the docstring is rewrapped; both writer tests share a
_run_stub_job helper.
_archive_answer searched archives without a "**Prompt Characters:**"
stamp for a "**Response Characters:**" frame. No writer on main ever
produced that shape (git log -S'Response Characters' on main is empty;
it only existed in the PR's intermediate commits), so the branch only
handled hypothetical archives - and on a real legacy archive whose
answer quotes a "**Response Characters:** N" line it returned None
instead of the legacy last-heading split.
Unframed archives now always take the legacy rpartition path; framed
archives keep exact-length validation, folded into the framed branch.
The view-only decision for a never-closed cron run keyed on `is_active`,
a 300s activity window. A live run inside a long tool call writes no
heartbeat, reads inactive exactly like a zombie, and was latched
read-only for the rest of the app session.
- Backend: run rows (and the session detail of a cron run) carry
`scheduler_owned` — the job's in-flight execution is held by a live
process (pid + start-time fingerprint) and the session started under
that claim. An older never-closed run of the same job stays a zombie.
- Desktop: the latch becomes a per-run verdict that is re-evaluated on
every Cron surface poll and, authoritatively, right before any send
(primary chat and tiles). A run that ticks, is owned, or closes becomes
writable again; one whose process dies becomes view-only.
- Restored route/tab after a restart: the pre-send refresh recognises a
cron run by its session id and gates it even though no surface ever
evaluated it; an unreadable unknown run fails closed.
Refs #88443
Follow-up to the in-process schedule tool:
- run-lifecycle guard: an unbound/malformed HERMES_KANBAN_RUN_ID now refuses
kanban_schedule, so a stale worker cannot park a card a successor owns
- goal_mode tasks are refused: `scheduled` ends the goal loop like `blocked`,
so parking would bypass the completion judge (#38696)
- stop nudge treats kanban_schedule as a handoff
- Codex app-server workers get it via EXPOSED_TOOLS
- cron/AGENTS.md lists it with the worker toolset
Refs #108554, #108566
Preserve the optional-value disambiguation from #129282 while removing its flag-count cutoff. Make token and whitespace runs possessive, and retain standalone dash values. Bound regression probes in subprocesses so a GIL-holding regex cannot wedge the test runner.
Co-authored-by: Yuan Li <dskwelmcy@163.com>
Extend Finn763's dead-owner recovery with owner-fenced failure delivery based on PR #120391 and retain stderr until the parent consumes an escaping worker traceback. Preserve unknown side-effect state and do not retry automatically. Cover both shared entry paths, an exception through the payload lifecycle, abrupt death, and concurrent recovery.
Co-authored-by: Denis Dresvyanskiy <dresvyanskiydenis@gmail.com>
A restart-safe cron worker that dies AFTER adopting its attempt was reported as
"Scheduler restarted after this execution's owner exited" and treated as success.
recover_interrupted_executions() only knows that an owner is gone, so that is all it
can write -- but the waiter holding the worker's Popen knows the owner was that
worker and knows its exit status. Worse, the sweep leaves the row terminal, so the
waiter returned True, run_one_job short-circuited, and mark_job_run never ran: the
job's fire claim outlived its lost execution and refused the next manual fire with
"Job is already being fired" for the whole 300s lease, with an empty last_error.
- executions.terminalize_dead_owner(): terminalize ONE attempt with an observed
cause. Refuses a live owner, this process, and a handoff inside the adoption
grace, so it can never steal a running worker's attempt; returns False so the
caller keeps the generic sweep.
- The waiter records that cause and RAISES, routing the existing post-handoff
bookkeeping: mark_job_run records the run and retires the claim, no dispatch
incident and no false crash notice. Also folds the worker's captured stderr into
the error, which the pre-ack path already did (#112729) and this path dropped.
(cherry picked from commit d3f3571f8c6f4d2f188e90151d8f8393f049a573)
The dashboard's manual Run now binds the sibling profile via
web_server_cron._cron_store_scope, not the ticker's _profile_cron_scope,
so the marker-based routed_profile_fire() stayed False there: the routed
profile's .env was reloaded into the shared os.environ and the managed
external-worker handoff serialized multiplex_active=False.
routed_profile_fire() now compares the fire's active HERMES_HOME with
get_routing_process_hermes_home(), so every entry point (ticker,
dashboard Run now, CLI, gateway forward) and the external-worker handoff
share one answer. Regression test drives the real dashboard fire path
A -> B -> A with no_agent jobs.
Main moved the no_agent script env to the factory (build_subprocess_env(strip_launch_profile=True),
e1c0896518) so the spawn site takes no raw os.environ snapshot. The restored routed-fire overlay
(installed scope, then managed keys, both before the scrub) now lives behind that same flag instead
of a dict(os.environ) copy at the spawn site: strip -> overlay -> sanitize, unchanged order, and a
no-op outside multiplex semantics, so a single-profile child stays byte-identical. The flag's only
caller is the cron script runner.
(cherry picked from commit 800c37e7323443c4d4719c0d6a43e028018286e3)
_launch_external_cron_worker hand-rolled the same hydrate -> set_secret_scope ->
set_multiplex_context(routed) sequence, and the same context-then-scope reset, that
_install_fire_secret_scope/_reset_fire_secret_scope already encode for the in-process
fire. Two copies of the ordering is how the two paths drift apart; the helper is the
one place that owns it. The only observable difference (re-setting an already-active
multiplex context for the span) is a no-op the token reset undoes.
Rename _scope_token in _run_one_job_body to _fire_scope_tokens: it has held the helper's
(scope, context) tuple since the routed-fire change, not a single token.
(cherry picked from commit 840c00c124)
(cherry picked from commit cf5074f092194d8b989ecec588920c2165e9cdc2)
_launch_external_cron_worker wrapped fifty lines of dispatch, payload
write and scope hydration in the routed-fire multiplex context, though
only build_subprocess_env / strip_launch_profile_env read it. Compute
`multiplex_active` once (process flag OR routed fire), serialize it into
the payload, and set the context for exactly the env build inside the
existing secret-scope try/finally. Drop restore_managed_env on this path:
the worker re-runs load_hermes_dotenv -> _apply_managed_env at import
and strip_launch_profile_env already leaves managed keys in place.
_run_job_script now overlays the installed scope and re-applies managed
keys only under multiplex. In a single-profile process the scope is
os.environ, so the overlay could only re-sanitize values the child
already inherits; the "no-op outside multiplex" comment is now literal.
(cherry picked from commit 0fd59b447a)
(cherry picked from commit abfdbbc3b3e3e49906936d802a96e51e4c4961ef)
