Commit Graph

592 Commits

Author SHA1 Message Date
Teknium
13fb87af92 refactor(cron): decompose scheduler run_job/_deliver_result and dedupe delivery lanes
cron/scheduler.py (8535 -> 6353):
- _deliver_result split into per-target helpers: _resolve_target_transport (live/relay/standalone
  transport + enablement), _deliver_via_live_adapter (_live_route_metadata for Telegram DM-topic
  vs forum routing, _live_send_text with the cancel()-based timeout disambiguation,
  _live_send_media, _seed_live_delivery_sessions), _standalone_send/_deliver_standalone. The
  three interpreter-shutdown skip branches and the repeated log+append+continue pattern collapse
  into one _note_target_error / one shutdown message; _TargetDelivery carries per-target state.
- Thread/channel session seeding unified into _seed_cron_session (was two near-identical
  functions).
- run_job decomposed into _run_no_agent_job, _apply_monitor_gate, _load_cron_job_config
  (_CronJobConfig), _resolve_job_runtime, _check_model_drift, _open_cron_session_db,
  _run_agent_with_watchdog, _finalize_cron_session, plus one _run_doc_header and one _audit
  closure for the success/failure paths.
- _run_one_job_body: ownership-lost bookkeeping, delivery composition and outcome classification
  extracted; tick: _acquire_tick_lock/_release_tick_lock, _maybe_reap_dead_owners,
  _sweep_stale_inflight_for_tick, _process_due_job, _submit_with_guard.
- _build_job_prompt: context_from injection and skill loading extracted; one
  _prepend_context_block for the four fenced-data blocks.
- One _start_heartbeat_thread for the script- and fire-claim heartbeat threads.
- Dropped unreachable return in SharedRouteAdapters.get; boolean-return and nested-if shapes
  collapsed.
- Comments/docstrings compacted by hand, rationale kept (fd-leak reason for the late SessionDB
  close callback, title-persistence rules, no_agent classification gate, inactivity-vs-provider
  timeout ordering, stale-claim force-release, interruption token keying).
2026-09-02 13:31:43 -07:00
Teknium
e4711620a1 refactor(cron): simplify jobs.py store and lifecycle_guard
jobs.py (4654 -> 3332):
- _acquire_flock/_release_flock: one bounded LOCK_NB acquisition shared by _jobs_lock and
  _fire_job_lock (was two copies).
- _get_due_jobs_locked split into per-concern helpers over a _DueScan state object
  (record repair, persisted-error re-arm, timezone-migration catch-up, missed-run fast-forward,
  per-job due evaluation); raw-record repair loops unified.
- atomic_write_text used for heartbeat/counter/error/output writers (drops
  _atomic_write_counter/_atomic_write_epoch duplicates); save_jobs stage+verify loop deduped.
- create_job/update_job field normalization table-driven (_normalize_job_updates,
  _normalize_str_list, _normalize_context_from); parse_schedule croniter validation via
  _cron_schedule; single-record mutators routed through _with_job; completion/activation record
  writers and claim refresh shared; _parse_aware used for record repair and stale-error re-arm.
- Dead: clear_drift_alerted, _set_preflight_alerted shim (zero references).
- Comments/docstrings compacted by hand; rules, invariants and WHY (profile-home anchoring,
  bounded lock rationale, timezone-naive schedule rule, missed-run visibility) kept.

lifecycle_guard.py (1271 -> 906): shared cloud-path check, executed-command index,
resolve-or-skip yield, control-preserving segment split; regexes and matching semantics
byte-preserved; comments compacted.
2026-09-02 13:30:19 -07:00
Teknium
064bcaf3c1 refactor(cron): compact scheduler_provider and small cron modules
- scheduler_provider.py: _profile_entry/_profile_cron_scope replace four hand-rolled
  home-override+store blocks in the multiplex ticker; comments compacted to the WHY.
- incidents/executions/notepad/monitor/suggestions/blueprint_catalog/__init__: docstrings
  and comments compacted; no code change (AST-identical).
2026-09-02 13:29:30 -07:00
kshitijk4poor
95f62ca3bf fix(cron): validate failure_deliver at preflight and dashboard update lanes
Follow-up to the failure_deliver salvage (#100375):

- _preflight_check_delivery also checks the failure lane, so a typo'd
  failure_deliver platform blocks at config-validation time instead of
  surfacing only when a failure occurs — exactly when the notice must
  not be lost. Duplicate lanes are checked once.
- The dashboard cron-update normalizer treats failure_deliver like
  deliver (text normalization; empty clears the optional override
  instead of coalescing), closing the one update path that could write
  an unnormalized value into jobs.json.

4 guard tests; both fixes mutation-checked (neutralize -> red, restore -> green).
2026-09-02 20:16:14 +05:30
Victor Kyriazakos
fd35e1ec5a fix(cron): delivery bookkeeping reads the failure lane it actually routed through
Review findings (Salt, NS-788):

B1: delivery_outcome classification, unresolved_origin, and incident
'alerted' marking all read the deliver lane while the notice itself was
routed through failure_deliver — a silenced failure recorded
delivery_outcome='delivered' and marked its incident alerted (corrupting
the 'failure seen' vs 'operator was pinged' distinction the incident
store documents), and a failure delivered via failure_deliver over an
unresolvable deliver=origin recorded 'not_configured'. New
_delivery_lane_value() helper feeds the SAME lane to routing and
bookkeeping at all five sites (both classifiers, both unresolved_origin
computations, both zero-target checks). Three regression tests assert
outcome + alerted-marking; verified to bite on the pre-fix classifier.

S1: failure_deliver now goes through _resolve_cron_context_deliver on
tool create/update, matching deliver — a job created from inside a cron
run can no longer store literal 'origin' in its failure lane.

S2/T1: corrected the false 'same helper' comment in create_job; the
str/list flatten mirrors the tool layer for direct callers.

Full cron suite + interrupt tests: 87 files, 1112 passed, 0 failed.
2026-09-02 20:16:14 +05:30
Victor Kyriazakos
c9491e6a7d feat(cron): per-job failure_deliver — route or suppress failure notices (NS-788)
Coatue FR (Frank Long): jobs delivering into shared channels publish
engine failure notices ('⚠️ Cron X failed…') to those channels with no
opt-out. Adds an optional per-job failure_deliver field sharing
deliver's grammar: on failure, targets resolve from failure_deliver
when set (local = structural silence; state still recorded in
last_status/last_error/run history). Success delivery is unchanged;
absent field = today's behavior byte-for-byte.

Honored by every failure-category engine notice: the run_job failure
summary (+streak nudge), the escaped-failure retry path, drift-skip and
blocked-config alerts (composed into the same delivery), and the
gateway-shutdown interrupted-run notice (_notify_interrupted_cron_jobs).

Surfaces: cronjob tool create/update (same bot-chat validation as
deliver; '' clears on update), hermes cron create/edit
--failure-deliver, docs tip in automate-with-cron.

Existing fake_deliver test doubles gained **kwargs for the new
for_failure keyword — signature-compat only, no behavior change.
2026-09-02 20:16:14 +05:30
Teknium
a6351a71e5 fix(cron): route a credentialless satellite's cron delivery through the primary adapter for exact profile_routes targets (#101113)
Under gateway.multiplex_profiles a shared-token satellite profile (routed
via gateway.profile_routes, no bot credential of its own) got an empty
adapter map from the multiplex ticker, so _deliver_result fell through to
the standalone sender under the satellite's secret scope and failed with
"DISCORD_BOT_TOKEN is not set" — even though the primary adapter owns the
exact routed channel and had delivered the same target before. Preflight
already rescued this topology (#97476); the delivery half did not.

- cron/scheduler.py: factor the preflight's primary-config route loader into
  `_primary_profile_routes_for_current_home()` (one owner for both halves,
  so route semantics cannot drift) and add `SharedRouteAdapters`, a
  read-only view over the primary adapter map that resolves an adapter for
  a (platform, target) ONLY when an enabled primary route with a
  chat_id/thread_id maps that exact target to the current profile —
  using the same `ProfileRoute.matches` predicate as inbound routing.
  `_deliver_result` resolves the transport per target from it; everything
  else (unmatched chat, disabled route, route for another profile, no
  primary adapter, guild-only route) is a miss and never uses the primary
  bot. Execution stays scoped to the satellite; no credential is copied.
- cron/scheduler_provider.py: a secondary with no adapter map of its own
  gets the SharedRouteAdapters view instead of `{}`. This is NOT a default
  fallback: with no matching route the view is falsy and delivers nothing.

Fixes #101113
2026-09-02 06:27:24 -07:00
Teknium
62a4599f89 fix(cron): keep profile scope on the standalone fallback pool; desktop ticker stands down for profiles with their own gateway (#100489)
Two mechanisms let the desktop multiplex ticker deliver a secondary
profile's cron output through the default profile's identity:

1. _deliver_result's `asyncio.run` ThreadPoolExecutor fallback (taken when
   the caller already has a running loop — the desktop dashboard shape) ran
   the standalone sender on a fresh thread with NO profile ContextVars: the
   home override and secret scope were gone, so the sender resolved the
   process default's home/token (or, fail-closed under multiplex, raised
   UnscopedSecretError). Wrap the submit in copy_context().run like the
   session-db (:6562), heartbeat (:4650) and parallel-pool (:8314) workers.

2. _start_desktop_cron_ticker ticked EVERY local profile, including ones
   whose own gateway (with live adapters) is running; winning the tick-lock
   race meant the adapter-less desktop ticker delivered standalone. The
   multiplex loop gains an optional per-cycle `profile_gate(name, home)`;
   the desktop wires it to `_check_gateway_running(home)` so such profiles
   are neither ticked nor heartbeated by the dashboard while their gateway
   is alive (re-evaluated every cycle, no restart needed).

Fixes #100489
2026-09-02 06:27:24 -07:00
Cyber-Yichen
a2fea79de6 fix(cron): isolate multiplex profile failures per profile (#74878)
One profile's broken cron store no longer takes the whole multiplex ticker
down with it:

- startup recovery loop: a per-profile exception (e.g. an unreadable
  executions.db raising sqlite3.DatabaseError) was uncaught and killed the
  ticker thread before its first tick — no profile ever fired.
- tick loop: only CronTickYielded was caught per profile; any other
  exception escaped to the cycle-wide handler, skipping every remaining
  profile that cycle and marking all of them failed.

Both loops now catch per profile, record the failure into THAT profile's
ticker_last_error (`hermes cron status`), and keep ticking the siblings.
The existing CronTickYielded/_profile_errors semantics and the #87644
EMFILE reclaim/backoff are preserved (backoff is applied once per cycle
from the worst per-profile failure).

Salvaged from PR #70747 (@Cyber-Yichen); the recovery test's real
sqlite3.OperationalError shape is from PR #74888 (@OYLFLMH). Same class
also reported in PR #74952 (@webtecnica).

Co-authored-by: OYLFLMH <95945448+OYLFLMH@users.noreply.github.com>
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
2026-09-02 06:27:24 -07:00
이민재
e48bb828d4 test(cron): preserve explicit suggestions path override 2026-09-02 06:27:24 -07:00
wanliqin
9da8842585 fix(cron): resolve profile store paths per call
Resolve notepad and suggestion paths at transaction time so multiplexed profile ticks cannot write into the import-time home. Preserve explicit test overrides and cover writes after a profile context switch.

Co-authored-by: 이민재 <19909783+honor2030@users.noreply.github.com>
2026-09-02 06:27:24 -07:00
muhifni
1cd736ff63 fix(terminal): scope terminal config per turn under profile multiplexing
A multiplexed Hermes process (gateway.multiplex_profiles, unified
dashboard/TUI, or cron) serves several profiles at once, but terminal.*
resolved through process-global TERMINAL_* env vars bridged ONCE at
startup from the launch profile (gateway/run.py ~2700-2760) plus the
one-shot _ensure_terminal_env_bridged() guard. Every routed profile
therefore inherited the launch profile's backend, cwd, docker volumes,
SSH target and shared-container key: a local profile ran inside another
profile's docker sandbox (or a docker profile escaped to the host), and a
container labeled profile A carried profile B's RW bind mounts.

Fix: an authoritative per-profile terminal policy seam, mirroring
agent/secret_scope.py:

- tools/terminal_scope.py: ContextVar holding the routed profile's
  COMPLETE effective TERMINAL_* policy (defined defaults <- profile .env
  TERMINAL_* <- config.yaml terminal:). While bound, terminal_env()
  resolves ONLY from it - an omitted key yields the defined default,
  never os.environ. Unreadable/malformed policy installs a refusal
  scope; terminal_tool / execute_code refuse instead of running under
  ambient launch-process policy (fail closed).
- Installed at every in-process profile boundary: gateway
  _profile_runtime_scope, tui_gateway session/build/turn scopes, cron
  per-job fire. The unscoped single-process path is byte-identical.
- Every terminal.* consumer reads through the scope: terminal_tool
  (_get_env_config, _resolve_container_task_id shared key, orphan
  reaper lifetime, degraded mode), gateway/platforms/base.py docker
  media translation (volumes, shared key, persistence), runtime_cwd /
  agent_init / skill_utils / code_execution_tool / file_tools cwd
  anchors, prompt_builder / browser_tool / env_probe backend checks,
  gateway footer, @-refs and slash-command cwd. env_probe resolves the
  backend in the caller's context, since the probe worker thread does
  not inherit the ContextVar.

Salvage of #99225 onto current main: adds the three ambient reads the PR
missed (tools/file_tools.py TERMINAL_CWD, tools/browser_tool.py and
tools/env_probe.py TERMINAL_ENV; shape from #79117) and trims the test
module to the leak matrix driven through the real gateway boundary,
omitted-key defaults, refusal, and boundary reset.

Fixes #68559
Fixes #94200
Fixes #101132
Fixes #95470

Co-authored-by: x7peeps <9640837+x7peeps@users.noreply.github.com>
Co-authored-by: Eva <239388517+100yenadmin@users.noreply.github.com>
Co-authored-by: ExitMaster <292490062+ExitMaster@users.noreply.github.com>
2026-09-02 05:34:28 -07:00
kshitijk4poor
bcb412cd9d refactor(cron): share the fire-path telemetry recorder
_record_timezone_migration_catchup was a line-for-line clone of
_record_persisted_error_recovery (counter bump, bounded recent list,
best-effort jsonl append). Extract _append_telemetry_record and route
both through it; one shared history cap replaces the two per-counter
constants. Also correct the "distinct from catch_up_occurrences" comment:
a migrated row that is also past its grace window increments both.

No behavior change; both recorders write the same entries to the same
files.
2026-09-02 15:13:46 +05:30
deinte
187251800b fix(cron): don't silently skip a due run after a timezone-offset migration
Upgrading from a UTC-scheduling build to one that honours the profile
timezone (Europe/Brussels) left daily cron jobs sitting in jobs.json with
pre-migration instants — e.g. next_run_at "2026-09-02T04:00:00+00:00" for
expr "0 4 * * *". _ensure_aware normalizes that to 06:00+02, which the
expression excludes, so the stale-expression guard (#93049) read it as a
direct jobs.json edit, logged exactly that, and re-anchored to tomorrow
without firing. The due occurrence disappeared with no error anywhere.

The guard only asked "is the stored instant an occurrence of the current
expr?", never "why not?" — and the two possible answers demand opposite
actions. Add _classify_stale_cron_next_run, which distinguishes them by
whether normalization itself moved the wall clock:

  * expr_edit           — wall clock unchanged (or the stored wall clock is
                          not an occurrence either): the instant is genuinely
                          excluded by the current expression. Re-anchor
                          without firing, exactly as before.
  * timezone_migration  — the stored value's own wall clock IS a legal
                          occurrence and it only left the lattice because
                          _ensure_aware converted it to a different offset.
                          Fall through and fire the overdue run once.

Because every value written by this build carries the configured offset, a
real expr edit leaves the wall clock untouched and can never be reclassified
as a migration, so the #93049 protection is intact. At-most-once is
unchanged: the fire flows through the normal due path and the usual
advance_next_run / mark_job_run re-anchor rewrites next_run_at in the
current offset, so the legacy instant is never read again. Future local
wall-clock occurrences are untouched — not-yet-due rows never reach the
guard, and the #28934 offset-repair branch still runs first for a
still-future stored wall clock.

The migration case is classified explicitly rather than retried broadly: it
logs cron.timezone_migration.catch_up with the stored and normalized
instants plus both offsets, and increments a probe-visible counter
(get_timezone_migration_catchup_stats, timezone_migration_catchups.jsonl)
kept separate from catch_up_occurrences so an operator can tell "the upgrade
backlog is draining" from "runs are missing their grace window".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-02 15:13:46 +05:30
Teknium
00a7115a02 fix(cron): make cron push-notify configurable (cron.delivery.notify) and surface UNVERIFIED live deliveries in cron list/doctor
De-risking for the notify=True UX change: the marker is now driven by
cron.delivery.notify (config.yaml, default true = current behaviour), read
once per delivery and applied to both the text and media routes; a missing or
malformed section keeps the default.

An evidence-free live-adapter ack (bare SendResult(success=True) from
Slack/Matrix/Mattermost) is still accepted, but the target is recorded on the
job as last_delivery_unverified (cleared by the next evidenced delivery) so
the state shows up in 'hermes cron list' (⚠ Delivery UNVERIFIED), 'hermes cron
doctor', and the cronjob tool listing — not only in a WARNING log line.

Live repro (real _deliver_result + real 'hermes cron list' against a temp
HERMES_HOME, Slack target, SendResult(success=True)): before — list showed
nothing beyond the Deliver line and route metadata always carried
notify=true; after — list prints the UNVERIFIED line, and
cron.delivery.notify: false yields notify=false in the route metadata.
2026-09-02 00:56:52 -07:00
yoma
4b69ba22fc fix(cron): mark live deliveries as final notifications 2026-09-02 00:56:52 -07:00
kaiomp
dd72b42ba4 fix(cron): require positive evidence for live-adapter delivery confirmation
A cron job fired, the scheduler logged "delivered to telegram:<chat> via
live adapter", and nothing reached Telegram (#77763). The log line was not
evidence of a send:

* the silence-narration filter returns {"success": True, "delivered": False}
  (a successful *drop*), and the dict-normalization branch read only
  "success", so a filtered message counted as delivered;
* an empty payload (no text, no media) skipped the send entirely and still
  fell into the "delivered" branch;
* the log line named the chat but not the lane, so a wrong-thread delivery
  and a phantom one are indistinguishable after the fact.

_confirm_adapter_delivery now inspects both result shapes: an explicit
`delivered: False` is a rejection even with a truthy `success`, and a
success with no message_id and no raw_response is accepted but logged as
UNVERIFIED. The empty-payload case fails closed into the existing
standalone/warn handling, and the delivered log carries thread= and
message_id=.

Failing closed on the live lane is only half the fix on a native target:
the standalone fallback sent the same empty payload, and the Telegram
adapter returns SendResult(success=True) for empty content without an API
call — a phantom live delivery became a phantom standalone one. Both
_send_to_platform call sites now sit behind one skip guard, so "empty
payload fails closed" holds on every lane (#77763).
2026-09-02 00:56:52 -07:00
Justin Wilson
8fd76fd1d6 fix(cron): surface delivery_failed instead of last_status ok
A successful agent run whose delivery failed used to persist
last_status=ok and bury the failure in last_delivery_error. CLI list
painted that as green and the run looked identical to a quiet success.

Record last_status=delivery_failed instead, keep last_delivery_error,
do not increment failure_streak, and teach cron list/doctor not to
treat it as ok.

Fixes #83993
2026-09-02 00:52:58 -07:00
Teknium
71c4bcf7af fix(cron): surface missed-fire catch-up lateness in hermes cron list / status
The catch-up machinery already re-ran jobs missed during gateway downtime,
but the late execution rendered as an ordinary on-time success — no
scheduled-vs-actual time, no lateness, no disposition (issue #99879, the
visibility half).

- Due-scan now persists a `last_dispatch` stamp on every recurring dispatch:
  scheduled_at, dispatched_at, lateness_seconds, and kind
  (on_time / late / catch_up, classified against the ticker tolerance and
  the schedule's catch-up grace window). Manual triggers and one-shots are
  not stamped (no scheduled instant to be late against / retired beyond
  grace).
- `hermes cron list` renders a per-job Dispatch line; late/catch-up runs
  show "⚠ catch-up after missed fire: scheduled ..., ran ... (31m late)".
- `hermes cron status` calls out jobs whose last dispatch was late or a
  catch-up, in both the built-in ticker and external provider paths.

CLI surface only — no new tools, no policy engine.

Addresses the visibility half of #99879.
2026-09-01 08:31:37 -07:00
kshitijk4poor
db339f0051 fix(state): consolidate gateway SessionDB writers via process-wide shared registry
A gateway process opened state.db from ~12 call sites, each minting its
own writer connection, self._lock, close-time WAL checkpoint, and
token-writer thread. With N independent writers on one WAL file, one
connection's close-time checkpoint could race another's growth — the
lost/reordered-page-write signature across 11+ incidents (#90837).

Adds hermes_state_registry.py: a process-wide, per-path, refcounted
shared registry owning the writer boundary.

- acquire(path): same resolved path returns the same instance (one
  writer connection, one lock, one token-writer thread) for every
  long-lived in-process caller (gateway runner, SessionStore, per-agent
  lazy recall, cron per-job, mirror, channel_directory, slash_commands,
  shutdown_flush, session_search, react_to_message, delegate, mcp_serve,
  auto_archive, tui_gateway).
- close() on a shared instance is a NO-OP — the registry owns the
  lifecycle, so one caller's close can never tear down a writer other
  callers still hold.
- Generation-aware retirement on inode change: a replaced state.db
  RETIRES the live generation (never lent again) but keeps it alive for
  existing holders; release is object-keyed so holders of the old
  generation drain it independently of the new one. The old
  generation's own write path still fails with the typed
  StateDbReplacedError (existing protection, unchanged).
- Replacement-open failure leaves NO registry entry for the path —
  the next acquire retries fresh, never hands out a closed stale object.
- All teardown runs OUTSIDE the registry lock: a final release's WAL
  checkpoint can never stall acquisition for every state.db.
- close_shared_session_dbs() at gateway shutdown drains every
  generation (live + retired) as the final safety net.

CLI one-shots, recovery flows, and read-only cross-profile opens keep
using SessionDB() directly with their own close() — only long-lived
in-process sites route through the registry.

References #90837 (root-cause tracker stays open: the #10 EOF signature
and the WAL-lifecycle A/B verdict remain under investigation there).
2026-09-01 20:55:35 +05:30
Teknium
be597fc730 fix: extend corrupt-config fail-closed guard to gateway, serve, and cron surfaces
Follow-up to the salvaged #81988 CLI guard (issue #81952):
- gateway/run.py::main() refuses startup (exit 2) on unparseable config.yaml
- hermes serve headless path (cmd_dashboard) gets the same guard
- cron run_job() fails the job with the guard error before AIAgent
  construction (no_agent script jobs exempt — no token spend)
- HERMES_IGNORE_USER_CONFIG=1 / --ignore-user-config escape hatch honored
  on every surface
2026-09-01 07:00:22 -07:00
Teknium
c74cf2333c fix: restore _inactivity_watchdog_loop dropped in rebase conflict resolution 2026-08-31 10:42:39 -07:00
HexLab98
85bc25c949 fix(terminal): bound env.execute wait so a wedged poll cannot disable every timer
A hung terminal wait on the loop thread silently disabled asyncio deadlines
and let cron jobs idle thousands of seconds past HERMES_CRON_TIMEOUT. Drive
the wait from run_bounded_sync (sliced Event.wait, kill-on-timeout) and
move the cron inactivity monitor onto a daemon thread with the same kernel
timeout primitive. Copy the caller ContextVar scope and activity callback
onto the wait worker so profile secrets, session id, and heartbeats survive
the thread hop (#94285).
2026-08-31 10:42:39 -07:00
Justin Adkins
d957e0e403 fix(cron): bound local fire-fence waits 2026-08-31 09:59:47 -07:00
Andrew Bagrin
b7c59bda54 fix(cron): isolate per-execution working directories 2026-08-31 09:59:39 -07:00
Jay. (neocode24)
9a7732b45f fix(cron): stale ticker yields its tick to a fresh gateway
A long-lived process whose checkout was updated underneath it (hot git
pull, interrupted hermes update) serves mixed sys.modules. When such a
stale process races a fresh gateway for the cron tick lock and wins the
minute, every agent job it dispatches can die on ImportErrors whose real
cause is staleness — and the fresh gateway's ticker skips the same minute
as lock-loser, so the user's scheduled job fires broken or not at all.

tick() now checks, BEFORE acquiring the tick lock:

  skew detected (boot fingerprint != disk revision)
    AND this process does not own the gateway runtime lock
    AND that lock is held (a fresh gateway is alive)
      -> raise CronTickYielded, skipping the tick entirely

Each arm alone keeps the old behavior:
- skew + self-owned lock -> proceed (delivery-path stale-code hint stays
  the surface for gateway-owned dispatches)
- skew + no lock holder -> proceed (desktop-standalone users must not
  lose their only ticker to a silent yield)
- skew None (non-git install, no boot fingerprint, probe failure) ->
  proceed; yielding is a certainty claim, never a guess

The yield RAISES instead of returning 0 so the provider loops record it
via record_ticker_error and mark the heartbeat success=False — a yielded
tick must not look like a healthy one (hermes cron status shows why),
mirroring the EMFILE propagation contract (#87644). Yield logging is
throttled to once per skew episode. Self-healing: when the fresh gateway
dies, its lock releases and the stale ticker's next tick proceeds.

Multiplex loop: a yield for one profile no longer cancels sibling
profiles' ticks in the same cycle; only the yielding profile records an
unsuccessful beat.

gateway/status.py gains owns_gateway_runtime_lock() —
is_gateway_runtime_lock_active() is True for the lock's own owner too, so
a caller deciding whether to yield to a FRESH gateway needs the
in-process handle as the discriminator.
2026-08-31 09:59:07 -07:00
fangliquanflq
fd1d8271db fix(cron): isolate lazy imports from stale modules 2026-08-31 09:58:51 -07:00
Teknium
3a351a9665 feat(worktree): pushed open-PR lanes reclaim their disk; cron tick prunes worktrees
Two growth leaks closed:

1. Pushed-branch tier (the dominant survivor class — 24 of 33 preserved
   trees, ~18GB on the reporting box): managed installs fetch with a
   single-branch refspec, so pushed PR branches never get refs/remotes/*
   entries and read as 'unpushed' forever. When a clean tree's branch head
   EXACTLY matches origin (one lazy git ls-remote per sweep), the checkout
   is redundant: reap the TREE, keep the BRANCH ref (shielded from the
   orphaned-branch pass). Anything diverged/unverifiable stays preserved.
   Applied to both the startup pruner and hermes worktree prune/list.

2. Cron-tick maintenance: the pruner only ran on hermes -w launches, so
   gateway-driven boxes accumulated trees for days. The scheduler tick now
   dispatches the same conservative pruner on a daemon thread, throttled
   to once per 6h, against the install checkout + job-workdir repos that
   have a .worktrees/ dir.
2026-08-31 07:27:50 -07:00
Teknium
43ca78aa40 fix(cron): keep SessionDB kwargs-free in the context-preserving worker
The late-result close callback (#72782) retrieves the future's SessionDB
and closes it; passing an explicit db_path kwarg broke the hanging-init
test's mock shape. contextvars.copy_context().run(SessionDB) alone is
sufficient — SessionDB resolves its default path from get_hermes_home(),
which reads the profile ContextVar.
2026-08-31 06:02:32 -07:00
liuhao1024
d5d0613778 fix(cron): profile_routes rescue for satellite-profile delivery preflight
Under gateway.multiplex_profiles the primary gateway's in-process ticker
fires satellite-profile jobs and delivers through the primary's live
adapters (#69377) — the satellite home intentionally holds no platform
credentials (its own token would be a duplicate_credential fatal).
_preflight_check_delivery loads the gateway config of the job's OWN home,
so a profile_routes-routed platform reads as unconnected there and the
job is permanently blocked before any LLM call with a misleading
"not connected" error (#97476).

When the own-home config reports a platform unconnected, consult the
primary home's profile_routes: an enabled route matching the platform
that points at the profile currently being served means delivery is the
primary gateway's to make — pass the check. The primary config.yaml is
read directly (both top-level and nested gateway. forms) instead of via
load_gateway_config() so no primary platform config leaks into the
satellite process's environment. Lookup failures and missing configs
fail closed (the block stands).
2026-08-31 06:02:32 -07:00
Kevin
dfa5004a04 fix(cron): preserve profile context during session DB init 2026-08-31 06:02:32 -07:00
Jason S. Mow
edd6ad53ab fix(cron): resolve multiplex home-channel chat id from the owning profile's secret scope
The #83182 fix moved the secret scope to span execute→deliver so the
delivery path resolves platform TOKENS (e.g. TELEGRAM_BOT_TOKEN) from the
job-owning profile's .env instead of the host process os.environ. But
cron delivery also resolves the DESTINATION via the legacy
<PLATFORM>_HOME_CHANNEL env mirror, and _env_home_target_chat_id /
_get_home_target_thread_id read it through raw os.getenv — NOT
get_secret. So under multiplex, a socrates cron job whose tick was won
by the default gateway process resolved DISCORD_HOME_CHANNEL from the
default profile's os.environ and landed in the default pax-hermes-chat
instead of socrates-hermes-chat.

Make the chat-id and thread-id resolution legs read through get_secret
when a profile secret scope is installed (mirroring the token leg), so
the owning profile's home channel wins. Non-multiplex / no-scope callers
keep reading os.environ as before.

Verified: with os.environ=default chat and the socrates scope active,
_env_home_target_chat_id('discord') returns the socrates chat; with no
scope it returns the os.environ legacy value.
2026-08-31 06:02:32 -07:00
Muhammad Usama
a07370abd0 fix(cron): reserve shared adapters for the default profile only
Addresses review feedback on #73363. The previous truthy
`profile_adapters.get(name)` check fell back to the shared (default-profile)
adapters whenever a secondary profile's adapter map was empty — which is the
normal state before that profile's bot connects (the map is created empty and
filled only on a successful connection). That reintroduced wrong-bot
cross-delivery for the secondary until it connected.

Thread the default profile identity and reserve the shared `adapters` set for
it alone; every other profile uses its own adapter map, or an empty set when
its bot has not connected yet (so it simply does not deliver that tick).

Add regression coverage for the default, connected-secondary, empty-secondary
and missing-secondary adapter-map cases.
2026-08-31 06:02:32 -07:00
Muhammad Usama
51e377d550 fix(cron): deliver each profile's cron via its own adapter in multiplex
Builds on a61183b (per-profile HERMES_HOME scoping in the multiplex ticker).
Cron EXECUTION is now scoped per profile, but _start_multiplex still passed
the default profile's `adapters` to `cron_tick` for every profile. As a
result all secondary profiles' cron delivery (origin/home-channel replies and
failure summaries) was sent through the DEFAULT profile's platform adapter --
e.g. a secondary Telegram profile's scheduled-job output went out the DEFAULT
bot instead of that profile's own bot.

Thread a `profile_adapters` map (profile name -> adapters) from the gateway
(GatewayRunner._profile_adapters) into InProcessCronScheduler.start and select
the per-profile adapter set in the multiplex tick loop, falling back to the
shared `adapters` for the default profile (which owns them). No change to
single-profile behavior.
2026-08-31 06:02:32 -07:00
ghosty93
b8c18b8cb7 docs(cron): explain why the secret-scope reset must stay function-level
Review follow-up: the comment records that the reset scopes delivery,
deferred-agent teardown, claim-loss handling and bookkeeping, and that an
earlier inner-finally placement left _deliver_result unscoped — so a
tidy-up does not silently reintroduce the bug.
2026-08-31 06:02:32 -07:00
ghosty93
07f6518b6d fix(cron): retain profile secret scope through delivery 2026-08-31 06:02:32 -07:00
Teknium
52e5e7c034 fix(cron): coerce string repeat values on the UPDATE path too
The create-path coercion (salvaged from #78928) left update_job and the
cronjob tool's update handler comparing/storing raw strings: repeat=
'forever' via update raised TypeError in the tool path and stored the raw
string via update_job, breaking the next mark_job_run ('str' has no
.get). Extract normalize_repeat_value as the shared chokepoint (shape
from #77366 by @andrexibiza, with garbage-rejection semantics) and route
create_job, update_job, and the tool update handler through it.
Completed counters are preserved across repeat updates.

Class: #66824 #64520 #7142 #71987 #95706, update half of #77366.
2026-08-29 20:08:53 -07:00
Teknium
609a0b47fb fix(cron): teach the parse error the corrected bare-duration contract 2026-08-29 19:19:40 -07:00
andrexibiza
e8bab87ba7 fix(cron): bare durations are recurring intervals; coerce repeat string forms
Contract bug (2026-08-04): the cronjob tool schema documents '30m' as
'(every 30 minutes)' — recurring — but parse_schedule returned
kind='once' for bare durations, silently creating a one-shot job for a
recurring request (agent passed '30m' for 'every 30 min', job ran once
and died). Bare durations ('30m','2h','1d') now parse as recurring
intervals matching the documented contract; explicit one-shot by
duration is 'in 30m'/'in 2h' (fires once that far from now). ISO
timestamps stay one-shot.

Also fixes the sibling repeat-coercion class (#66824/#64520/#7142):
repeat='forever'/'once'/'N' strings now coerce in create_job instead of
raising "'<=' not supported between instances of 'str' and 'int'".

Tool description rewritten to teach the corrected contract and steer
relative requests to 'in Nm' (no more hand-computed ISO timestamps).
Supersedes the doc-only direction of #53739 while keeping its goal
(relative one-shots must be expressible) via the 'in X' form.

Signed-off-by: andrexibiza <84248988+andrexibiza@users.noreply.github.com>
2026-08-29 19:19:40 -07:00
Teknium
57ad23c5dc fix(cron): accept weekday lists and no-'every' natural schedules (#51975)
Widens _natural_every_to_cron to consume comma/'and'-separated weekday
lists ('Monday, Wednesday at 9am' -> '0 9 * * 1,3') and applies the same
helper to schedules without the 'every' prefix, matching the exact forms
the Desktop dialog advertises in the #51975 repro.
2026-08-29 19:14:51 -07:00
devorun
77fd6db4c6 fix(cron): accept named months/weekdays in cron schedules
`parse_schedule()` detected cron expressions with a digit-only field pattern
(`^[\d\*\-,/]+$`), so any field using named months or weekdays — `MON`, `JAN`,
and common ranges/lists like `MON-FRI` or `MON,WED,FRI` — failed detection and
fell through to a confusing "Invalid schedule" error, even though croniter
supports them and they're standard cron.

Allow letters in the field pattern so these route to croniter for validation.
Truly-invalid expressions (`0 9 * * FUNDAY`, `99 9 * * MON`) are still rejected
there with a clear "Invalid cron expression" message; duration/interval/ISO
parsing is unchanged.

Adds tests for named weekdays/months (incl. ranges and lists) and that an
invalid named field is still rejected.
2026-08-29 19:14:51 -07:00
Teknium
ecdc03c616 fix(cron): accept no-'every' natural day/time schedules like 'weekdays at 9am'
The Desktop dialog's advertised form in #51975 omits the 'every' prefix.
Reuse _natural_every_to_cron() on the bare schedule so 'weekdays at 9am',
'monday at 9:30', and 'daily at 7am' parse to cron expressions. Also
switch the salvaged branch's HAS_CRONITER check to _ensure_croniter()
(lazy-import refactor landed after the PR was cut).
2026-08-29 19:14:51 -07:00
Drexuxux
ab9d85287d fix(cron): accept documented "every <weekday> <time>" schedules
parse_schedule's "every " branch passed everything after the prefix
straight to parse_duration(), so documented natural-language schedules
like "every monday 9am" and "every day at 9am" (AGENTS.md, SKILL.md,
cron docs) were rejected with "Invalid duration". Convert weekday and
daily/weekday/weekend phrases to cron expressions before the duration
fallback; "every 30m"/"every 2h" interval parsing is unchanged.
2026-08-29 19:14:51 -07:00
Patrickk
792dbea777 fix(cron): forward request_overrides into scheduled-job agents
Salvaged from #56876 (cron half only; the delegation half is superseded
by #98237). run_job's ephemeral AIAgent constructor passed api_key /
base_url / provider / api_mode from the resolved runtime but dropped
request_overrides, so cron jobs on custom providers silently lost
extra_body / extra_headers request settings.
2026-08-29 19:12:51 -07:00
salch-cred
260f007c32 fix(cron): mention bare units in the duration parse error 2026-08-29 18:33:19 -07:00
salch-cred
1d13fe70b9 fix(cron): accept bare duration units like 'hour' in schedule parsing 2026-08-29 18:33:19 -07:00
kshitijk4poor
84d29488e9 fix(cron): harden _is_named_profile_path against symlinked profile homes
Check both resolved and unresolved path parts so a symlinked named
profile (e.g. profiles/dev -> /mnt/data/dev) is still detected.
Also use _ensure_cron_dir for output_dir in ensure_dirs() for consistency.

Simplify-code Phase 2 finding (medium severity).
2026-08-28 13:39:19 +05:30
kshitijk4poor
0dc9367163 fix(cron): widen deleted-profile protection to all cron mkdir sites
Replace #96637's inline active_profile_homes() closure with #96508's
module-level _existing_profile_homes() filter (testable in isolation).
Widen _ensure_cron_dir from 3 to 12 mkdir sites across cron/ so every
directory creation fails closed for deleted named profiles, not just
the 3 originally protected. Add _is_named_profile_path() that checks
'profiles' in path parts (works for subdirs like cron/output/<job> and
scripts/ that the original parent.name heuristic couldn't reach).

Co-authored-by: misterdas <das7514@gmail.com>
2026-08-28 13:39:19 +05:30
Gille
000d22b9db fix(cron): keep deleted profiles from returning 2026-08-28 13:39:19 +05:30
Teknium
31579f781e fix(cron): transient run prompt survives the relay-fronted gateway forward
cronjob(action='run', prompt=...) context was silently dropped when the
manual run forwarded to the gateway (#96010 follow-up): POST
/api/jobs/{id}/run took no body. The forward now sends {prompt} in the
request body; the api_server validates it (length cap + strict injection
scan, same as stored prompts) and trigger_job stamps it as a transient
manual_run_prompt alongside manual_run_at. run_one_job consumes the stamp
for that single fire and mark_job_run clears it, so it never persists
into the job definition or later scheduled fires.
2026-08-27 20:53:02 -07:00