Anthropic answers a probe that asks about "earlier reasoning" with a 200
content_filter refusal and no usage block, which left cross-turn cells
without a baseline. Both refusal shapes now rerun the cell with neutral
wording, and opaque cells pick the newest turn-1 capture per position so a
partial rerun does not mix captures.
Newer OpenAI/Gemini models emit no reasoning on a trivial turn 1, so step 2
had nothing to replay; turn 1 now poses a small arithmetic task and escalates
effort until a reasoning carrier and tool call come back. Opaque cells are
compared only within one turn-1 capture (run_id), xAI's safety-layer 403 on
"earlier reasoning" falls back to neutral probe wording per cell, xAI
cost_in_usd_ticks is read as billed cost, and keyless local servers cost $0.
A 200 only proves a provider accepted a replayed reasoning field; the gateway or
chat template may still drop it. This stdlib-only harness measures whether each
carrier (reasoning_content / reasoning / reasoning_details / Responses reasoning
items / Gemini thought parts and signatures) reaches the model's prompt: the
prompt-token delta against a byte-identical baseline, plus canary recall, for
in-loop and cross-turn positions, and real turn-1 opaque carriers replayed
verbatim vs stripped. Portal calls are keyed by lane (gen- = OpenRouter) and
can be resampled until a direct-lane result exists. Results stay out of tree.
_get_due_jobs_locked repairs store-side problems mid-scan (half-paused records
self-disable, completed one-shots are swept, records are normalized) and then
persists them with a bare save_jobs(...) before returning the due list. Any
store-write failure there — ENOSPC on a full disk, a read-only mount, a
permissions problem — raised out of get_due_jobs(), so cron_tick() aborted and
NO job on that profile ran until a write succeeded. Live on a full /opt/data:
"Cron tick error for profile at /opt/data: [Errno 28] No space left on device"
from that exact call site, repeating every tick, with every job then reporting
"missed its scheduled time" and re-anchoring.
The repairs are already applied in memory — that is what the scan keys off — so
the persist is a side effect the next tick retries. Contain the store write and
keep dispatching; a non-store error still propagates.
(cherry picked from commit 76676fb0084f06890f76280f50387325ad2d59c7)
tirith ran `tirith check` on every terminal command, default-on with fail-open, as a
pinned PM binary auto-downloaded at startup. Measured against the shipped 0.4.2 binary
it caught 14/16 attack commands but flagged 12/24 everyday agent commands
(`echo $((6*7))`, `"$VAR/x.sh"`, `gh api | python3`, LAN curls, `.dev` hosts,
`pip install` via analysis_incomplete), and in cron/-q deny mode each of those is a
hard block. It only ever saw terminal strings (execute_code, writes and the browser
bypass it), fail-open meant no protection wherever the binary could not install
(Windows, Termux, x64 musl), and it produced a steady issue/PR stream (fail_open
semantics four times, install loops filling disks, a SIGSEGV hang, session-approval
scoping, third-party promo text on approval cards).
Removed: tools/tirith_security.py, the tirith branches of the command guard (the
is_tirith tuple, session-only "always" downgrade, import-error/fail-open paths,
unattended tirith pass), the CLI and gateway startup installers, the PM package and
lock pins, the nix runtime dep, security.tirith_* defaults, the CLI warning string
and three tirith tips (gaps refilled from the tail so translations survive), plus
the gateway's "no automated assessor" warning that only existed to point at tirith.
Config v50 drops security.tirith_* and enables nothing in its place. Hermes ships
no tirith plugin; if tirith upstream publishes one, it goes through normal catalog
admission under their name.
plugins.isolation: host imports every non-bundled plugin in a host process started
on demand per profile with only that profile's environment, and hands plugins a
RemotePluginContext; Hermes registers proxies for tools, hooks, commands, skills and
provider objects (subclasses of the ABC it checks). Re-entrant: handlers may call
ctx back into Hermes while Hermes waits on them. Every user-code import path routes
to the host (general, memory/context-engine/cron, model-provider profiles via a
cached credential-free extraction, dashboard APIs via an ASGI bridge) or refuses.
One table names what cannot cross (live gateway objects); the static audit shown
by `hermes plugins validate/show` reads it. A host that dies is restarted with its
plugins (bounded). Default stays in_process: nothing changes unless opted in.
Why: shared-pool deployments must never run one tenant's plugin Python in an
interpreter holding another tenant's data, without plugins becoming tenant-aware.
The worker isolated HERMES_HOME only. The longrange_backup_pipeline task
("schedule it nightly at 2am") let agents run the operator's real crontab
and write into the real ~/.hermes: a past battery left three nightly cron
jobs and two scripts behind on the host, pointing into deleted run dirs.
sandbox.isolate_host turns on the product's own TERMINAL_HOME_MODE=profile
contract, points the worker's HOME at the same profile home (in-process
~ expansion for write_file), puts a file-backed crontab shim first on PATH
and unsets the systemd user bus. README quickstart drops its /tmp paths.
The probe scrubs the environment down to PATH/LANG/LC_ALL/TZ, which also
dropped the harness's HERMES_DISABLE_LAZY_INSTALLS. In a fresh home with no
tirith on PATH (CI), GatewayRunner() then starts tirith's startup install on a
background thread: the PM runtime downloads and unpacks Python and uv from
GitHub while the cases run. When that unpacking overlaps the archive case, its
buffers land in the process-wide tracemalloc window and push the peak past the
8 MiB bound, so the probe reports 19/20.
The loopback fence also missed it: hermes_bootstrap's happy-eyeballs
create_connection dials with connect_ex, so the downloads reached GitHub with
blocked_external_attempts empty.
- set HERMES_DISABLE_LAZY_INSTALLS=1 in the probe environment
- fence socket.connect_ex as well as connect
- assert the receipt records no external network attempts
tests/gateway/test_kanban_wake_acceptance.py intermittently hung until the
CI runner SIGKILLed it at 600s (normally ~4s). The faulthandler stack of a
reproduced hang is the loop thread inside the test's own drain() helper:
while adapter._background_tasks:
await asyncio.gather(*list(adapter._background_tasks))
Since Python 3.12, asyncio.gather() completes eagerly when every child is
already done, so `await gather(...)` returns without yielding to the event
loop. When a session task finishes in the same loop iteration that resumes
the test (e.g. right after the notifier's advance() to_thread hop), the task
is done but its _background_tasks.discard done-callback is still queued for
the next iteration. drain() then re-checks a set that can never shrink and
spins forever without ever letting the loop run that callback.
Yield once after each gather so queued done-callbacks run before the set is
re-checked. Same fix for the identical helper in
tests/gateway/test_completion_admission.py and evals/heartbeat_idle_wire.py.
Production code is not affected (BasePlatformAdapter.cancel_background_tasks
filters done tasks and is bounded to 5 rounds).
* refactor(desktop): split clarify-tool.tsx into a clarify/ folder
Pure moves, no behaviour change. Parsing, the question-card core
(shell, choice rows, question block), the delivery watchdog, and each
pending/settled card get their own file so the core can be reused.
* feat(clarify): one questions[] shape with per-question status and one outcome
The clarify tool now takes only questions=[{question, choices?, multi_select?}].
A wrong shape is a tool error that names the right one, so a model that
sends the old top-level question/choices corrects itself on the next call.
Every result has the same shape on every surface: each response carries
status (answered, skipped, unanswered) with user_response null unless
answered, and the result carries one outcome (submitted, cancelled,
timed_out, undelivered) plus an optional surface-written notice. The
timeout and cancel sentences that used to pose as the user's answer are
gone, and the compressor reads status instead of matching their prefixes.
The callback contract is callback(questions) -> {answers, outcome, notice?}
(None = skipped, missing qid = unanswered). tui_gateway settles a batch the
same way for the last lock, a cancel, an interrupt and the deadline, and
clarify.lock keeps a null answer as a skip. A multi-select answer that is
not a JSON array counts as one typed ("Other") answer. The tool-row preview
reads the first question.
* feat(clarify): every surface asks through the one question card
CLI: the batch panel is the only panel; Enter on an empty field skips a
question, Ctrl+C cancels and keeps the locked answers, the deadline returns
timed_out. -q and -z return undelivered with a no-user notice.
Ink TUI: the single-question prompt is gone; the card supports multi-select
(Space or a digit toggles, Enter locks, typed Other text joins the picks),
an empty submit skips, Esc cancels, and "Batch" names are dropped.
Desktop: the single-question card is deleted and its keyboard handling moved
into the one card (arrows, letters and digits, auto-advance, Enter picks then
confirms). Confirm enables at one answer; blank questions lock null; Skip and
a composer message cancel; the settled card shows No answer for unanswered
questions. The bots room card follows the same contract.
Messaging: one card per question with a "Reply skip to skip" line in every
locale; timeouts and delivery failures map to timed_out and undelivered.
* fix(clarify): cancelled for a released messaging card, labels win over the skip word
A prose reply to a messaging choice card, /new and session end released
the wait with "", which read as timed_out. They now resolve with a
CANCELLED marker and the tool gets outcome cancelled.
A typed reply that matches a choice label ("Skip") now resolves to that
choice; the skip word applies only when no choice matches.
The bots room card sends the picked labels as they are; the tool already
strips the recommendation label. Unused OUTCOMES and a new docstring and
comment are removed.
* fix(tui): re-editing a multi-select answer keeps its picks
The Ink card stored a multi-select answer as display text ("A, B"), so
revisiting the question put the whole string into Other and re-locked it
as one item. The answers map now keeps the raw JSON array, the card
restores the picks on revisit, and only the display lines format it.
Comments that earlier edits reworded are restored to their original
words, minus the phrases the change made wrong. The compute-host clarify
lock accepts None for a skipped question.
* test(clarify): align three checks with the one-answer confirm and the regenerated keys
The desktop Cmd+Enter test now expects a send with one answer (the blank
question locks null), the compressor test expects the single-answer summary
the code gives, and locales/_keys.desktop.json is regenerated for the
removed and added clarify strings.
* test(tui): the one-question card header is singular
elide() already returns short text unchanged, so the eval runner's length
guard was a second copy of the same check. The timeout-diagnostic test
read its log as utf-8-sig although the writer never emits a BOM, and it
interrupted a stub child that never ran.
The subagent timeout diagnostic and the session_search eval harness still
appended a bare "...[truncated]" marker, the imitable wording #121548
replaced everywhere else. Route both through agent.compression_marker.elide
so every elision in the tree mints the same counted, guard-matched marker.
Salvaged from #122392 (only the two call-site hunks; base already ships
the elide helpers the PR re-defined). Refs #121572.
Conflict resolutions and semantic fixups:
- tools/environments/base.py: main's hard-exit kill fence (kill a spawn the
fence missed, deregister from _live_foreground in a finally) wrapped around
pm-clean's output collector.
- pyproject.toml: pm-clean's marker list plus main's new `live` marker.
- hermes_cli/main.py: pm-clean runs startup recovery from hermes_bootstrap, so
the old early-recovery block stays gone; main's interrupted-pull restore
(auto-merged above it) runs right after bootstrap, as on main.
- hermes_cli/update_cmd.py: main's interrupted-pull marker now guards
pm-clean's first tree mutation (release-tag detach, ff-only, or reconcile)
and is cleared once git is done. The marker's target is the ref git actually
moves to (a release tag, not always origin/<branch>), since the restore
compares against it.
- hermes_cli/_early_recovery.py: restore `import subprocess`, which pm-clean
had dropped and main's auto-merged restore needs (NameError on the first
launch after a killed update; test_update_interrupted_pull red -> green).
- apps/desktop/src/i18n/{de,es,fr}.ts: main's new locales carry the full
settings.about block; trim it to `updates` as pm-clean's type and the other
overlays do (tsc: 27 errors -> 0).
- main's new e2e tests: `import yaml` -> hermes_yaml; wake-word import table
names pyopen_wakeword (pm-clean's wake-openwakeword extra); the anthropic
key-leak switch leg needs the SDK, and the api_server two-tenant test needs
aiohttp, both PM runtime extras the test env does not carry.
On a custom branch the updater runs `git merge --no-edit origin/<branch>` inside the marker window.
Its files are the merge of both sides, a blob that is neither pre nor target, so the restore took
them for user edits: it put the upstream-only files back to pre, kept the merged ones and spent the
marker, leaving a mixed tree (a real random-kill of that merge: 104 of 500 trials broken).
- _early_recovery: when pre and target diverge, `git merge-tree --write-tree pre target` gives the
tree the merge was writing; its blobs (and prefixes of them, for a file cut short) count as git's
like the target's. Conflicted paths, and on git < 2.38 every path both sides changed, count as
git's whatever their content. Paths with a newline are hashed one by one (`--stdin-paths` is
newline-delimited). The docstring lists the by-design limits.
- run_agent (`hermes-agent`) and acp_adapter.entry (`hermes-acp`) never import hermes_cli.main, so
they now run the same restore right after hermes_bootstrap (run_agent only when hermes_cli.main is
not loaded, since it is also a library module).
- Tests: the second test kills inside a clean custom-branch merge (merged file, upstream-only file,
a cut-short file, a user edit); the first pins that each console script's entry module imports
no other checkout module before the restore runs. Both red on the previous head.
- evals/update_pipeline/interrupted_pull_ab.sh gains scenario E: a kill inside the custom-branch
merge, then the `hermes-agent` import.
Disposable origin/install, real autostash + _pull_updates, real SIGKILLs and
the real entry point; covers the user-edit cases from the review (kill before
git wrote, failed update + manual merge, torn tree with a user edit) and a
linked-worktree install. One VERDICT line.
Resolved toward the branch: PM provisions uv/python (main's install.ps1 uv-shim
salvage + its test and workflow steps dropped), the shim re-exec stays retired,
package.json carries no electron-builder block (afterExtract identity stamp wired
into electron-builder.config.cjs instead; after-pack.mjs keeps signing only),
Desktop workspace-deps helpers stay retired. Main's scratch-dir bootstrap
(export_scratch_tmp_env) is taken and re-run after profile resolution.
jev_cycles_report.py renders the cycles/freed/floor/end-state table from
jev_cycles.py outputs; README carries the exact commands, thresholds and
cost so the diminishing-returns and stuck/fallback findings can be
reproduced. Fallback records now count the real tool calls.
Closed-book arms scored the summary with its session_search pointer unused
(43% vs 79% on the same banks) and made external compactors look like wins.
Bare policy names stay available as an explicit opt-in floor.
Against summary + one session_search round-trip (78.9% @ 55K) Jev's default
arm (75.5% @ 115K) loses on recall and retains 2.1x the tokens; the earlier
closed-book comparison scored our summary with its recovery pointer unused.
Programmatic tool-result removal as the primary compaction leaves ~2x the
context billed every turn and tightens compaction cadence every cycle (each
one a prompt-cache break); the summary's one-time ~90% reduction keeps a
long conversation cache-warm. The recall gap points at summariser retention
of identifiers in assistant text, not at a cadence change.
jev_cycles.py compacts a lineage with Jev every time the estimate crosses a
threshold and records freed tokens, text floor and fitting stage per cycle.
Six runs show the floor (never-removed user/assistant text) is the binding
limit: freed-per-cycle decays to 0-8% and a 200K-window host is stuck after
0.42M tokens; >~500 tool calls between compactions cannot fit the 32K state.
Scorecard gains a decisive verdict section.
Adds an eval-only budget selector to the Jev arm (rank tool pairs by Jev
keep_result or by recency, keep until a token budget) so Jev's judgment can
be separated from the effect of keeping text verbatim, and records the
3-transcript scorecard: default Jev +32 pts recall at 2.1x retained tokens,
1/9 the compaction cost; at matched budget Jev ranking ties recency.
Adds a Python port of tamaratran/fast-jev-compaction as an eval policy
(`engine: jev`): TypeSafe's Jev decision model scores every tool call and
result over the whole history and stale ones are dropped or truncated, with
no summary and no rewriting of user/assistant text. Transport is OpenRouter's
Decisions API (~typesafe/jev-latest). A state that cannot fit the plugin's
25K-token ceiling is recorded as a fallback (the plugin's own behaviour)
rather than scored.
Every arm now reports what its compaction step cost (calls, tokens, USD —
Jev reports cost directly; summary calls are metered through the
compressor's call_llm binding and priced at OpenRouter list), and the run
writes the harness's own answer/judge token bill so the eval's spend is
visible. The question cache key includes --cap-tokens so different caps on
one transcript no longer share a bank.
Why: measure remaining tokens, compaction cost and recall accuracy of the
"decide, don't summarize" approach against current/lean on real Hermes
lineages before deciding whether any of it belongs in the compressor.
Conflicts resolved toward the branch: PM owns dependency preparation, the
Windows shim re-exec/hand-off path stays retired (main's shim-parent wait,
gateway-resume env token and update_cmd_deps tests dropped), docs describe
the PM update flow. The docker workflow parks install-stamp.json around the
toolchain step instead of deleting it so tests/docker can compare provenance.
Eval probes wrote fixtures, receipts and evidence to hard-coded /tmp paths and
two live A/B tasks literally instructed the model to work in /tmp. They now
derive locations from tempfile.gettempdir() / os.tmpdir() (env overrides kept),
usage examples use relative output names, and the desktop e2e screenshot dirs,
perf scripts and the short-session repro fixture stop naming /tmp. Also adds
the explicit encoding= the windows-footgun check wants in the touched files.
* refactor(desktop): Capabilities is a module at /capabilities
The Capabilities page lived in `app/skills/`, was exported as `SkillsView`
and was routed at `/skills`, although Skills is only one of its four tabs.
The name sent every reader to the wrong place.
- `app/skills/` becomes `app/capabilities/`, with one folder per topic:
`skills/`, `plugins/`, `mcp/`, and `catalog/` for the browser that the
skills and plugins tabs share.
- `SkillsView` becomes `CapabilitiesView`, in the plugin SDK too. The
bundled bots plugin reads the new export name.
- The route, its id and its view become `/capabilities` and
`capabilities`. The keybind action becomes `nav.capabilities` and the
sidebar item id follows, with its i18n keys in every locale.
- The `hermes://open/...` link that the plugin notice fires follows.
No alias is kept for the old route. A remembered `/skills` route no longer
matches a page; the restore path already drops a route it cannot validate
and opens the last session.
No behaviour change otherwise.
* refactor(desktop): each Capabilities tab owns its list, detail and writes
`CapabilitiesView` was one 770-line function. It held the tab routing, the
whole Skills tab and the whole Tools tab, while the Plugins and MCP tabs
already lived in their own files.
- `capabilities/index.tsx` is the page shell: tab selection, the search
header, the profile and connection scope, the refresh hotkey, and one
table that maps a tab to its component (1190 lines down to 230).
- `skills/` and `toolsets/` each hold their tab, detail pane, data hooks and
helpers. `scope-selector.tsx` holds the scope hook and its selector.
`primitives.tsx` holds what two tabs share.
- The shell still fetches the two installed lists, because the tab pills
count them for the tab the user is not on. Query keys are unchanged;
`store/hub-actions.ts` imports the skills key instead of copying it.
- Every tab is keyed on the scope, so a profile or connection switch mounts
a fresh tab. This replaces the epoch counters and manual resets. One small
change follows: a return to a scope seen before selects the first row, not
the row selected last time.
Reconcile plugin declarations and validation through PM's atomic generation publication; preserve external runtimes, target markers, and conflict refusal. Keep one source-update completion owner and port upstream lifecycle changes to the PM desktop/runtime paths.
hermes_cli/AGENTS.md: the update pipeline no longer runs pulled code in the pre-pull
interpreter; the receipt crosses the hand-off; purge/reload/isolate fixes are not to be
reintroduced. Stale comments in update_receipt / update_abort_recovery / update_cmd_maint
that explained the old shape are rewritten to say why the code is the way it is now.
evals/update_pipeline/post_swap_handoff_ab.sh: installs a real clone at a given SHA,
publishes a target commit whose new module-level import (a symbol added to a module the
old updater kept cached) and tagged completion line show which interpreter ran the tail,
runs `hermes update` against a disposable HOME with no npm and --no-gateway-restart, and
prints one VERDICT line. base (6005aa1f): exit 1, "Could not check config version", old
completion line. fix: exit 0, config check runs, "[completion printed by: pulled-code]",
one receipt carrying post_swap_pid.
Use hermes-tools for the probe's managed server configuration and tool-call target so it exercises the endpoint receiving worker overrides. Leave intentional user-defined hermes-mcp fixtures unchanged.
Scope-risk: narrow
Tested: Three complete isolated probe runs on macOS with Codex CLI 0.137.0; related regression selection 202 passed, 2 existing skips; Ruff and repository static checks
Not-tested: Full repository suite, native Linux/Windows, hosted CI, cloud-model turns
Extend the real-wire device fixture with a `multi_issuer` mode whose
protected-resource metadata lists an issuer-mismatching server before the
valid one, and run the production CLI login through it. Red on main
(`Authorization server metadata issuer mismatch`), green with the scan.
Ported from PR #112068.
Catch unexpected delivery exceptions after claim, retain diagnostics and continue
sibling admissions without authorizing replay. Preserve indefinite retention.
Reproduced PermissionError at target traversal after discovery. Native Electron
controlled-fault A/B confirms the healthy sibling settles and renders once.
Extend deferred dispatch's destination pin to ordinary CLI fallback, so
custom-root and active-profile changes cannot redirect a checked target.
Refuse a missing destination before launch and name the target on failure.
Replace the old env-clearing expectation with two behavioral invariants
and retain the native Electron custom-root reproduction.
Adapted from the root-boundary fix and diagnosis in #104066.
Related #104055, #104066.
Co-authored-by: fangliquanflq <fangliquan@qq.com>
Carry the original destination home and delivery ID into deferred drain and
its child, rather than re-resolving a mutable profile/root. Missing destinations
fail closed; supported-owner handoffs remain transferred, not ambiguous failures.
Capture the producer root before the background thread starts, and retain/log
malformed JSON without stopping healthy admissions or the whole cron tick.
Two invariants reproduced failures on the published head. Real Electron root
change and malformed-record cases are red before and green after; nested DM
control remains passing. No automatic retry of claimed or uncertain turns.
