restore_heartbeat_watches entered _profile_scope_for_source for every routed
session on every poll. Each entry hydrated the profile secret scope and rebuilt
the terminal policy, and both re-parsed the profile config.yaml from disk, so N
routed sessions cost 2N YAML parses per poll even though nothing changed.
- Group entries by resolved profile home and enter the scope once per group.
- Add utils.load_yaml_file_readonly (file_signature-keyed cache) and use it in
env_loader._load_secrets_config and terminal_scope.build_profile_terminal_scope,
which were both open()+fast_safe_load per scope entry. Present-but-unparseable
still fails closed: parse errors propagate and are never cached.
Measured on a 3-profile host: one _profile_runtime_scope enter/exit 1.80 ms -> 0.11 ms.
(cherry picked from commit c6b16629bd38799bbf166206c73a6140a9559a61)
`hermes_cli.config.atomic_config_write` is now THE config.yaml writer: it delegates to
`utils.atomic_roundtrip_yaml_save` (ruamel round-trip), which merges the new state onto the
on-disk document so user comments, key order, quoting and blank lines survive every write.
Why: config.yaml is hand-edited and commented, and every writer that re-serialised the parsed
dict through PyYAML (`save_config`, `config set/unset`, migrations, plugin bookkeeping, auth
provider reset, credential scrub, channel strip, backup restore, profile seed, telegram topic
persistence) destroyed those comments — and `save_config` re-appended the stock boilerplate on
top (#92554, #63039, #50698, #109611, #107511, #66752). The round-trip writer existed
(tui_gateway only) but nothing else used it, so each new writer regressed the class.
- save_config / _write_user_config / atomic_config_write -> round-trip merge; the commented
example blocks are appended only when the file is created.
- round-trip merge only reassigns nodes whose value changed (element-wise for lists), so an
untouched scalar/list keeps its inline comments; YAML 1.1-ambiguous strings (off/yes/no...)
are force-quoted at every depth; duplicate keys are tolerated like PyYAML.
- direct PyYAML writers in auth.py, credential_lifecycle.py, profile_channels.py, backup.py,
profiles.py, telegram adapter and tui_gateway/server.py now call atomic_config_write.
shutil.rmtree stops at the first entry it cannot unlink, and Git leaves loose object files read-only on Windows (r--r--r-- trees on POSIX package installs behave the same). Checkpoint clear, MCP catalog uninstall/reinstall and git-installed plugin removal all deleted clones with a bare rmtree, so each aborted mid-tree and left partial state.
Add utils.rmtree_readonly: clear the write bit on the failing entry and its parent, retry that one operation, and keep rmtree semantics for every other failure (both the 3.11 onerror and 3.12+ onexc callback shapes).
Round-2 gate folds: utils.base_url_path sits next to base_url_hostname on
the same scheme-tolerant parser, so a scheme-less override (api.minimax.io)
keeps host and path checks agreeing; a default with no path is an explicit
"nothing to compare" case; the dead `or "chat_completions"` after
determine_api_mode is gone; the test fixture seeds the catalog defaults
unconditionally so a warm models.dev cache cannot change what the contract
compares against, and the stale #53054 test comment states the new contract.
The cherry-picked commit extends the config/profile/MCP/managed-scope/completer/
OAuth/skills-manifest signatures. This commit finishes the class and trims it:
- `file_signature()` lives in `utils.py` next to the other stat/metadata helpers
instead of `hermes_cli.managed_scope` (gateway/ and agent/ callers no longer
reach into the managed-scope module for a generic stat helper).
- `hermes_cli/config_effective.py` was left comparing 2-/4-wide prefixes against
the widened `_RAW_CONFIG_CACHE` / `_load_config_cache_sig` records, so
`load_user_config_effective()` re-parsed on every call (3 parses for 3 calls on
an unchanged file, 1 before); index by the new widths.
- Sibling caches keyed on the same (mtime, size) shape and reading the SAME files
now use the helper: `load_env()` memo, `agent/skill_utils` raw-config and
external-dirs caches, `hermes_cli/model_switch` alias identity, `agent/moa_loop`
preset stamp, `hermes_cli/auth` global auth-store memo.
- Tests trimmed to one invariant each (pinned-mtime replacement invalidates; an
unchanged file still hits), both red on origin/main.
Left alone on purpose: `tools/registry.py`, `tools/skills_tool_dedup.py`,
`gateway/status.py`, `hermes_cli/banner.py`, `hermes_cli/main.py`,
`hermes_cli/session_recovery.py` — those fingerprint source files, PID/lock files
or write to persisted on-disk caches shared across processes, where an inode/ctime
key would churn on every checkout/copy rather than catch a replaced config.
Port from earendil-works/pi#8337 (UTF-8 BOM normalization in text inputs):
sibling sites the merged #81967 BOM sweep missed. json.loads hard-fails on
a leading U+FEFF and every one of these loaders swallows the exception and
silently falls back to defaults — a user who edited mem0.json, honcho.json,
hindsight/config.json, or supermemory.json in Notepad lost their whole
config with no error, and Qwen CLI OAuth creds saved with a BOM raised
qwen_auth_read_failed.
- plugins/memory/{honcho,mem0,hindsight,supermemory}: 13 read sites -> utf-8-sig
- hermes_cli/auth.py: _read_qwen_cli_tokens -> utf-8-sig
- tests: BOM regression tests per loader (sabotage-proven) + plain-UTF-8 guard
One `/model --global` produced four config.yaml shapes. CLI wrote
default/provider/base_url/api_mode and cleared the context pin on a route
change; the gateway rewrote the whole `model:` block (whole-file save_config)
and only set api_mode for `custom`; the TUI wrote three keys and never
touched api_mode, so a switch off an Anthropic-wire endpoint left a stale
`api_mode: anthropic_messages` in config; the dashboard main slot had its own
switched-provider logic, wrote `base_url: ""` and always dropped
context_length. ACP `session/set_model` and `POST /api/model/set` accepted
any model string (parse_model_input + detect_provider_for_model) so a model
no catalog knows, or a provider with no credentials, was handed to the
session / persisted and only failed at inference time.
Canonical: `hermes_cli.model_switch.model_selection_config_updates` (the
shape) + `persist_model_selection(result, config_path=None)` (targeted
per-key `atomic_roundtrip_yaml_update` writes, so sibling
`model_slots`/`model_fallback` keys survive; explicit path for the
multiplexed gateway's profile config) + `apply_model_selection` (same shape
applied to an in-memory `model:` dict for callers that save a whole
document). `atomic_roundtrip_yaml_update(value=None)` now REMOVES the key
instead of writing `key: null`, so per-key and whole-document writers land
the same file. Shape = CLI/gateway semantics: default, provider, base_url
(cleared when the target has none), api_mode (cleared when unresolved),
context_length cleared only when `should_clear_context_pin` says the route
identity changed, inline api_key/api cleared for non-custom targets.
Sites -> canonical:
hermes_cli/cli_model_switch_mixin.py::_persist_global_switch -> deleted; _commit_model_switch calls persist_model_selection
hermes_cli/cli_model_switch_mixin.py::_clear_persisted_context_for_model_switch -> deleted (folded into the shape)
gateway/slash_commands_model.py::_persist_model_switch_to_config -> to_thread forwarder: persist_model_selection(result, ctx.config_path)
tui_gateway/model_switch.py::_persist_model_switch -> deleted; _apply_model_switch calls persist_model_selection
hermes_cli/web_server_config.py::_apply_main_model_assignment -> apply_model_selection(result) (+ explicit custom api_key)
hermes_cli/web_server_config.py::_validated_main_model_selection -> NEW: switch_model(--provider) gate; rejection -> HTTP 400
hermes_cli/web_routers/{models,profiles,config_env}.py main-slot paths -> through _validated_main_model_selection
acp_adapter/server.py::_resolve_model_selection -> deleted; _switch_model calls switch_model (provider:model -> --provider), rejection -> ValueError
Behavior changes: TUI --global now writes/clears model.api_mode and clears a
route-changed context pin; gateway --global no longer rewrites the whole
model block (sibling keys survive) and clears api_mode for every target;
dashboard main slot / profile-create model / custom-endpoint activate now
reject unknown/uncredentialed/unlisted models (HTTP 400) and persist the
resolved base_url/api_mode instead of `base_url: ""`; ACP rejects the same
(ValueError surfaced by the command/protocol handler). Gateway persist runs
on a worker thread against the routed profile's config_path (multiplex-safe).
Cleared keys are removed from config.yaml rather than left as `null`. ACP
still never persists.
Kept `_normalize_main_model_assignment`: switch_model rejects a vendor name
posing as a provider (`moonshotai` -> "Unknown provider"), so the
vendor->aggregator repair is not a duplicate; E2E verified both branches.
No config migration: readers already coalesce `base_url: ""` to absent
(`_config_base_url_for_provider`) and gate api_mode on provider match
(`_provider_supports_explicit_api_mode`), so no stale-shape reader bug.
Tests: tests/hermes_cli/test_model_persist_one_shape.py (four surfaces land
one block; same-route re-pick keeps the pin), tests/acp_adapter/
test_acp_dashboard_model_switch_validation.py (rejection + explicit
provider prefix). Replaces test_acp_set_model_explicit_provider.py and the
two TUI-only persist tests; tests that intercepted the old per-surface seams
(`cli.save_config_value`, `load_config_readonly`, `tui_gateway.server.
_persist_model_switch`) now intercept the canonical seam. Each fix
sabotage-verified red.
Six of eight memory providers spawned plain threading.Thread for prefetch/sync/
writer work. A plain thread starts with an EMPTY contextvars.Context, so under
multiplex profiles the worker resolved the DEFAULT profile's HERMES_HOME (and
fails closed on scoped secrets). honcho and hindsight had each noticed and
written their own copy_context() wrapper; core had a third in memory_manager.
One canonical pair now lives on the ABC module every provider already imports:
agent/memory_provider.py::ctx_bound / spawn_context_thread. memory_manager,
honcho, hindsight, mem0, retaindb, byterover, supermemory and openviking all use
it; the honcho and hindsight wrappers and memory_manager._ctx_bound are deleted.
Five "json.loads(path.read_text()) or {}" readers (mem0._read_mem0_json,
honcho client/oauth/cli _read_config, hindsight save_config/_load_config) fold
into utils.read_json_or_empty, the read half of every read-merge-atomic_json_write
sidecar store.
holographic.save_config was the only config.yaml writer in the tree that
bypassed hermes_cli.config.save_config: raw open("w") + yaml.dump with no config
lock, no managed-mode refusal, no atomic replace, and a swallowed exception. It
now calls save_config(..., merge_existing=True). Behavior change: a managed
install refuses the write (previously silently rewrote config.yaml); other
sections are deep-merged instead of round-tripped through a raw dump.
openviking._hermes_home_path guarded an impossible ImportError of
hermes_constants (the module already imports agent.*) with a ~/.hermes fallback
that is wrong on Windows and under profile overrides; it is replaced by
get_hermes_home() directly.
Tests: tests/plugins/memory/test_provider_threads_inherit_profile.py drives each
provider's real spawn path with a fake backend and asserts the thread sees the
spawner's HERMES_HOME override (sabotage: retaindb back on threading.Thread ->
red). tests/plugins/memory/test_holographic_save_config.py pins merge-with-
existing-sections and managed-mode refusal (sabotage: raw yaml.dump -> red).
Every hand-rolled writer this PR folded into utils._atomic_write created a
NEW file with write_text()/open("w"), i.e. at 0o666 masked by the umask
(0644 under 022). The canonical helper publishes through mkstemp, whose
temp is 0600, and with no explicit mode and no existing target to copy bits
from it left that 0600 in place - so debug, model_catalog, profiles,
breadcrumbs, worktree_ops, web_result_cache, plugin_compat, write_approval,
rich_sent_store, active_sessions and the google_meet state files were
silently tightened to owner-only, the volume-mount hazard
_restore_file_metadata's own docstring warns about. Undeclared in the PR.
Fix at the canonical: when mode is None and the target does not exist,
apply default_new_file_mode() (0o666 masked by the umask, read via the
umask two-call trick with a transient 0o077 so a racing thread can only get
a tighter file). The helper is hermes_cli/backup._default_new_file_mode
moved into utils and reused. Secret writers (mode=0o600) are 0600 before,
during and after as before; an existing target keeps its bits; on non-POSIX
the helper returns None so nothing is chmod'd.
The canonical writer defaults to ensure_ascii=False, but ~10 of the sites
repointed onto it (terminal breadcrumbs, shell-hook allowlist, active
sessions, debug pending, model-catalog cache, the credential writers)
previously used json's ensure_ascii=True default. A surrogate-escaped str
(os.fsdecode of a non-UTF-8 cwd/argv) that json used to persist as \udcff
now made the utf-8 text handle raise UnicodeEncodeError - a ValueError that
the callers' `except OSError` never catches, so breadcrumbs silently stopped
writing and the other sites leaked a new error type.
Fix at the canonical: serialize to a str first (so nothing lands in the
temp file on failure), and on UnicodeEncodeError retry the dump with
ensure_ascii=True. That escape round-trips - json.loads returns the same
str with the lone surrogate - whereas encoding with surrogateescape emits a
raw 0xFF byte the reader's utf-8 decode rejects. The happy path is
unchanged: normal content keeps its raw UTF-8 bytes on disk.
Ten hand-rolled "write a token file safely" routines each carried a
different subset of {0600-on-create, fsync, atomic_replace, parent-0700
guard, BaseException cleanup}. Two of them (iron_proxy state files,
the exchanged-JWT store) still opened the temp file at process umask
and chmod'ed afterwards - the exact TOCTOU window the others document
as fixed. None of the bare-os.replace copies got atomic_replace's
Windows-contention retry or EXDEV fallback.
utils gains fsync_dir= (absorbs auth.py's dir fsync), atomic_write_bytes
(vault blob) and mode= on atomic_write_text; the ten sites become 1-3
line callers. mkstemp creates the temp file O_EXCL at 0600 regardless of
umask, so the payload is never umask-readable.
Behavior change: iron_proxy proxy.yaml/mappings.json and the exchanged-JWT
store are now 0600 from creation and fsync'd; every credential write goes
through atomic_replace (symlink-preserving, Windows retry, EXDEV copy).
auth_nous shared store now uses atomic_replace too (it forced os.replace
with no recorded reason). secret_sources cache parent-0700 goes through
the guarded secure_parent_dir instead of an unguarded chmod.
`_append_entry` moved to `utils.atomic_json_write`, which dumps with
`ensure_ascii=False` through a utf-8 text handle. An argv token holding
surrogate-escaped bytes (a non-UTF-8 project path via os.fsdecode) makes
json.dump raise UnicodeEncodeError — a ValueError, so the `except OSError`
does not catch it and callers silently lose their registration.
Expose `ensure_ascii` on `atomic_json_write` (default unchanged) and pass
True at the ledger call site, restoring the previous json.dumps behaviour
while keeping mode=0o600. One round-trip test, red on base.
Follow-up to #109156.
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
Builds on webtecnica's escape-aware _split_key_path (#84152, cherry-picked
with authorship preserved; earliest fix in the family was RelaxJonh's #80253
greedy-match approach — both behaviors now ship together):
- _greedy_literal_match: when navigating an EXISTING mapping, prefer an
existing literal key equal to the dot-join of the next N path segments
(longest match wins). Dotted model IDs are the norm, so the common
unescaped command (config set providers.p.models.grok-4.6.supports_vision
true) now hits the real key across set/get/unset instead of creating a
phantom sibling. Plain dotted paths with no dotted-key collision split
exactly as before.
- _phantom_sibling + ValueError in _set_nested: refuse to CREATE a new
intermediate mapping that would shadow an existing dotted literal sibling
(Soju06's fail-loudly suggestion on #84064); set_config_value surfaces it
as a clean CLI error with the escaped spelling to use.
- utils.py::atomic_roundtrip_yaml_update (the second split site, #91607 —
/model + TUI persistence) now uses the same escape-aware split + greedy
literal matching.
- CFG-04 empty-segment guard now splits escape-aware so escaped keys are
not misclassified.
- Tests for every repro shape in the family: #84064 provider model keys,
#80006 Matrix room IDs, #91095 dotted models under custom_providers list
index (incl. escaped creation-when-absent), #91607 model_overrides via
atomic_roundtrip_yaml_update, #99124 dotted leaf keys; plus
backward-compat coverage. Also fixed the carrier's one stale assertion
(structured-value coercion landed on main after #84152 branched) and
removed its dead _MCP_SECRETS_CONFIG fixture flagged in review.
- Docs: 'Dots inside key names' section in website/docs/reference/cli-commands.md.
Fixes#84064, fixes#80006, fixes#91095, fixes#91607, fixes#99124
Salvaged from PR #84199 by @RickyYii. DirectAlias gains api_key/key_env; the direct-alias override re-resolves credentials against the alias endpoint (host-gated, #28660) and reuses the pre-alias key only on an origin match; oneshot -m <alias> passes the alias key as explicit_api_key; direct-alias branch gains the OLLAMA_API_KEY host gate. Fixes#83612.
os.replace onto a file that any other handle holds open is denied on
Windows — CPython opens files without FILE_SHARE_DELETE. atomic_replace
only fell back for EXDEV/EBUSY, so the exception propagated and, because
most callers swallow it, the write was silently dropped. gateway_state.json
loses status updates at every turn boundary while status readers poll it;
auth.json surfaces the same race to the user as 'agent init failed'.
Classify winerror 5/32/33 as contention candidates and retry the rename
with jittered backoff; a retry that wins keeps the write fully atomic.
Only a handle that outlives the budget falls through to a rewrite.
Measured on Windows 11 build 26200 / CPython 3.11: a held *target* handle
reports winerror 5, not 32 — 32 is what a held *source* reports. Keying
recovery on 32 alone misses every real occurrence of this bug.
The codes are ambiguous (a genuine ACL denial is also 5) and cannot be
told apart up front: os.replace needs delete-child rights on the parent
directory, so probing the target with os.access reports a directory-level
denial as writable. Rather than guess, both cases take the same bounded
path and a genuine denial is re-raised unchanged with its pending temp
file intact.
The last-resort rewrite writes through the existing file instead of
shutil.copyfile: a copy truncates the target to zero first, and a
concurrent reader can observe an empty auth.json mid-write. Writing
through the target also preserves its ACL, which os.replace does not.
Co-authored-by: LewfKrad <lEWFkRAD@users.noreply.github.com>
Co-authored-by: ruochu88s <ruochu88s@users.noreply.github.com>
Co-authored-by: lost9999 <lost9999@users.noreply.github.com>
Co-authored-by: guanla-zz <guanla-zz@users.noreply.github.com>
Co-authored-by: zapabob <zapabob@users.noreply.github.com>
tui_gateway/server.py:_save_cfg called yaml.safe_dump on a deep-loaded
config dict, which reordered top-level keys alphabetically, stripped
every user-edited comment, and re-escaped non-ASCII (kaomoji/Chinese)
personality prompts to \uXXXX. Every TUI setting change - /personality,
/reasoning, /details_mode, /skin, /prompt - rewrote the file top to
bottom.
Changes:
* Add atomic_roundtrip_yaml_save(path, new_state) in utils.py - a
comment-, ordering-, and unicode-preserving full-state replacement
for yaml.safe_dump(cfg, f). Uses ruamel round-trip mode like the
existing atomic_roundtrip_yaml_update, but accepts the whole cfg
dict so callers that mutate multiple keys before saving (the
_save_cfg pattern) don't have to be rewritten. Recurses into nested
dicts, deletes keys missing from new_state (preserves the
cfg.pop()-then-save semantic), and overwrites lists/scalars
wholesale.
* Fail closed on an unreadable existing config.yaml the same way
hermes_cli.config.atomic_config_write does, via a lazy import of
require_readable_config_before_write (avoids a module-level circular
import, since hermes_cli.config itself imports from utils). Also
preserves both file mode and owner across the write, matching the
existing atomic_roundtrip_yaml_update contract.
* Force-quote any new string value that YAML 1.1 would misparse as a
bool/null (yes/no/on/off/true/false/null/~). ruamel's round-trip
dumper resolves against the YAML 1.2 core schema and emits these
unquoted, but PyYAML-based readers elsewhere in the codebase parse
under YAML 1.1 rules - so an unquoted `approvals.mode: off` would
silently round-trip back as the boolean False.
* tui_gateway/server.py:_save_cfg now delegates to
atomic_roundtrip_yaml_save. Drop-in - all call sites (/personality,
/reasoning, /details_mode, /prompt, etc.) inherit comment
preservation and the fail-closed contract.
Tests:
* tests/test_utils_atomic_roundtrip_yaml_save.py - unit tests covering
create-from-empty, top-level key-order preservation, comment
preservation, readable Unicode, append-new-keys, delete-missing-keys,
scalar/list overwrite, nested-dict recursion, refusal on an
unreadable existing config, and owner preservation.
* tests/test_atomic_replace_symlinks.py - owner-preservation regression
test mirroring the existing atomic_roundtrip_yaml_update coverage.
* tests/test_tui_gateway_server.py - 4 new tests pinning _save_cfg
comment preservation, top-level key-order preservation, and
unicode-readability under unrelated writes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Post-review fixes on the preserve_mode/create_mode follow-up:
- create_mode is now applied ONLY when the target does not exist, on
both atomic_write_text and atomic_yaml_write. Previously
atomic_write_text(path, s, create_mode=X) without preserve_mode would
silently chmod an EXISTING file to X (docstring/code mismatch, latent
trap -- no caller relied on it), and a stat failure on an existing
file could fall through to create_mode instead of leaving the mode
alone.
- atomic_yaml_write now fchmods the temp fd BEFORE the replace when a
mode is known, matching atomic_write_text: a freshly created
distribution.yaml no longer transits through mkstemp's 0600 (a crash
between replace and chmod could previously leave it 0600 forever).
The post-replace _restore_file_mode stays as the Windows path.
- fchmod moved inside the fdopen context in atomic_write_text, so a
raising fchmod can no longer leak the fd.
Tests: create_mode-never-rewrites-existing guard (mutation-checked) and
a monkeypatch.delattr(os, 'fchmod') test covering the Windows
post-replace branch that the win32 module skip left uncovered.
Follow-up to the salvaged #79323 commits. The three hand-rolled
stat -> atomic_write_text -> chmod blocks (xai migration, uninstaller
shell-rc rewrite, dashboard SOUL.md editor) collapse into an opt-in
preserve_mode=True kwarg on utils.atomic_write_text, plus create_mode=
on both atomic_write_text and atomic_yaml_write for first-create paths
(SOUL.md first save, write_manifest's allowlist create path).
Beyond deduplication this closes two gaps the hand-rolled copies had:
- Owner preservation: the old in-place writes kept the inode, so file
ownership survived root-run rewrites for free. atomic_write_text
swaps in a new inode owned by the writing user, and the hand-rolled
blocks restored only the mode -- a root-run 'hermes migrate xai' or
sudo uninstall on a user-owned Docker/NAS volume would flip
config.yaml / ~/.zshrc ownership to root. preserve_mode now routes
through the same _preserve_file_owner/_restore_file_owner helpers
atomic_yaml_write and atomic_json_write already use.
- chmod-after-replace window: the mode is applied to the temp fd via
fchmod BEFORE the replace (mirroring atomic_json_write's mode= param),
so the target never transits through mkstemp's 0600.
Also removes write_manifest's caller-side existed/chmod block (and its
small TOCTOU) in favor of atomic_yaml_write(create_mode=0o644), and
corrects the SOUL.md mode comment (the default profile's runtime seeder
does run it through _secure_file; named profiles do not).
preserve_mode defaults to False so the existing callers (memory store,
skill manager, cron, agent importer) keep their current semantics.
New tests in tests/test_atomic_write_text_metadata.py cover mode
preservation, owner restore through symlinks, fchmod-before-replace,
create_mode on both writers, and no-behavior-change without opt-in;
all mutation-checked.
Deduplicates the mkstemp→fsync→atomic_replace pattern that existed in
three places: agent_import.py (added by #72983), MemoryStore._write_file,
and skill_manager_tool._atomic_write_text. All three now call a single
utils.atomic_write_text helper.
Also wraps the atomic_write_text call in _merge_memory_entries with
try/except OSError so a write failure records a per-item error instead
of propagating uncaught and aborting the entire import with no record.
Follow-up to #72983.
Follow-up hardening on top of the C14 cherry-picks (#57860/#44026/#66742/#60009):
- Slack file downloads (_download_slack_file/_download_slack_file_bytes)
now require an https URL on a Slack CDN host (files.slack.com,
*.slack.com Enterprise Grid, *.slack-files.com legacy shares) before
attaching the bot token. url_private/url_private_download only ever
point at the Slack CDN, so a forged file object from a malicious
workspace app or compromised event stream pointing the Bearer-token
download at an arbitrary PUBLIC host (token exfiltration) is now
refused — a hole #44026's generic private-IP SSRF check alone could
not close.
- The same two download paths now use create_ssrf_safe_async_client
(from #57860) so the preflight-validated hostname is resolved once,
validated, and dialed by IP — closing the DNS-rebinding TOCTOU window
for the token-bearing inbound fetches as well.
- #60009's slack_tokens.json permission warning is generalized into
utils.warn_if_credential_file_broadly_readable() (POSIX-only,
fail-quiet) and wired into the other read path with the same gap:
google_chat's load_user_credentials(). google_chat already writes
0o600 via _write_private_json; the read-time warning covers
hand-provisioned/legacy files. Nothing in-repo writes
slack_tokens.json (user/OAuth-provisioned), so there is no write
path to chmod for Slack.
Security tests both directions: non-CDN/lookalike/http URLs and
connect-time DNS rebinds are blocked before any TCP connect; real
files.slack.com, Enterprise Grid, and slack-files.com URLs still reach
the network layer; 0o600 files stay silent while 0o644/0o640 warn with
a chmod hint. A/B: all 10 new download-guard tests fail with the
hardening reverted and pass with it applied.
The startup config/manifest reads used PyYAML's pure-Python SafeLoader,
which is ~8x slower than the libyaml-backed CSafeLoader C extension.
config.yaml is parsed several times during launch (cli config, raw
config, early interface/redaction bridge, logging config) and every
plugin manifest is parsed once — all on the slow path.
Add utils.fast_safe_load (CSafeLoader-preferring, pure-Python fallback,
true drop-in for safe_load) and route the hot startup parse sites
through it: hermes_cli/config.py (config + manifest reads),
hermes_cli/plugins.py (manifest parse), env_loader, cli.load_cli_config,
hermes_logging, and the two pre-config early YAML bridges in main.py.
Behavior is identical (same restricted safe tag set); only speed changes.
safe_load calls on the startup path drop from ~79 to ~0, cutting the
YAML parse cost from ~0.9s to ~0.15s under profiling.
Adds tests/test_fast_safe_load.py asserting equivalence with safe_load
across input shapes, empty-doc falsiness, C-loader preference, and that
python/object tags are still rejected (safe, not full loader).
atomic_yaml_write used default yaml.dump which emits indentless
sequences (list items at column 0), while atomic_roundtrip_yaml_update
(ruamel.yaml) emits 2-space-indented sequences. Cross-path writes to
the same config.yaml toggled indentation on every save, eventually
producing a mixed-indent file that js-yaml rejects with 'bad indentation
of a mapping entry', silently dropping custom_providers and breaking
model switching.
Add IndentDumper SafeDumper subclass that forces indentless=False,
route atomic_yaml_write through it. Route tui_gateway._save_cfg and
the Telegram adapter's config writer through atomic_yaml_write so all
paths emit the same 2-indent layout.
Salvaged from #32034 by @xxxigm. Adapted to current main which already
has allow_unicode=True (from #51356) but was missing IndentDumper.
Closes#31999
atomic_yaml_write (and two sibling config writers) called yaml.dump
without allow_unicode=True. The default personalities shipped in cli.py
contain emoji/kaomoji, so PyYAML escaped astral-plane chars as 8-digit
\\UXXXXXXXX sequences inside multi-line double-quoted strings wrapped
with \\ line-continuations. Stricter/non-PyYAML parsers, editors, and
hand-edits break that structure into unclosed quotes, failing the whole
config parse -> silent fallback to defaults -> custom_providers lost.
Add allow_unicode=True to the canonical writer plus tui_gateway/server.py
and the telegram adapter's atomic config write so config is written as
readable UTF-8 with no escape/fold artifacts.
Fixes#51356
Mirrors the existing env_int() helper: returns the default when the
variable is unset or non-numeric instead of raising ValueError. Used by
the follow-up commit to guard malformed float env vars across the gateway.
Salvaged from #48735 (@annguyenNous). The PR's api_server.py change is
now redundant — main guards HERMES_MAX_ITERATIONS via
_current_max_iterations().
Third-party OpenAI-compatible endpoints (self-hosted gateways, OpenRouter,
Azure proxies) fronting gpt-4o / gpt-4.1 / gpt-5+ / o1-o4 models silently
received max_tokens and 400'd with unsupported_parameter, because the three
kwarg-selection sites only checked base_url_hostname(...) == "api.openai.com"
and fell through to max_tokens on every other host. The constraint is
enforced server-side by the model family, not by the URL, so name-based
detection is required as a fallback.
Changes:
- utils.py: new shared helper model_forces_max_completion_tokens(model) that
prefix-matches gpt-4o, gpt-4.1, gpt-5, o1, o3, o4 families on normalized
(lowercased, vendor-prefix-stripped) names.
- run_agent.py: _max_tokens_param ORs the helper into the URL check.
- agent/auxiliary_client.py:
- auxiliary_max_tokens_param gains an optional keyword-only model arg.
- _build_call_kwargs inline branch applies the same check for both
provider == "custom" and non-custom paths.
Tests:
- tests/test_model_forces_max_completion_tokens.py: 31 new cases covering
positive families, negatives (classic gpt-4, claude, llama, mistral, qwen,
deepseek), vendor prefixes, case-insensitivity, whitespace, None/empty,
and substring-not-prefix guards.
- tests/run_agent/test_run_agent.py::TestMaxTokensParam: 5 new model-based
cases (custom + gpt-5.4, openrouter + gpt-4o-mini, custom + o1-preview,
classic gpt-4-turbo keeps max_tokens, llama3 keeps max_tokens).
- tests/agent/test_auxiliary_client.py::TestAuxiliaryMaxTokensParam: new
class, 7 tests covering the URL x model matrix.
os.fchmod is Unix-only; the Windows os module has no fchmod (only
chmod). Passing mode= (e.g. 0o600 when saving the Hindsight config
during `hermes memory setup`) crashed on Windows with:
AttributeError: module 'os' has no attribute 'fchmod'
Guard the fchmod fast-path with hasattr(os, "fchmod"). Skipping it on
Windows is safe: mkstemp already creates the temp file as 0o600, and
the existing post-replace os.chmod(real_path, mode) — already wrapped
in try/except — applies the final mode durably (as far as Windows
honors it).
Adds regression tests: one simulating a Windows os module without
fchmod (must not raise), and one asserting the durable 0o600 mode on
POSIX.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Self-hosted Honcho setup had four sharp edges:
- local/cloud URLs ending in /vN double-prefixed by the SDK (/v3/v3/... 404)
- authenticated local servers had no setup prompt for a JWT/bearer token
- profile-derived host keys could be dot-containing workspace IDs Honcho rejects
- memory-provider config files with API keys written world-readable per umask
This keeps existing behavior but makes those paths safer:
- strip a trailing /vN version segment from any configured baseUrl before SDK
init (the SDK's route builders always prepend their own version prefix);
auth-skipping stays loopback-only
- add an optional local JWT/bearer prompt in honcho setup, stored under
hosts.<host>.apiKey
- derive new profile host keys with underscores, still reading legacy
hermes.<profile> blocks
- write memory-provider config files atomically with 0600 via a shared
utils.atomic_json_write(mode=) arg (honcho/hindsight/mem0/supermemory)
- skip honcho.json parsing in gateway cache-busting unless Honcho is the active
memory provider; memoize by honcho.json mtime when active
- bust the gateway agent cache on memory.provider change
- add a hermes memory setup <provider> one-liner so fresh installs can configure
a named provider without the picker (the per-provider hermes <provider>
subcommand only registers once that provider is active)
Closes#20688, #29885, #26459, #30246, #33382, #32244.
Co-authored-by: BROCCOLO1D
Extract the islink/realpath guard from the 16743 fix into a single
atomic_replace() helper in utils.py, then migrate every os.replace()
call site in the codebase to use it.
The original PR #16777 correctly identified and fixed the bug, but
only patched 9 of ~24 call sites. The same bug class (managed
deployments that symlink state files silently losing the link on
every write) still existed at auth.json, sessions file, gateway
config, env_loader, webhook subscriptions, debug store, model
catalog, pairing, google OAuth, nous rate guard, and more.
Rather than add another 10+ copies of the same three-line guard,
consolidate into atomic_replace(tmp, target) which:
- resolves symlinks via os.path.realpath before os.replace
- returns the resolved real path so callers can re-apply permissions
- is a drop-in replacement for os.replace at the use sites
Changes:
- utils.py: new atomic_replace() helper + atomic_json_write /
atomic_yaml_write now call it instead of inlining the guard
- 16 files: all os.replace() call sites migrated to atomic_replace()
- agent/{google_oauth, nous_rate_guard, shell_hooks}.py
- cron/jobs.py
- gateway/{pairing, session, platforms/telegram}.py
- hermes_cli/{auth, config, debug, env_loader, model_catalog, webhook}.py
- tools/{memory_tool, skill_manager_tool, skills_sync}.py
Tests: tests/test_atomic_replace_symlinks.py pins the invariant for
atomic_replace + atomic_json_write + atomic_yaml_write, covers plain
files, first-time creates, broken symlinks, and permission preservation.
Refs #16743
Builds on #16777 by @vominh1919.
os.replace(tmp, path) replaces the symlink itself with a regular file,
breaking users who symlink config.yaml, SOUL.md, or .env from ~/.hermes/
to a dotfiles repo or managed profile package.
Fix: resolve symlinks via os.path.realpath() before os.replace(), so the
real file is overwritten in-place while the symlink survives.
Fixed in 7 files covering all os.replace call sites:
- utils.py (atomic_json_write, atomic_yaml_write — fixes save_config)
- hermes_cli/config.py (env sanitizer, save_env_value, remove_env_value)
- tools/skill_manager_tool.py (_atomic_write_text — SOUL.md writes)
- tools/memory_tool.py (memory file writes)
- tools/skills_sync.py (manifest writes)
- cron/jobs.py (job state + output file writes)
- agent/shell_hooks.py (hook file writes)
FixesNousResearch/hermes-agent#16743
Aslaaen's fix in the original PR covered _detect_api_mode_for_url and the
two openai/xai sites in run_agent.py. This finishes the sweep: the same
substring-match false-positive class (e.g. https://api.openai.com.evil/v1,
https://proxy/api.openai.com/v1, https://api.anthropic.com.example/v1)
existed in eight more call sites, and the hostname helper was duplicated
in two modules.
- utils: add shared base_url_hostname() (single source of truth).
- hermes_cli/runtime_provider, run_agent: drop local duplicates, import
from utils. Reuse the cached AIAgent._base_url_hostname attribute
everywhere it's already populated.
- agent/auxiliary_client: switch codex-wrap auto-detect, max_completion_tokens
gate (auxiliary_max_tokens_param), and custom-endpoint max_tokens kwarg
selection to hostname equality.
- run_agent: native-anthropic check in the Claude-style model branch
and in the AIAgent init provider-auto-detect branch.
- agent/model_metadata: Anthropic /v1/models context-length lookup.
- hermes_cli/providers.determine_api_mode: anthropic / openai URL
heuristics for custom/unknown providers (the /anthropic path-suffix
convention for third-party gateways is preserved).
- tools/delegate_tool: anthropic detection for delegated subagent
runtimes.
- hermes_cli/setup, hermes_cli/tools_config: setup-wizard vision-endpoint
native-OpenAI detection (paired with deduping the repeated check into
a single is_native_openai boolean per branch).
Tests:
- tests/test_base_url_hostname.py covers the helper directly
(path-containing-host, host-suffix, trailing dot, port, case).
- tests/hermes_cli/test_determine_api_mode_hostname.py adds the same
regression class for determine_api_mode, plus a test that the
/anthropic third-party gateway convention still wins.
Also: add asslaenn5@gmail.com → Aslaaen to scripts/release.py AUTHOR_MAP.
atomic_yaml_write() and atomic_json_write() used tempfile.mkstemp()
which creates files with 0o600 (owner-only). After os.replace(), the
original file's permissions were destroyed. Combined with _secure_file()
forcing 0o600, this broke Docker/NAS setups where volume-mounted config
files need broader permissions (e.g. 0o666).
Changes:
- atomic_yaml_write/atomic_json_write: capture original permissions
before write, restore after os.replace()
- _secure_file: skip permission tightening in container environments
(detected via /.dockerenv, /proc/1/cgroup, or HERMES_SKIP_CHMOD env)
- save_env_value: preserve original .env permissions, remove redundant
third os.chmod call
- remove_env_value: same permission preservation
On desktop installs, _secure_file() still tightens to 0o600 as before.
In containers, the user's original permissions are respected.
Reported by Cedric Weber (Docker/Portainer on NAS).
Remove read_json_file, read_jsonl, append_jsonl, env_str, env_lower —
all added in #7917 but never imported anywhere in the codebase. Also
remove unused List and Optional typing imports.
env_int, env_bool, and the other helpers that have real consumers are
kept.
- add regression coverage for BaseException cleanup in atomic_json_write
- add dedicated atomic_yaml_write tests, including interrupt cleanup
- document why BaseException is intentional in both helpers
