A V4A `*** Add File:` operation is meant to create a new file. The apply
path called `write_file` unconditionally and `_validate_operations` had no
pre-check for ADD, so an Add targeting a path that already existed
overwrote the file with only the patch's `+` lines, returned success, and
emitted a `--- /dev/null` diff that hid what was lost. Models frequently
confuse Add with Update, so this destroyed existing file contents with no
error. The MOVE path already guards its destination against clobbering;
ADD now follows the same rule.
Makes a V4A `Add File` operation fail when its target already exists,
instead of silently overwriting the existing file. Validation now rejects
the operation before any write happens, so the two-phase
validate-then-apply contract ("no files were modified" on a validation
failure) holds for ADD as it already does for UPDATE/MOVE/DELETE. A
matching re-check in the apply phase closes the validate-to-apply race.
N/A
- [x] 🐛 Bug fix (non-breaking change that fixes an issue)
- `tools/patch_parser.py`: add an ADD branch in `_validate_operations`
that errors when `read_file_raw` finds an existing file, and a
defensive existence re-check in `_apply_add` before `write_file`,
mirroring the existing MOVE destination guard.
- `tests/tools/test_patch_parser.py`: add a test asserting an Add onto an
existing path fails validation and leaves the original bytes unwritten;
add `read_file_raw` to three ADD-path LSP fakes so they match the real
`file_ops` interface now exercised on ADD.
1. Build a V4A patch with `*** Add File: <path>` where `<path>` already
exists on disk.
2. Apply it via `apply_v4a_operations`. Before this change the file is
overwritten with the patch's `+` lines and the result is success;
after, the result is a validation failure and the file is untouched.
3. Run `scripts/run_tests.sh tests/tools/test_patch_parser.py` —
`TestApplyOperations::test_add_onto_existing_file_fails_and_preserves_contents`
covers the regression.
- [x] I've read the [Contributing Guide](https://github.com/NousResearch/hermes-agent/blob/main/CONTRIBUTING.md)
- [x] My commit messages follow [Conventional Commits](https://www.conventionalcommits.org/) (`fix(scope):`, `feat(scope):`, etc.)
- [x] I searched for [existing PRs](https://github.com/NousResearch/hermes-agent/pulls) to make sure this isn't a duplicate
- [x] My PR contains **only** changes related to this fix/feature (no unrelated commits)
- [x] I've run `pytest tests/ -q` and all tests pass
- [x] I've added tests for my changes (required for bug fixes, strongly encouraged for features)
- [x] I've tested on my platform: macOS 15 (Darwin 25.5.0)
- [x] I've updated relevant documentation (README, `docs/`, docstrings) — or N/A
- [x] I've updated `cli-config.yaml.example` if I added/changed config keys — or N/A
- [x] I've updated `CONTRIBUTING.md` or `AGENTS.md` if I changed architecture or workflows — or N/A
- [x] I've considered cross-platform impact (Windows, macOS) per the [compatibility guide](https://github.com/NousResearch/hermes-agent/blob/main/CONTRIBUTING.md#cross-platform-compatibility) — or N/A
- [x] I've updated tool descriptions/schemas if I changed tool behavior — or N/A
Sweep of `ruff check . --select F821 --target-version py311`: 2,234 hits. 2,201 are left
alone on purpose: tui_gateway (2,169; bind_module rebinds bodies onto server.py globals,
all names verified to resolve there), the Feishu adapter (27; globals().update() SDK
binding) and the godmode script (5; dead standalone script). The other 33 were all
genuine defects. No lint config change; no TYPE_CHECKING escape hatches — every
annotation names a real, imported type; ty on the touched files: 0 new diagnostics.
- gateway/slash_commands.py: HISTORY_UNREADABLE never imported after #102117
→ NameError on the /btw error branch (same one-liner as #102952).
- gateway/platforms/whatsapp_common.py: `-> Path` return annotation with no Path
import (the body uses `_Path`). Never raised at runtime thanks to
`from __future__ import annotations`, but `typing.get_type_hints()` and ty
both fail on it.
- gateway/run.py: ActivityProvenance imported at module level
(agent.session_activity has no gateway deps); stringly annotation and the
lazy in-function import are gone.
- tools/patch_parser.py: PatchResult imported at module level; real return
annotation. The "avoid circular import" lazy import guarded a cycle that
does not exist (file_operations_common never imports patch_parser).
- gateway/platforms/helpers.py: base.py imports helpers at module level, so
MessageEvent cannot be named here; TextBatchAggregator only reads .text and
.source, so it is typed by a BatchableEvent Protocol that MessageEvent
satisfies structurally.
- tools/mcp_tool_sampling.py: mcp_tool imports this module, so MCPServerTask
cannot be named here; ElicitationHandler only reads
owner._pending_call_context, typed by an ElicitationOwner Protocol.
- plugins/platforms/sms/adapter.py: aiohttp is an optional dep ([messaging] extra) →
module-level try/except ImportError binding `aiohttp = web = None`, the pattern the
homeassistant / webhook / whatsapp_cloud adapters already use. Retires three lazy
in-function imports and the `_aiohttp_available()` wrapper; `_handle_webhook` typed
`web.Request -> web.Response`.
- plugins/platforms/teams/summary_writer.py: plain module-level `import httpx` — httpx is a
hard core dependency (pyproject `httpx[socks]==0.28.1`), so the lazy import and the
"imported on every CLI start" docstring premise were both wrong (plugin discovery never
imports this module; it is reached only via the Teams adapter / meeting pipeline).
Tests:
- tests/hermes_cli/test_config.py: a test body orphaned by the wave-1 prune
(6b81590c55) sat inside the class as dead code with self/tmp_path unbound
— header restored, so the v11→12 custom_providers migration is covered.
- tests/tools/test_mcp_tool.py: @staticmethod recursing on `self` in the
win32 branch; call portalocker directly.
- tests/test_background_review_list_shapes.py: main() still ran 3 pruned tests.
- tests/agent/test_cursor_optimizations_parity.py: bench() used names only
imported inside a sibling test.
- GatewayRunner / FeishuAdapter / Dict / Optional: missing imports.
- patch_parser: shared _fail/_written/_unified_diff apply helpers, single PatchResult build,
flattened apply loop, merged inert/no-op hunk skips, overlay _remove, dropped never-set
PatchOperation.content
- self_repo_guard: conditional git-mutation predicates as a dispatch table over one _has_flag,
unified git global-option/alias parse, suppress() for resolve/alias/shlex, folded scope and
heredoc scanners
- shell_heredoc: unit scanner and operator parser as single if/elif ladders, unified quote masking
- read_extract: groupby gap runs, single try for anydoc, islice/suppress xlsx loops, folded
notebook/docx render loops, _zip_xml optional -> empty element
- schema_sanitizer: folded rename/type-array/recovery-strip loops, _dict_nodes/_carry_union_meta
- docstrings/comments compacted by hand (every WHY kept)
Verified: tool-schema dump byte-identical (SCHEMA-OK); golden corpus over pure functions and
every registry schema + edge cases byte-identical vs 113f046 base (GOLDEN-OK); 95 test files
2517 passed (2 pre-existing reds in test_web_tools_config also fail on base).
The apply phase already skips a hunk whose -/+ lines are identical
(patch_parser.py '(search_lines == replace_lines): continue'), but the
validation phase lacked the guard: such a hunk reached
fuzzy_find_and_replace, whose identical-strings error names
old_string/new_string — parameters that don't exist in patch mode — and
failed the whole atomic patch that apply would have accepted. Mirror
the apply-phase skip in validation; regression test drives a mixed
degenerate+live patch end-to-end (short text dodges the
is_already_applied >=8-char rescue).
- write_file: encode content once, share bytes between bytes_written and
the sha256 verification (drops a second full-content encode per write)
- patch_parser: replace the except-TypeError retry around
write_file(pre_content=...) with signature-based feature detection so a
TypeError raised inside a capable implementation propagates instead of
triggering a duplicate write; tests for both duck-typing contracts
- tests: real-ops V4A BOM round-trip + _file_has_bom disk-probe guard
(the teknium1-review regression previously only covered by a fake)
- comment: document dirs_created's long-standing "parent ensured" meaning
Bug 1 (UTF-8 BOM loss on V4A UPDATE):
_file_has_bom() trusted pre_content for BOM detection, but the most
common pre_content provider — read_file_raw() — deliberately strips
BOMs so the agent never sees U+FEFF glyphs. Passing BOM-stripped
content through pre_content caused a false-negative: the method
returned False and write_file() silently removed the marker on rewrite.
Fix: _file_has_bom() now always probes the first 3 bytes on disk
(head -c 3), ignoring pre_content for BOM purposes. pre_content is
still used by two other consumers — line-ending detection and lint/LSP
delta computation — neither of which is affected by BOM stripping.
Bug 2 (backward compatibility):
_apply_update() called write_file(path, content, pre_content=...) as a
keyword argument. Duck-typed file_ops implementations that only
implement the two-argument write_file(path, content) contract would
raise TypeError.
Fix: wrap the call in try/except TypeError, falling back to the
two-argument form when the keyword is not accepted.
Also declare tomli in pyproject.toml (pre-existing conditional import
for pre-3.11 Python, caught by the pre-commit dep scan after staging
file_operations.py).
Tests:
Add TestV4ABomRoundTrip with two cases:
- UPDATE on BOM-bearing file preserves the marker
- UPDATE on plain file does not inject a BOM
Addresses teknium1 review on PR #55661.
write_file currently spawns up to 6 subprocesses per call:
1. mkdir -p (separate call before atomic write)
2. cat (to read pre-content for lint/BOM/line-ending detection)
3. _atomic_write (mktemp + write + mv — the essential one)
4. wc -c (to measure bytes written)
5. _check_lint_delta (post-write lint — also essential)
6. LSP snapshot (also essential)
This PR removes three of them without changing any observable behavior:
1. Fold mkdir -p into _atomic_write shell script (−1 subprocess/write)
The atomic write script already runs a single shell; adding mkdir -p
to it costs zero extra processes.
2. Add optional pre_content parameter to write_file (−1 subprocess/patch)
patch_replace and V4A _apply_update already read the file for fuzzy
matching. Passing that content as pre_content skips the redundant cat
inside write_file. Fully backward-compatible: callers that don't pass
pre_content still read from disk as before.
3. Replace wc -c with len(content.encode('utf-8')) (−1 subprocess/write)
We already have the content in memory; encoding it to get the byte count
is equivalent to wc -c for UTF-8 text.
4. Remove redundant _check_lint loop in apply_v4a_operations (−N subprocesses/V4A)
write_file already runs _check_lint_delta internally. The old code ran a
bare _check_lint(f) loop over all modified files — a re-read + re-lint
without post_content context. Now lint results propagate from write_file
via a four-tuple return, zeroing out the extra subprocesses.
Net effect:
- write_file: 6 → 3 subprocesses per call (new files)
- patch_replace: 6 → 5 subprocesses per call (pre_content skips cat)
- V4A multi-file patches: saves 1 subprocess per modified file
- A typical 4-file V4A patch drops from ~28 to ~16 subprocess calls
The #1 patch failure class in production (state.db mining, 250k-window)
is a re-send of an edit that already landed: 'old_string and new_string
are identical' (299 occurrences) plus a share of hunk-not-found errors
where the new text is already in the file. These errored, sending
models into re-read/re-patch loops.
New tools/fuzzy_match.is_already_applied(content, old, new) — a
conservative check requiring (1) non-trivial new_string (>=8 chars),
(2) EXACT presence of new_string, (3) old_string gone (unless
identical). Wired into three sites:
- patch_replace (replace mode): returns success + no_change: true +
an explicit note instead of the identical-strings / no-match error.
- V4A validation phase: an already-applied hunk validates as a no-op
so multi-hunk patches no longer fail wholesale when one hunk landed
in a prior call.
- V4A apply phase: mirrors the same skip so the two phases agree.
Genuine no-matches (new text absent) and half-applied renames (old
text still present) keep their error behavior — covered by tests.
Two V4A parse/validate bugs found in a core-tools audit, reproduced live:
1. CRLF patch body injected stray carriage returns. parse_v4a_patch split
on '\n' only, so a CRLF-encoded patch kept '\r' inside every HunkLine
and wrote mixed line endings into an LF file; the anchored Begin/End
markers could also fail to match because of the trailing '\r'. Strip a
trailing '\r' from each line at split time.
2. Move-then-Update of the same file was rejected. _validate_operations read
the UPDATE target from disk before the MOVE ran, so 'Move a->b' + 'Update
b' failed validation with 'b: file not found'. Added a small pending-move
overlay so UPDATE/DELETE/MOVE reads during validation see prior ops'
effects (moved-in destinations resolve, moved-away sources read as gone),
while a genuine 'destination already exists' conflict is still caught.
Both verified with sabotage-checked regression tests. 113 patch/fuzzy/file
tests pass.
The boundary scan in parse_v4a_patch used substring matching, so a
content line mentioning "*** End Patch" (docs about the patch format,
nested patch text) truncated the patch, and "*** Begin Patch" in
content reset the start boundary — silently dropping already-parsed
operations while reporting success. Match only whole-line markers at
column 0, preserving the no-space "***Begin Patch" tolerance.
* fix(lint): skip per-file shell linter when LSP will handle the file
`_check_lint` ran `npx tsc --noEmit FILE.ts` after every `.ts`/`.tsx`
edit. `tsc` ignores `tsconfig.json` when given an explicit file argument
(documented quirk) and defaults to no-lib / ES5, so every ES2015+ stdlib
reference reports as missing:
- `Cannot find global value 'Promise'`
- `Cannot find name 'Map' / 'Set' / 'ReadonlySet' / 'Iterable'`
- `Property 'isFinite' does not exist on type 'NumberConstructor'`
- `Module 'phaser' can only be default-imported using esModuleInterop`
- `import.meta is only allowed when --module is es2020+`
On real TypeScript projects this floods the `lint` field on
WriteResult / PatchResult with up to 25K tokens of false positives
per edit. The delta filter in `_check_lint_delta` is supposed to mask
them, but a tiny edit shifts line numbers and every phantom resurfaces
as "introduced by this edit". The result is a 1MB+ phantom-error dump
on every patch that eats the agent's context budget. Same shape for
`.go` (`go vet` outside a module) and `.rs` (`rustfmt --check` outside
a Cargo project).
PR #24168 added an LSP tier on top of this — real `tsserver` / `gopls`
/ `rust-analyzer` diagnostics surface in the separate `lsp_diagnostics`
field. But the broken shell linter kept running underneath, so the
phantom-error dump kept happening even when LSP was giving us a clean
authoritative signal.
This change short-circuits the shell linter for the structurally-broken
extensions (`.ts`, `.tsx`, `.go`, `.rs`) when an LSP server is active
and claims the file via `LSPService.enabled_for(path)`. The LSP tier
runs as before and carries the real diagnostics in `lsp_diagnostics`.
Other shell linters (`py_compile`, `node --check`) keep running
unconditionally — they're fast, file-local, and correct.
Default behavior (LSP disabled, LSP misconfigured, remote backend, file
outside a workspace) is unchanged — the existing fallback paths trigger
when `_lsp_will_handle` returns False, so users who haven't opted into
LSP get the same shell-linter behavior they had before.
Drive-by: `.tsx` was missing from the `LINTERS` table entirely, so TS
React files got no post-edit syntax check at all. Added it for
symmetry; in practice it now hits the LSP-skip path.
Tests:
- `tests/agent/lsp/test_shell_linter_lsp_skip.py` — 14 tests covering:
* skip happens for each redundant extension when LSP claims the file
(asserted by patching `_exec` to raise on any shell-linter call)
* shell linter still runs when LSP is inactive (regression guard)
* `.py` / `.js` continue to run unconditionally even with LSP active
* `_lsp_will_handle` is exception-safe: returns False on None
service, remote backend, or `enabled_for` raising
* `.tsx` is in both `LINTERS` and `_SHELL_LINTER_LSP_REDUNDANT`
- All pre-existing tests in `tests/agent/lsp/` and
`tests/tools/test_file_operations*.py` still pass (233/233).
* fix(lint): address Copilot review on #29054
Two fixes from copilot-pull-request-reviewer on PR #29054:
1. `.tsx` regression with LSP disabled
(https://github.com/NousResearch/hermes-agent/pull/29054#discussion_r3271017282)
The first revision added `.tsx` to the `LINTERS` table so that
TypeScript React files would hit the LSP skip path. Side effect:
when LSP is *disabled* (the default), `.tsx` edits would suddenly
run `npx tsc --noEmit FILE.tsx` and inherit the same phantom-error
dump this PR is supposed to fix. Pre-PR behavior was implicit
`skipped` (no `LINTERS` entry); restore that.
- Remove `.tsx` from `LINTERS`.
- Remove `.tsx` from `_SHELL_LINTER_LSP_REDUNDANT` (the skip path
is unreachable without a `LINTERS` entry — falls through to
`ext not in LINTERS` first).
- When LSP IS enabled, `.tsx` is still covered by the LSP tier
via `_maybe_lsp_diagnostics` (typescript-language-server's
`extensions` tuple includes `.tsx`), so the diagnostics still
surface — just on the `lsp_diagnostics` channel, not `lint`.
- Update test_shell_linter_lsp_skip.py to reflect this contract
(drop `.tsx` from the parametrize lists; add
`test_tsx_stays_out_of_linters_table_for_default_compatibility`
and `test_tsx_default_check_lint_returns_skipped`).
2. V4A patches dropped `WriteResult.lsp_diagnostics`
(https://github.com/NousResearch/hermes-agent/pull/29054#discussion_r3271017295)
`tools/patch_parser.py::apply_v4a_operations` calls
`file_ops.write_file()` per operation, then calls `_check_lint()`
directly afterwards — but never propagates `WriteResult.lsp_diagnostics`
to the `PatchResult`. The shell-linter skip introduced in this PR
makes the gap visible: a `.ts` / `.go` / `.rs` V4A patch with LSP
active would return `lint = {f: {skipped: True}}` and zero
diagnostics from any channel.
- `_apply_add` and `_apply_update` now return
`Tuple[bool, str, Optional[str]]` where the third element is
`WriteResult.lsp_diagnostics` (or `None` on failure / no diags).
- `_apply_delete` and `_apply_move` stay 2-tuples — they don't
produce diagnostics, no write goes through `write_file`.
- `apply_v4a_operations` accumulates per-file diagnostics blocks
and surfaces a combined block on `PatchResult.lsp_diagnostics`.
Each block already carries its `<diagnostics file="...">` header
from `LSPService.report_for_file`, so concatenation preserves
per-file attribution.
Tests added (`test_patch_parser.py::TestV4ALspDiagnosticsPropagation`):
- ADD op: `WriteResult.lsp_diagnostics` flows to `PatchResult`
- UPDATE op: same
- No diagnostics → `PatchResult.lsp_diagnostics is None` (not "")
- Multi-file patch: combined block contains every per-file block
Verification:
- Targeted test scope: 257/257 pass
(tests/agent/lsp/, tests/tools/test_file_operations*.py,
tests/tools/test_patch_parser.py)
- Wider sweep: 5400 pass; 11 failures all pre-existing on origin/main
(file_staleness / file_read_guards / file_state_registry — unrelated
macOS /var/folders tmp-path sensitivity issues, confirmed by
re-running on a clean origin/main checkout)
* docs(test): align shell-linter LSP skip docstring with .tsx behavior
Copilot review feedback (review #4324947616, comment #3271049036):
the test module docstring still listed .tsx alongside .ts/.go/.rs in
the skip contract, but .tsx is now intentionally NOT in LINTERS or
_SHELL_LINTER_LSP_REDUNDANT. Updated the bullet list to drop .tsx from
the skip contract and added a paragraph documenting why .tsx is left
out (preserves pre-PR implicit-skip behavior for LSP-disabled users;
LSP coverage still happens via _maybe_lsp_diagnostics).
* test(lsp): drop unused tmp_path from _make_fops helper
Copilot review #3271069484: the helper accepted tmp_path but never
used it. Callers still need tmp_path themselves for the file they're
asserting against, so we just drop the helper's parameter.
Replace with for all literal-tuple
membership tests. Set lookup is O(1) vs O(n) for tuple — consistent
micro-optimization across the codebase.
608 instances fixed via `ruff --fix --unsafe-fixes`, 0 remaining.
133 files, +626/-626 (net zero).
Follow-ups on top of @teyrebaz33's cherry-picked commit:
1. New shared helper format_no_match_hint() in fuzzy_match.py with a
startswith('Could not find') gate so the snippet only appends to
genuine no-match errors — not to 'Found N matches' (ambiguous),
'Escape-drift detected', or 'identical strings' errors, which would
all mislead the model.
2. file_tools.patch_tool suppresses the legacy generic '[Hint: old_string
not found...]' string when the rich 'Did you mean?' snippet is
already attached — no more double-hint.
3. Wire the same helper into patch_parser.py (V4A patch mode, both
_validate_operations and _apply_update) and skill_manager_tool.py so
all three fuzzy callers surface the hint consistently.
Tests: 7 new gating tests in TestFormatNoMatchHint cover every error
class (ambiguous, drift, identical, non-zero match count, None error,
no similar content, happy path). 34/34 test_fuzzy_match, 96/96
test_file_tools + test_patch_parser + test_skill_manager_tool pass.
E2E verified across all four scenarios: no-match-with-similar,
no-match-no-similar, ambiguous, success. V4A mode confirmed
end-to-end with a non-matching hunk.
- Bug 1: replace read_file(limit=10000) with read_file_raw in _apply_update,
preventing silent truncation of files >2000 lines and corruption of lines
>2000 chars; add read_file_raw to FileOperations abstract interface and
ShellFileOperations
- Bug 2: split apply_v4a_operations into validate-then-apply phases; if any
hunk fails validation, zero writes occur (was: continue after failure,
leaving filesystem partially modified)
- Bug 3: parse_v4a_patch now returns an error for begin-marker-with-no-ops,
empty file paths, and moves missing a destination (was: always returned
error=None)
- Bug 4: raise strategy 7 (block anchor) single-candidate similarity threshold
from 0.10 to 0.50, eliminating false-positive matches in repetitive code
- Bug 5: add _strategy_unicode_normalized (new strategy 7) with position
mapping via _build_orig_to_norm_map; smart quotes and em-dashes in
LLM-generated patches now match via strategies 1-6 before falling through
to fuzzy strategies
- Bug 6: extend fuzzy_find_and_replace to return 4-tuple (content, count,
error, strategy); update all 5 call sites across patch_parser.py,
file_operations.py, and skill_manager_tool.py
- Bug 7: guard in _apply_update returns error when addition-only context hint
is ambiguous (>1 occurrences); validation phase errors on both 0 and >1
- Bug 8: _apply_delete returns error (not silent success) on missing file
- Bug 9: _validate_operations checks source existence and destination absence
for MOVE operations before any write occurs
V4A patches with only + lines (no context or - lines) were silently
dropped because search_lines was empty and the 'if search_lines:' block
was the only code path. Addition-only hunks are common when the model
generates patches for new functions or blocks.
Adds an else branch that inserts at the context_hint position when
available, or appends at end of file.
Includes 2 regression tests for addition-only hunks with and without
context hints.
Salvaged from PR #3092 by thakoreh.
Co-authored-by: Hiren <hiren.thakore58@gmail.com>
- Introduced file manipulation capabilities in `model_tools.py`, including functions for reading, writing, patching, and searching files.
- Added a new `file` toolset in `toolsets.py` and updated distributions to include file tools.
- Enhanced `setup-hermes.sh` and `install.sh` scripts to check for and optionally install `ripgrep` for faster file searching.
- Implemented a new `file_operations.py` module to encapsulate file operations using shell commands.
- Updated `doctor.py` and `install.ps1` to check for `ripgrep` and provide installation guidance if not found.
- Added fuzzy matching and patch parsing capabilities to improve file manipulation accuracy and flexibility.
