- patch_parser: shared _fail/_written/_unified_diff apply helpers, single PatchResult build,
flattened apply loop, merged inert/no-op hunk skips, overlay _remove, dropped never-set
PatchOperation.content
- self_repo_guard: conditional git-mutation predicates as a dispatch table over one _has_flag,
unified git global-option/alias parse, suppress() for resolve/alias/shlex, folded scope and
heredoc scanners
- shell_heredoc: unit scanner and operator parser as single if/elif ladders, unified quote masking
- read_extract: groupby gap runs, single try for anydoc, islice/suppress xlsx loops, folded
notebook/docx render loops, _zip_xml optional -> empty element
- schema_sanitizer: folded rename/type-array/recovery-strip loops, _dict_nodes/_carry_union_meta
- docstrings/comments compacted by hand (every WHY kept)
Verified: tool-schema dump byte-identical (SCHEMA-OK); golden corpus over pure functions and
every registry schema + edge cases byte-identical vs 113f046 base (GOLDEN-OK); 95 test files
2517 passed (2 pre-existing reds in test_web_tools_config also fail on base).
Efficiency review (measured with timeit probes) found two unbounded
costs on adversarial inputs:
- The masked-range rebuild copied the whole string once per range
(O(n*k)): 50k tiny heredocs took 1.7s. Replaced with a single-pass
segment join over the (sorted, non-overlapping) ranges: 152ms, and
newlines are now counted on the original command instead of
re-slicing.
- After the last '<<' occurrence no opener can start, but the scanner
still walked the remaining text per-char: one heredoc followed by a
1MB tail cost ~150ms. An rfind bound breaks out of the unit loop
once the scan passes it: 0.3ms.
Typical commands are unaffected (the '<<' fast path already returns
first). 30/30 guard tests pass; mutation check re-run on the final
stack (no-op mutation -> 11 tests fail, restore -> green).
The previous commit's regex-based stripper removed EVERY heredoc body,
which review flagged as bypassable: a fake '<<EOF' marker inside a
comment or quoted string enters the unterminated path and swallows a
later REAL background operator, and unquoted ('cat <<EOF' — expansion
runs) or shell-consumed ('bash <<'EOF'' — body IS shell) bodies are
executable content that must stay visible to the guard.
Replace it with tools/shell_heredoc.strip_inert_heredoc_bodies(), a
conservative shell-state scanner: a body is masked ONLY when every
delimiter on the opener is quoted (no expansion), every heredoc is
terminated by an exact delimiter line, the opener composes a single
command (no list/pipeline operators, no nested $()/backtick/process-
substitution scope), and the consumer is an allowlisted non-shell
interpreter (python/osascript/cat). Anything ambiguous is returned
unchanged — a false positive on exotic syntax is acceptable; hiding a
real background operator is not. Masked bodies become newlines so line
structure is preserved for MULTILINE regexes.
The helper is a standalone stdlib-only module (precedent:
tools/ansi_strip.py) because the same heredoc-as-data false-positive
class exists in the blocked-command regex checks (#83104) and the
gateway lifecycle guard (#81721/#79835, cron/lifecycle_guard.py) —
which must not import the terminal-tool module graph.
Adapted from Wolfram Ravenwolf's security-hardened rework of #63788
(69c7663c6de6b6cb05bf99203fa39673efe01ccf); test scenarios for the
bypass cases derive from his suite.
Co-authored-by: Wolfram Ravenwolf <github.com@wolfram.ravenwolf.de>
