The fold added in #131679 runs `git gc --auto`, which on a large partial
clone is a full repack: a 9.6 GB checkout with 463 packs sat in it for
20 minutes per update, got killed at the cap, and started over on the
next update without ever finishing. It also never freed disk (on-demand
packs are kept, only merged). Turn it off until a bounded, progressive
cleanup replaces it.
The partial-clone keys that stop git writing a commit-graph (#127711)
stay: settle_partial_clone_maintenance re-applies them every update and
leaves full clones alone, and git's own automatic gc still folds packs.
73c17151f6 (Sep 29) persisted maintenance.auto=false, gc.auto=0 and
fetch.writeCommitGraph=false in treeless install checkouts. 6880ffc1af
stopped writing gc.auto=0 but never removed it, and `gc --auto` with
gc.auto=0 is a no-op (git skips every heuristic, gc.autoPackLimit included),
so the update's pack fold never ran on those checkouts: 203 -> 203 packs.
The previous commit's fingerprint required gc.writeCommitGraph=false, which
only the Oct 2 cut wrote, so the Sep 29 cohort also kept maintenance.auto=false.
Both cuts wrote maintenance.auto=false together with
fetch.writeCommitGraph=false and neither wrote
maintenance.commit-graph.enabled; that pair is now the fingerprint, and
gc.auto=0 is removed under it as well. An operator's own lone
maintenance.auto=false still never matches.
Real clear_git_debris() on a treeless clone with 203 lazy-fetch packs, git
2.53.0 and 2.55.0 alike:
Sep 29 keys: main 203 -> 203, #131741 203 -> 203, this 203 -> 1 (both keys gone)
Oct 2 keys: 203 -> 1, maintenance.auto gone
operator's own maintenance.auto=false: 203 -> 1, setting kept
_restore_auto_maintenance unset any local maintenance.auto=false, so an
operator's own setting on a partial-clone checkout was erased on every
update (nothing records who wrote it). Remove it only with the first cut's
fingerprint: maintenance.auto, gc.writeCommitGraph and fetch.writeCommitGraph
all false and maintenance.commit-graph.enabled not yet set. The first pass
of disable_tree0_auto_maintenance writes that key, so the migration runs once
and an operator's setting is left alone before and after.
Review point from @Godhet on #131718.
#131679 set maintenance.auto=false in the treeless install checkout to stop the
commit-graph lazy-fetch loop (#127711). That also switched off the post-fetch
`gc --auto` that git <= 2.53 runs after every lazy fetch, so packs piled up
between updates and only folded on a manual `hermes update` / `--check`
(nothing runs `--check` automatically).
The loop needs a commit-graph write, and gc.writeCommitGraph=false +
fetch.writeCommitGraph=false already stop gc and fetch from writing one.
maintenance.commit-graph.enabled=false closes the last writer (a commit-graph
maintenance task, e.g. from a user's global maintenance.strategy) while
leaving automatic maintenance, and its gc fold, on.
Checkouts that already got maintenance.auto=false from the first cut are
converged: the next update/--check unsets exactly that value.
Measured on a real tree:0 clone with a Bloom commit-graph, 100 lazy fetches
after one updater run:
git 2.53.0: main 101 packs, this change 1 pack, 0 commit-graph writes
git 2.55.0: 101 / 101 (stock 2.55 maintenance does not fold), 0 writes
The fold was killed after 120 s on the assumption it takes "well under a second". That holds
for ~100 small packs, not for the checkouts it exists for (#129712: 2,475 packs / 39 GiB): those
need a full repack, and a killed fold restarted from scratch on every update without a word,
so the worst-hit installs never converged.
- Bound raised to 20 minutes (measured: 750 packs / 530 MB of hermes-agent history fold in
5.8 s at 7.4 cores; a 39 GiB pack dir on a slow disk is a different order of magnitude).
- consolidate_lazy_fetch_packs returns None on timeout; the shared update_cmd_check reporter
(used by both `hermes update` and `hermes update --check`) prints the hand-fold command.
- on_fold_start announces a fold that will actually run (pack count past gc.autoPackLimit), so
a multi-minute first fold is not a silent hang.
- Test fixture disables auto-maintenance: each lazy fetch's detached gc otherwise raced the
test and folded the packs itself.
Replaces the salvaged tests, which mocked subprocess and the gc command line,
with real-git invariants on a blobless clone:
- lazy-fetch packs fold, keep their .promisor markers, and the history still
reads under GIT_NO_LAZY_FETCH, with the maintenance keys applied (gc.auto=0
makes this fail);
- the fold adds no pack for commits a Bloom commit-graph has not seen (11
packs instead of 1 without the guards);
- `hermes update --check`'s debris cleanup is where the fold happens;
- a full clone keeps git's stock maintenance.
Every on-demand fetch on the promisor remote of a tree:0 installer
checkout writes its own small packfile, and no update-path step ever
consolidates them, so .git grows without bound (2,475 packs / 39 GiB
observed for a ~1 GiB repo, #129712).
Ride git's own threshold: run `git gc --auto` from the debris-cleanup
step both `hermes update --check` and `hermes update` already run.
Below gc.autoPackLimit (default 50) small packs it exits in
milliseconds; past it, the lazy-fetch packs fold into one. The folded
pack is re-marked .promisor so git 2.53+ fetches keep working (#124272),
and the gc does not write a commit-graph, which git <=2.50 misreads on
the next lazy fetch.
- _deepen_shallow_repo marks packs in its finally without a git config probe
that could itself raise and turn a finished unshallow into False; markers
are inert in a clone without a promisor remote.
- Installers warn instead of aborting the stage when a marker cannot be
written, matching the Python helper.
- Docs: the manual commands follow HERMES_HOME, and the missing-object
backfill feeds IDs through `fetch --stdin` so a large list cannot exceed
the Windows command-line limit.
- Test and docstring comments describe the contract, not the incident.
git 2.53+ `index-pack --promisor` hands every object outside the promisor
packs to pack-objects, which BUG()s (should_include_obj) on the missing
objects those objects lead to. A partial clone with unmarked packs therefore
keeps crashing its fetches.
The retry added for this ran `git -c remote.origin.promisor= fetch`. That
override does nothing: git registers promisor remotes additively, so the
repo's own `true` still wins (GIT_TRACE on 2.50 and 2.55 shows
`index-pack --promisor` and `filter tree:0` under the override). The retry
only passed when the crashed fetch had already stored the pack; a retry that
has to transfer again crashed again.
- On the crash, give every pack without a `.promisor` marker one, then retry
the same fetch once. The clone stays partial, no refetch, no config change.
If git still crashes, the updater points at the docs section.
- The two fetches that turn a full clone partial now mark the clone's old
packs afterwards: the depth-1 `tree:0` unshallow in
`fetch_full_commit_graph`, and `worktree_ops._deepen_shallow_repo`
(`--filter=blob:none`). git writes the partial-clone config before it
fetches, so this also runs when the fetch fails. Without it, a converted
depth-1 install crashed every later fetch on git 2.55 (3 of 3 rounds in a
local repro).
is_partial_clone_pack_objects_crash() required "index-pack failed" next to the
pack-objects assertion, but git 2.55 builds dropped that string: Homebrew git
2.55.0 (macOS) only reports "fatal: fetch-pack: invalid index-pack output", so
the recognizer always returned False there, fetch_with_partial_clone_recovery()
never retried with the promisor machinery disabled, and `hermes update` exited 1
on the crash (#125138 reports the same bare-assertion failure on 2.55.0.windows.5).
Fingerprint on the assertion text — the only version-stable part of the failure
— and accept the 2.55 wrapper as a terminator. The POSIX/Windows shapes and the
"fingerprint without a terminator" rejection behave as before.
(cherry picked from commit 74103b15ceb3eacb2346fca669c3a3dc81457201)
The upgrade E2E cells are the regression proof; the unit tests keep one
invariant per behaviour: full clone stays full / partial clone keeps its own
filter (#122353), crash recognizer + single promisor-disabled retry (#124272),
heal of a depth-1 clone / of a --depth-grafted full clone (#123254, #123346).
Co-authored-by: salch-cred <salch-cred@users.noreply.github.com>
Git for Windows 2.54 reports the same partial-clone pack-objects BUG()
without the POSIX 'pack-objects died of signal 6' line (#124293 field
evidence): the BUG assertion is followed directly by 'fatal: could not
finish pack-objects to repack local links' and 'fatal: index-pack failed'.
The recognizer demanded the signal wording, so the recovery retry missed
this Windows occurrence entirely.
Require the BUG fingerprint plus EITHER terminator of the same abort —
the POSIX signal line or the Windows repack fatal — and pin both stderr
shapes plus a fingerprint-without-terminator negative in the regression.
On a partial clone (clone --filter=tree:0), git 2.53/2.54 crashes every
fetch: index-pack --promisor runs repack_local_links(), which feeds
pack-objects --exclude-promisor-objects-best-effort, and pack-objects
BUG()s (SIGABRT, 'should only be called on existing objects') when the
link traversal hits a legitimately missing promisor object. The crash is
deterministic, so every fetch — and with it the CLI's update/check flows
— stays broken until the user applies the manual workaround themselves.
The recovery is the reporter's verified workaround, wired in: on this
exact crash (all three stderr markers), retry the fetch once with
-c remote.origin.promisor= (per-invocation only; the user's filter
choice stays in their config). Wired into both fetch sites — the update
fetch and the check fetch (which also serves forks via upstream).
Unrelated fetch failures are returned untouched; if the retry still
crashes, the update path prints the manual heal command.
Fixes#124272
`scripts/run_tests.sh tests/<dir>/` is how a change gets its regression
coverage run, so a test filed under the wrong directory is a test nobody
runs when that code changes. Two kinds of drift had accumulated.
Parallel directories for one source package, folded into the mirror:
tests/acp -> tests/acp_adapter (its __init__/conftest move with it)
tests/cli -> tests/hermes_cli (prompt_toolkit fixture merged into
hermes_cli/conftest.py)
tests/run_agent -> tests/agent (backoff fixture becomes
agent/conftest.py)
tests/relay -> tests/gateway/relay
tests/state -> tests/hermes_state
246 loose files at tests/ root, routed by the package they import/patch:
hermes_cli, hermes_state, agent, gateway, tools, plugins, tui_gateway, cron.
Installer and desktop-update script tests go to tests/scripts/{install,
desktop_update}/. 43 tests of root-level modules (batch_runner, utils,
hermes_constants, packaging) stay at the root.
Filenames drop their issue numbers (95 files: test_89315_x.py -> test_x.py);
the number stays in the module docstring where it has context.
Collisions: test_cli_skin_integration.py existed in both tests/ and tests/cli
with different subsets — merged into one (10 tests, all kept);
run_agent/test_pre_compress_memory_context.py -> agent/..._handoff.py;
tests/test_account_usage.py -> agent/test_account_usage_fetch.py;
tests/test_web_server.py -> hermes_cli/test_web_server_ws_ping.py.
Deleted: test_minisweagent_path.py (empty since PR #2804),
test_model_picker_scroll.py (tested a private copy of the logic, imported
nothing), test_process_loop_event_loop_warning.py (asserted asyncio behaviour,
imported nothing from Hermes).
Repo-root path arithmetic (Path(__file__).parents[N], dirname chains) is
bumped for the 202 files that changed depth and verified by evaluating every
such expression against the new location. classify_changes' desktop-updater
lane prefix, tests-os.yml's ignore glob and every in-tree path comment follow
the moves. tests/test_tests_tree_layout.py keeps the tree from drifting back.
