Stopping only orphaned incumbents had also made a replacement boot give up
when the incumbent's owner was alive, adopting a router whose spawn-only
model list lacks the model just staged (e.g. the desktop downloading while
a CLI owns the server). That case now boots its replacement beside the
live owner's server exactly as before; the only behaviour change left is
that orphans are stopped.
_owner_is_dead had tightened "newer incarnation" to "any create_time
mismatch" for records without start_time. On WSL a stepped clock moves
create_time, so a live owner from an older Hermes could read as dead and
have its server stopped. Those records keep the original comparison.
_stop_state_server received _state_endpoint()'s {base_url, api_key} and
read a "pid" it never carries, so stale-preset replacement and the rescan
bounce never stopped an adopted router: the old one kept its VRAM and the
new one fell back to an ephemeral port. Its tests mocked a pid into the
endpoint. It now resolves the process from the identity-guarded state
record and stops it only when the recorded owner is gone; a live owner's
watchdog would respawn its router beside ours, so that server is left as
is and no second one boots.
Only the desktop backend stopped the server on exit; every other process
that booted it on demand (CLI, -z, `hermes model`, gateway) orphaned it
with its VRAM. The process that boots the server now stops it at exit
(atexit; one-shot's os._exit path calls it from its cleanup table).
Processes that merely adopted a server never stop it.
Verified on WSL: `hermes model`, `hermes -z` and interactive `hermes`
leave no llama-server behind after exit.
b11146 ships ubuntu-cuda-13.4-arm64 (+ cudart), so llamacpp-cuda gains
linux-arm64, auto resolves NVIDIA arm64 Linux to CUDA, and the desktop's
localModels flag ships on Linux arm64 too.
Every upstream Linux llama.cpp build (CPU included) links the system
libgomp, which minimal hosts (WSL, containers) lack, and a normal install
never touches the system package manager. Each Linux engine now pins
Ubuntu 24.04's libgomp1 .deb (glibc 2.38, the floor upstream already sets)
and stages libgomp.so.1 beside llama-server, where the $ORIGIN RUNPATH
finds it; the artifact mirror covers the pool URL after 24.04's EOL.
DebPackage's ar+tar unpack becomes pm.package.unpack_deb and reads zstd
data.tar (Ubuntu's default) with the stdlib codec PM's 3.14 runtime has.
Pre-PM engines on Linux carry only llama.cpp's own archives, so adoption
maps their manifests to those and records them as outdated (usable now,
update offered) instead of failing to match.
Verified on an RTX Spark N1X (WSL, Ubuntu 26.04, no system libgomp):
ensure_engine() for cuda, cpu and vulkan through PM's real download/stage/
verify path; ldd resolves libgomp from the entry; llama-server serves
completions (CUDA0 at 445 tok/s). linux-x64 cpu/vulkan staged through the
same path with no NEEDED library missing beyond glibc/libstdc++/OpenSSL 3.
launch_budget read free memory only from nvidia-smi, so a discrete AMD/Intel card launched at its
capacity window regardless of other programs' VRAM, the case that silently pages on Windows. It
now asks the engine's device probe afresh at launch (the polled view stays cached); the driver's
free count includes other processes (live: 16.2 GB held elsewhere read as 8.2 GB free of 26.0).
Docs: auto is a preference with the platform's pinned fallbacks, not an unconditional CUDA pick.
The managed llama-server is one per process and serves every profile it hosts, but the budget
re-derived its engine from whichever profile's config the current thread saw. The on-demand boot
thread and the Local Models job workers were bare threads, so under multiplex a named profile on
Vulkan booted a server sized from the default profile's CPU config (262144-token window instead of
98304, reproduced live by an independent review).
- bootstrap records the engine it boots (cleared on shutdown and on a failed boot); the device
probes price that engine whenever one is up or booting, config only otherwise.
- The on-demand kick and the Local Models job/refresh threads carry the caller's contextvars
(spawn_context_thread), so config=None and the post-download refresh resolve the right profile.
_boot_in_flight gated the on-demand boot on installed_engine() with the
default "auto", not the configured local_runtime.backend. With no GPU
vendor detected, auto resolves to the CPU build only, so a user on
`backend: vulkan` with only the Vulkan engine installed never got a boot
from `hermes chat` ("The local model server isn't running"), while
ensure_local_runtime itself would have booted that exact engine. Found
during the live run for this PR.
Both engine probes called get_config_value("local_runtime.backend", "auto"),
but get_config_value is the `hermes config get` printer with signature
(key, *, as_json, raw): the call raised TypeError, suppress(Exception)
swallowed it, and the probe returned None every time. That made the
discrete Vulkan/HIP budget dead code in production, and the CUDA
--list-devices pool fallback has been dead since it landed (bdb52ce711).
Found by a live run with NVIDIA hidden at the OS level: the device probe
answered, yet probe_budget still returned the RAM-as-unified budget. One
helper now resolves the configured engine via load_config_readonly(), and
the Vulkan test reads a real config.yaml instead of patching the probe out.
The previous commit budgets any Vulkan/HIP device that `llama-server
--list-devices` prints as a discrete card. That output carries no
integrated-vs-discrete verdict, so an iGPU (Intel ADL-N reports 11.8 GB of
a 16 GB machine's shared memory) would be counted twice: once as VRAM and
again as system-RAM spill.
Ask the installed engine's ggml library instead, in a bounded child
process: ggml_backend_dev_type() is GPU for a discrete card and IGPU for
an integrated one (Vulkan reads it from VkPhysicalDeviceType). llama.cpp
places layers on discrete devices whenever one exists and falls back to an
integrated GPU only otherwise, so the budget follows the same rule:
- discrete device -> its own memory, system RAM as spill (uma=False)
- integrated GPU, no answer, CPU/Metal -> RAM-as-unified, unchanged
The probe runs only when no NVIDIA device answered and the configured
engine is Vulkan/HIP, and is cached per engine binary (the hardware
endpoint polls every few seconds). The CUDA --list-devices fallback for the
NVIDIA unified-pool quirk is left as it was on main.
Co-authored-by: Belaquaa <eee_pokkk@mail.ru>
On Linux psutil derives create_time from /proc/stat's btime, which moves
whenever the wall clock is stepped. WSL re-syncs its clock within minutes,
so recorded_process() rejected a live router on exact create_time equality
and every turn failed with "The local model server isn't running."
server.json now also records start_time/owner_start_time from
gateway.status.get_process_start_time() (boot-relative ticks on Linux) and
recovery compares them with start_time_fingerprints_match(). create_time is
still written and is the fallback for records that predate start_time, so an
orphaned router stays adoptable across an update.
A console-less parent (pythonw backend, detached daemons) allocates a visible
console per bare spawn, so Windows startup flashes ~7-8 console windows
(#117781) and the GPU statusbar flashes two more every 5s (#101895). Threads
creationflags=windows_hide_flags() through every hermes_cli spawn site the
desktop reaches: gitlock (tasklist probe + git plumbing), the scheduled-task
PowerShell probe, the updater's _git_run/_no_prompt_git_kwargs/_GIT_TEXT_KW
runners, the safe-directory config probe, and consolidates the statusbar's two
nvidia-smi reads per poll into one cached hidden query. The statusbar also
stops polling while the document is hidden and resumes on visibilitychange
(#120262).
Supersedes #117805, #117833, #102047, #120285 (thanks @JoaoMarcos44, @Finn763,
@monerostar, @szzhoujiarui).
A spilled hybrid got `-ot blk\.\d+\.ffn_.*\.weight=CPU`, which moves every
block's FFN — full-attention blocks included — regardless of how many bytes
actually had to leave the GPU. The GGUF reader now records FFN weight bytes
per block, and spill_overrides picks the fewest recurrent (n_head_kv==0)
blocks whose FFNs cover the decision's spill_bytes, emitting them as an
explicit `blk\.(i|j|...)\.ffn_.*\.weight=CPU` selector. Unknown sizes fall
back to every recurrent block, never a full-attention one.
Based on #120778 by @kvnloo (recurrent-index selector).
Covers the placement half of #113329; the overhead/logits calibration
(RUNTIME_OVERHEAD_BYTES, ub_logits_bytes) is untouched pending measured data.
Refs #113329
llama.cpp permits attention.sliding_window_pattern as a scalar period as
well as a per-layer array; current writers for Gemma-family and other
architectures use the scalar form, which read_gguf_header/profile_from_gguf
silently dropped (isinstance(v, list) is False), sending unlisted
architectures back to the all-full-attention fallback this fix was meant to
avoid.
Add GGUFHeader.sliding_window_pattern_period and expand it per llama.cpp's
set_swa_pattern() semantics, including the small set of architectures
(cohere2moe, modern-bert, smallthinker, laguna) whose loader uses
dense_first=True instead of the default False.
Also add binary GGUF round-trip tests (real read_gguf_header() parsing, not
GGUFHeader built by hand): array pattern with non-bool element types, both
scalar dense_first variants, and a length-mismatched array falling back
safely.
profile_from_gguf() only consulted a hardcoded architecture -> SWA-fraction
table (_SWA_LAYER_FRACTION). Any architecture absent from that table (e.g.
gemma4) fell through to pricing every layer as full attention, wildly
inflating the estimated KV cache and causing the physics check to refuse
models that fit and run fine.
GGUF files that declare a per-layer SWA layout already carry
attention.sliding_window_pattern (plus attention.key_length_swa /
value_length_swa for the SWA layers' dimensions) — gguf.py had no accessor
for any of the three. Add them, and make profile_from_gguf() prefer the
file's own per-layer pattern over the name table, falling back to the
table and then to all-global exactly as before when the file doesn't
declare one.
* refactor(plugins): remove the Sep 2026 decomposition compat layer on schedule
The PLUGIN-COMPAT layer (2776813df3 + d63e380324 + 0a5164cebe) kept pre-#102117 import paths
alive for external plugins until 2026-09-14. That window closed two weeks ago; since then the loader
has already been skipping plugins that use the old paths. This removes the layer itself:
- 328 appended `PLUGIN-COMPAT` blocks (lazy `__getattr__` pointer tables, re-exported third-party
names, restored dead definitions) and the three re-export stub modules
(gateway/startup_watchdog, hermes_cli/observability/relay_runtime, tools/environments/modal_utils)
- COMPAT_MANIFEST.md, compat_manifest.json, scripts/check_compat_pointers.py and its lint step
- the reporting surfaces: CLI banner notice, `hermes plugins compat`, the `hermes doctor` section,
the post-update notice, the Desktop one-time dialog, the loader's pre-import skip and the
`plugins.allow_deprecated_imports` escape hatch
An external plugin that still imports an old path now fails to load with its ImportError as the
reason in `hermes plugins list`, the same path as any broken plugin.
hermes_cli/plugin_compat.py stays as three inert stubs (compat_report, removal_in_effect,
summary_lines): an already-running pre-removal `hermes update` lazy-imports them after the checkout
swap (tests/compat/old_updater_surface.json).
In-tree fallout, both already dead: hermes_cli/setup.py::_check_espeak_ng (no callers; its
`shutil` came from a compat block) and gateway/config.py::SessionResetPolicy ("retained solely for
the scheduled plugin-compat window"). Two test_run_agent patches targeted the removed
`run_agent.handle_function_call` pointer; they now patch `model_tools.handle_function_call`, the
seam production reads, like every sibling test in that file.
* chore: retrigger CI (zero-job startup_failure phantom)
* test: drop resolution allowlist rows for the two deleted which() sites
hermes_cli/setup.py::_check_espeak_ng (dead) and tools/skillevaluator_scan.py::scanner_available
(a restored definition inside a PLUGIN-COMPAT block) no longer exist; the stale-row gate requires
their allowlist entries go with them.
Boot presets and in-session growth priced the context window from card
capacity: total VRAM minus a fixed max(2 GiB, 9%) reserve. That reserve
covers a light desktop only. On an RTX 5090 with 4.5-6.3 GiB held by a
browser and other apps, Qwen3.8 27B booted at 216K, overflowed the card,
and Windows paged part of it to host memory without an error: decode
fell from ~90 to ~24 tok/s.
hardware.launch_budget subtracts what other programs hold now (device
total minus nvidia-smi free, minus our own server's footprint) plus
1 GiB of headroom. fit_to_free_memory narrows a resident plan's window
to that budget, down to the floor and never further: weights are never
moved to the CPU on a live reading, which once pinned a fitting model
to the CPU while the reading still counted the outgoing server.
- Boot: preset generation narrows every window through it.
- Idle: every 30 s with no model holding memory, the presets are
re-planned and the router re-reads them (GET /models?reload=1), so a
later on-demand load gets a window for the current desktop. The file
is never rewritten while a model is loaded, because reload unloads a
loaded model whose flags changed.
- Growth: the next rung must fit beside other programs, counting the
growing model's own memory as free.
Recommendations, quant selection and catalog pricing keep the capacity
budget. Unified-memory machines and failed probes keep today's plan.
The new tests pin the catalog 27B's window from 0 to 8.5 GiB of other
programs. Every boot, idle, growth and residency-cap test records its
probe_budget calls and fails on any made without planning=True, including
calls inside paths that swallow exceptions.
The package-manager switch (#102765) made engine detection read only PM's
store and pinned every llama.cpp backend to b10362. A machine with an
engine under runtimes/llamacpp/b<tag>/<backend>/ reported no engine, so
the local models pane showed one-click setup with models already on disk.
installed_engine() now moves the newest pre-PM install into the store
once. The old manifest's archive digests become the PM identity, the
package verifier runs llama-server --version, and os.rename puts the
directory in place before facts.json records it. A store that already
holds an engine is left alone, a store lock held by another PM operation
defers the move, and an install that fails verification stays where it
is for the rest of the process.
The lock returns to b10964 on all five backends, the default before the
PM switch. With b10362 pinned, the pane offered a downgrade from the
moved engine as an update. Upstream renamed the ROCm archives to 10.0 at
b10767, and the hip assets follow. Only the CUDA build is measured at
b10964.
Models used to live at <profile home>/models (adb2fdbf5c) before the managed
runtime moved them to the machine-scoped <root>/models (43e67d872f). GGUFs
staged under a named profile's old dir silently stopped being served.
adopt_legacy_models() renames them (and their assets/) into the current dirs,
so everything downstream keeps reading one directory: listing, presets,
delete, the --models-dir fallback. It runs at boot, before the "anything
staged?" check, and in the Local Models status route so the pane lists them
while the runtime is off.
os.rename only: instant within a filesystem, while shutil.move would silently
copy tens of GB across devices at session start; a cross-device dir stays put
with a warning. An existing destination name is never replaced.
On Windows, subprocess text=True without an explicit encoding decodes
child output with the ANSI code page (e.g. 'gbk'); non-ASCII bytes then
raise UnicodeDecodeError inside subprocess._readerthread, killing the
Hermes backend before it becomes ready and surfacing as the desktop boot
timeout.
Sweep every hermes_cli text=True subprocess call to encoding='utf-8',
errors='replace', and add an AST-based regression test that fails when a
future text-mode call omits the encoding.
Fixes#55658
utf-8-sig exists to tolerate BOMs that Windows tooling adds to files
users edit. /proc and /sys files are generated by the Linux kernel, never
BOM'd and absent on Windows, so -sig there only muddies the read/write
policy. Switch every literal /proc/ and /sys/ read to utf-8 and teach
the footgun read rule that string literals starting with /proc/ or
/sys/ are exempt (user-edited files keep utf-8-sig).
getconf _AVPHYS_PAGES counts page cache as used, so the Desktop statusbar
overreports Linux RAM. Read MemAvailable from /proc/meminfo, fall back to
MemFree, then the existing getconf probe. A reported 0 is a real available
value, not a missing field.
Fixes#102252
Symptom: a 22 GB catalog download ended with "[WinError 5] Access is denied:
'...\models\<model>.part'" even though the complete .gguf was on disk, and a
retry then found it and finished in seconds.
Root cause: download_file published the finished .part with shutil.move. On
Windows the just-closed file is commonly still open to an antivirus or
indexing scan, so os.rename fails with a permission error; shutil.move then
silently falls back to copy2 + unlink, which duplicates the whole file
(many seconds of disk churn and double the space) and finally reports the
leftover's failed delete as the download's failure. The except-path cleanup
could also replace the real error with its own unlink failure.
Fix: binaries.replace_when_released renames with os.replace and retries a
PermissionError with backoff through a bounded window (60 s), raising a
plain-language error if the hold outlasts it; there is no copy fallback.
Both the model download and the runtime-archive download publish through it,
and both cleanups suppress a failed leftover removal so the original error is
what the user sees. The job narrates the wait as "Finishing" so the bar does
not sit dead at 100%.
Tests: helper retries a transient refusal and lands the file; helper gives up
with the plain-language message chained to the OS error; route-level download
survives two refused renames with no .part left behind; a stuck leftover does
not mask the rename failure.
Branch semantics kept where main and PM disagree: update_cmd_deps.py,
constraints-termux.txt, the Electron update-api-check module and the
post-swap hand-off test stay deleted; the pending-fleet-restart catch-up
and the local_runtime tag/download ladder stay retired (PM owns engines).
Ported from main onto the branch's shape: profile_scoped_chore for the
auto-archive and plugin-update housekeeping chores, the local-runtime
cross-process boot lock and residency cap, the checkpoint tmp_pack sweep,
the cua daemon-liveness status probe, the remote-served Desktop update
flag (posix.sh / windows.ps1), sign-in for env-pinned remote gateways
(urlDisabled on RemoteSetupFields), the uvloop extra split (uvicorn
without [standard]), and the umask-scoping spawn test.
uv.lock regenerated with pm.build_env --lock-only; new utf-8 reads from
main switched to utf-8-sig (check-windows-footguns).
Residency was bounded by `local_runtime.models_max` alone, so a second model was admitted
against an already-full card. On Windows/WDDM that over-commit is not refused: the
allocation is paged to host memory and the child decodes at about a third of its speed for
the rest of its life — no error, no UI hint, and ejecting the incumbent afterwards does not
repair it (only a clean reload does).
The router now gets a cap priced from the same capacity budget the presets are priced
against: the count rises above one only while the largest staged model still fits twice, so
any pair of staged models is inside the card by construction and llama.cpp evicts its LRU
before an incoming child allocates. `models_max` stays a ceiling (a user's smaller number is
honoured) and an unpriceable budget or model keeps today's count.
Refs #116078
(cherry picked from commit d7abad7984d16d23ab2442b92d623b2270776586)
Review follow-up: the boot lock's mkdir/os.open ran in the context manager's
__enter__, outside the body's try/except, so a read-only runtimes dir or a
foreign-owned boot.lock raised OSError out of ensure_local_runtime — the
function main documented as never raising into session start. Catch OSError
around acquisition, warn, and yield unlocked (the contention-timeout shape).
Two Hermes backends starting within the same second (e.g. default +
non-default profile) each see no server.json yet and each spawn a
llama-server router on the stable port — the existing per-process
_SUPERVISOR singleton can't rule this out since every process starts
with its own copy. Wrap the state-check/spawn sequence in an OS-held
file lock (fcntl/msvcrt, mirroring managed_uv.py's install lock) so the
second caller waits for the first to publish its state and adopts it
instead of racing to spawn its own.
Fixes#116682
spawn_server also backs _subprocess_compat.bounded_probe_run (git probes,
PowerShell/tasklist scans, the update venv probe) and the scrub overrode an
explicit env= too, so every probe lost *_TOKEN/*_API_KEY/PASSWORD* (GH_TOKEN for
a gh credential helper, HF_TOKEN for a gated pull). Apply server_child_env in
supervisor._spawn and leave spawn_server's caller environment untouched.
The managed router child was spawned with the full process environment, so
every provider/tool credential (`*_API_KEY`, `*_TOKEN`, `*_SECRET`,
`*PASSWORD*`, `*_CREDENTIALS`) reached a native binary that talks to nobody
but us. On Windows that was not just a leak: the bundled libomp.dll died with
STATUS_HEAP_CORRUPTION (0xC0000374) during OpenMP initialisation with one
`*_API_KEY` present in the inherited Desktop environment and loaded fine
with only that variable removed — confirmed twice on a model-free
`ctypes.CDLL(libomp.dll).omp_get_max_threads()` probe (#116109).
Scrub those names at the single spawn boundary (`spawn_server`), leaving
PATH, CUDA_*/HSA_*/OMP_* and everything else untouched. The upstream
OpenMP defect itself is not ours to fix; keeping secrets out of the native
child is correct regardless and removes the trigger.
Keeps three invariants across #116156 (@fangliquanflq) and #116292 (@Finn763):
a configured providers.llamacpp entry wins over managed-server detection on the
runtime path; an external llama-server on a local_runtime.detect_ports port is
what /model > Local (switch_model with the picker's provider id) resolves to;
a staged GGUF alone keeps the picker's id resolvable so the runtime seam, not
the provider gate, reports a missing server.
Drops the endpoint.py config auto-load from #116292: both production callers of
resolve_llamacpp_endpoint() now pass the loaded config (#116156), so loading it
again inside the resolver was defense-in-depth. Drops the change-detector test
on the forwarded config object and the redundant negative control.
Docs: local-models page now shows detect_ports and the providers.llamacpp
override for a llama-server the user runs on a fixed port.
The /model -> local/ flow offers the 'llamacpp' row from staged GGUFs alone,
but resolve_provider_full only admitted that id behind a live endpoint, so
selecting a staged model died with "Unknown provider 'llamacpp'" before the
runtime seam could start or attach a server.
- providers.py: LLAMACPP_PROVIDER_ID/ALIASES are the single definition; the
llamacpp rung resolves when a server is reachable OR a model is staged.
- inventory.py: the picker row's slug comes from that definition.
- endpoint.py: honour local_runtime.detect_ports (every provider caller
invoked the resolver bare, leaving the documented knob dead).
- model_switch.py: a local-runtime alias failure surfaces the seam's own
message instead of an API-key hint.
Merges from origin/main resolved several files by taking main's side, which
re-inlined code paths this branch had already moved to PM. At HEAD that left:
- plugins/memory/hindsight: ImportError at module load (`_export_port_health_
grace_timeout` and `_MIN_CLIENT_VERSION` no longer exist) — the whole plugin
failed to import. Restored the side-env daemon client, folded main's
fail-closed profile-env rewrite guard into it, and dropped the version
auto-upgrade dance (the extra is pinned).
- mem0 / honcho / google_chat / memory_provider_migration / plugins_cmd_catalog:
imported the deleted tools.lazy_deps and hermes_cli.plugin_python_deps
modules; routed through pm.ensure_import / pm.sync_venv / pm.plugins_state.
scripts/ci/check_lazy_deps_imports.py exits 0 again.
- tools/skills_hub: the PLUGIN-COMPAT __getattr__ had been moved above
`_plugin_compat_prev_getattr = __getattr__`, so `SKILLS_DIR` recursed forever.
- tools/tirith_security: `import time` dropped; the circuit breaker NameError'd.
- hermes_cli/local_runtime/binaries: json/platform/os used without imports;
the manifest.json scan was for a layout PM no longer writes — the boot gate
now asks PM for an installed engine.
Same class as the Bot Chat drain wedge already on this branch: every JSON-file
scan guarded "did it parse?" and then assumed the value was a dict. A file
holding `42`, `"oops"` or `[1,2,3]` (corruption, truncated write, foreign tool)
passed the guard and raised AttributeError/TypeError at the first `.get()`,
usually before a single healthy sibling was processed. Each site now treats a
non-object payload like a corrupt file under that subsystem's existing policy:
- tools/bot_relay.py::_expire_if_stale / claim_pending_envelopes — the
envelope is skipped by the sweep and not claimed (same as unparseable).
- tools/browser_lightpanda.py::reap_orphaned_lightpanda — record unlinked,
scan continues.
- tools/write_approval.py::list_pending / get_pending — record skipped with
the existing "unreadable pending record" warning / None.
- tui_gateway/methods_session.py::_legacy_spawn_tree_entry / spawn_tree.load —
scalar snapshot reads as empty / returns the existing 5000 error instead of
violating the SpawnTreeLoadResult contract.
- hermes_cli/local_runtime/binaries.py::manifest_verified — False.
- plugins/platforms/a2a/protocol.py::load_conversation — non-dict lines are
dropped, keeping the declared list[dict] return.
- batch_runner.py::_load_dataset / _scan_completed_prompts_by_content /
_combine_batch_files — line skipped and counted as filtered.
- trajectory_compressor.py::process_entry_async — scalar entry passed through
unchanged.
Ported from the source hunks of PR #114241; its gateway/shutdown_flush.py
drain_transcript_spool hunk is left to open PR #84785, and its
recover_pending_to_db / cron / bot_live_delivery / bot_mode_dm hunks are
already on this branch or on main.
(cherry picked from commit d4b54568887e69b3ee3d363ebe4dcd657ccf64f9)
