Two concurrent Hermes turns in one session push two task scopes onto the same
physical stack. _finish_task popped its handle unconditionally, so the first turn
to finish hit the native "scope handle is not at the top of the stack" and logged a
full traceback on every overlap (#115471). pop_relay_scope_if_top compares the
handle to the current top inside the task context and skips the pop when a
sibling scope sits above it; the orphan is reclaimed by the existing session-close
drain in _close_scope_handle. Draining the sibling from the task close (as the
candidate PR did) would pop a LIVE turn scope, and guarding pop_relay_scope
itself would break _pop_with_drain, which relies on that raise.
Co-authored-by: Mohamad Kanso <91088196+MohamadKanso@users.noreply.github.com>
_open_session_scope wrapped both executor.submit() and future.result()
in one try/except RuntimeError. The except branch exists for executor
refusal during interpreter shutdown, but result() re-raises whatever the
worker raised, so a RuntimeError from relay.scope.push itself was
misclassified as refusal and the exit_fallback lane ran the same push a
second time.
Keep submit() alone in the try and move result() to the else branch:
executor refusal still pushes synchronously once via exit_fallback,
while a push-side RuntimeError propagates after a single attempt. This
mirrors the split run_in_session already uses, where result() only
catches FuturesTimeoutError.
Tests pin all three arms: push RuntimeError propagates after one call
with no half-populated session state, submit() refusal still uses the
sync fallback once, and the coordinator end-to-end degrades to an
uninstrumented lease then recovers on the next acquire.
3fad83df31 (Aug 11) moved Relay exporter config to a plugins.toml
selected by HERMES_NEMO_RELAY_PLUGINS_TOML. A .env still carrying the legacy
exporter vars and no TOML logs ONE warning and initialises no exporters, so
users who followed the earlier docs lost every trace silently (the
maintainer's stopped Aug 20, noticed Sep 14; five multiplexed profiles on
the same box carry the same eight vars today).
- `hermes_cli/relay_plugin_migrate.py`: build the document from the
`nemo_relay.observability` dataclasses (`ComponentSpec(...).to_dict()`,
so the `type = "file"` sink discriminator is emitted), validate it by
activating it through `nemo_relay.plugin.initialize` + `clear_async`,
write `<home>/relay-plugins.toml` (tomli_w when installed, minimal emitter
otherwise), set HERMES_NEMO_RELAY_PLUGINS_TOML in that .env, and comment
the legacy lines out (never delete). Defaults mirror the removed plugin so
files land where they used to.
- `hermes update` runs it for the default home AND every live named profile
(each writes its own TOML) as a best-effort post-update step, with a loud
notice; `hermes migrate relay [--all-profiles] [--no-validate]` runs it on
demand.
- The runtime WARNING and the `hermes doctor` finding now say "NO traces
are being exported" and name the exact command and file path.
- Docs: environment-variables.md + built-in-plugins.md carry the migration
note and a complete plugins.toml example including `type = "file"`.
The Sep 2026 decomposition (PR #102117) makes internal import paths a non-API: names now live in
the focused modules that define them. This commit is the ONLY thing keeping the old paths alive,
so external plugins have time to update. It is deliberately a single, unsquashed commit:
git revert <this sha>
removes every shim, stub and manifest at once on the announced date. Nothing in-tree may depend on
these pointers: scripts/check_compat_pointers.py (wired into lint.yml) fails CI if it does.
What it adds (see COMPAT_MANIFEST.md, compat_manifest.json):
- 332 facade modules get one delimited `PLUGIN-COMPAT` block appended at the end of the file
- 1,172 moved names resolved lazily via a module `__getattr__` (PEP 562) — never a top-level import,
so no import cycles; facades that already had `__getattr__` get a chained one
- 592 third-party/stdlib names the old modules used to expose, with their original import statements
- 266 public definitions that had been deleted as unused, restored byte-for-byte from the pre-decomposition
tree (+40 private helpers and 16 imports pulled in only because a restored definition needs them)
- 3 deleted modules recreated as re-export stubs (gateway/startup_watchdog, hermes_cli/observability/
relay_runtime, tools/environments/modal_utils)
- private names (`_x`) get no pointer: they were never API (3,792 skipped)
Verified: all 335 touched modules import under a fresh HERMES_HOME and every manifest name resolves;
the lint reports zero in-tree uses; ruff clean; targeted suites unchanged.
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
Older nemo-relay bindings reject metadata= on scope.pop, which aborted
turn finalization and left scopes open. Filter kwargs to what the live
binding accepts so close paths can complete.
The salvaged _close_scope_handle replaced direct scope.pop calls that
were bounded by _SCOPE_OP_TIMEOUT with an unbounded run_in_session
callback, regressing the bounded-finalization contract (CI:
tests/agent/test_relay_runtime_bounded_scope_ops.py — end_turn /
close_session / finish_logical_calls hung when the native pop wedged).
Pass timeout=_SCOPE_OP_TIMEOUT so the whole drain+close costs at most
one span and never blocks turn or session completion.
Follow-up to HexLab98's salvaged commits:
- Apply the same bounded worker join before raising InterruptedError at
the two sibling interrupt sites that share the raise-without-join
shape: the non-streaming API poll loop and the Bedrock streaming poll
loop. Both workers run Relay-managed physical attempts, so raising
immediately allowed turn teardown to race a still-open physical scope
exactly as in the streaming path.
- Address the #81601 review finding (egilewski): the pinned nemo-relay
binding's get_scope_stack() returns a native ScopeStack object which
scope.pop rejects with TypeError, so the orphan drain never drained
under the real binding. current_top() now prefers the version-correct
scope.get_handle() accessor and falls back to the old list-unwrap for
fake/legacy shapes. Handle comparisons go through same_handle(),
comparing by uuid, because native ScopeHandle instances do not
implement value equality.
- Add a real-binding regression test that reproduces the orphaned-scope
session close against the pinned native wheel (skips where the native
binding is unavailable), alongside the existing fake-based coverage.
Empty-stream stalls that trip interrupt were raising InterruptedError
before the stream worker closed its physical LLM scope, corrupting the
Relay LIFO stack and cascading into a CLI EIO redraw storm (#81521).
notify_session_compacted closed the old session scope immediately on a
legacy rotating compaction. A compaction can complete while a turn is
still live on the old session; closing then pops the session scope under
the live turn scope, violating the stack's LIFO order — the exact
invariant the rest of the segmentation feature protects.
Now: when the old session has an active turn, set close_pending instead;
that turn's end_turn consumes the flag after its own turn scope pops and
it unregisters from the active-turn table. Sabotage-verified: the new
test fails without the fix.
