hermes-admin/hermes-agent
1
0
Fork 0
Code Issues Pull Requests Actions 28 Packages Projects Releases Wiki Activity

fix(gateway): resolve handoff/loop-watch scopes off the event loop

Browse Source 
The 15s _loop_wakeup_watcher and the handoff watcher resolved watch scopes (_handoff_watch_scopes -> profiles_to_serve -> get_active_profile_name -> Path.resolve/realpath + profile-dir scans) synchronously ON the loop every pass. On a memory-thrashing host those syscalls stall past the loop-liveness watchdog 10s probe; 3 strikes -> exit 75 -> every in-flight session/cron is killed (mini wedges 24/9 20:58, 26/9 22:56, 27/9 00:21+00:33; [hermes] stack caught in posixpath.realpath). Resolve the scopes through the runner executor hop with the defensive getattr idiom from run_idle_gates.off_loop_gate (bare test stand-ins keep the historical on-loop resolve). 31 targeted tests green.

(cherry picked from commit 52952abc7f033d352fed69ac88fc1efb46110c55)
This commit is contained in:
Emir Saffar
2026-09-27 01:32:26 +02:00
committed by kshitij
parent 4c5b1ae4d0
commit bf668143ad
2 changed files with 17 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
gateway/run_adapters.py
View File

@@ -538,10 +538,16 @@ class GatewayAdapterLifecycleMixin:
with _log_suppressed(logging.DEBUG, "Stale-handoff reclaim failed", exc_info=True):
async with _scope(_phome):
await _reclaim_stale(self)
offload = getattr(self, "_run_in_executor_with_context", None)
try:
while self._running:
try:
for profile_name, profile_home in _handoff_watch_scopes(self):
# Resolve watch scopes OFF the loop (the profiles_to_serve() filesystem walk can
# stall the loop past the liveness probe on a thrashing host — see the sibling
# fix in run_goals._loop_wakeup_watcher). Stand-ins keep the on-loop resolve.
scopes = (await offload(_handoff_watch_scopes, self) if callable(offload)
else _handoff_watch_scopes(self))
for profile_name, profile_home in scopes:
# Idle gate (run_idle_gates): skip the scope entry when the profile's store
# holds no pending handoff. The root poll (None) is unscoped and stays cheap.
if profile_home is not None and not await off_loop_gate(

11
gateway/run_goals.py
View File

@@ -485,9 +485,18 @@ class GatewayGoalsMixin:
for sid, state in active_loops:
await self._loop_wakeup_fire_one(sid, state, now, warned_no_route, profile_name)
offload = getattr(self, "_run_in_executor_with_context", None)
while self._running:
try:
for profile_name, profile_home in _handoff_watch_scopes(self):
# Resolve watch scopes OFF the loop: _handoff_watch_scopes -> profiles_to_serve()
# walks the filesystem (realpath chains + profile-dir scans) every pass; on a
# thrashing host that stalls the loop past the liveness watchdog's 10s probe and
# the gateway self-exits 75 (mini 27/9 wedges: [hermes] caught in posixpath.realpath).
# Stand-in runners without the executor hop keep the historical on-loop resolve
# (same defensive idiom as run_idle_gates.off_loop_gate).
scopes = (await offload(_handoff_watch_scopes, self) if callable(offload)
else _handoff_watch_scopes(self))
for profile_name, profile_home in scopes:
# Idle gate (run_idle_gates): skip the scope entry when the profile's store holds
# no active loop. The root scan (None) is unscoped and stays cheap.
if profile_home is not None and not await self._run_in_executor_with_context(