fix(desktop): latch an expired remote OAuth session on the first confirmed 401 instead of flickering the Sign in overlay (#95701)

Cold-launching against a gated remote gateway whose stored session has been
invalidated server-side alternated between the connecting state and the
recovery overlay; the Sign in button was only intermittently clickable.

Root cause: fetchJson() built a bare Error("401: ...") for HTTP failures on
the native-bearer path, dropping res.statusCode. Every downstream classifier
is shape-based (isGatewayAuthRejection, isServerSideHttpError, the
ensureNativeAccessToken 401 check), so the confirmed rejection looked like a
transport blip: withTransientRetries hammered it, gatewayTicketFailure used
the transport copy, startHermes tagged the boot retryable, and the renderer's
bounded boot-retry loop re-emitted running:true over the overlay on every
attempt. Separately, gatewayTicketFailure only ever set needsOauthLogin,
which isReauthRequiredError ignores, so even a structured 401 from the cookie
path never latched.

- api-transport: httpStatusError() is the one HTTP-error shape; fetchJson and
  fetchPublicJson use it, matching fetchJsonViaOauthSession.
- mintGatewayWsTicket: the gateway never rotates a native bearer server-side
  (dashboard_auth/middleware.py), so a bearer 401 gets ONE forced
  /auth/native/refresh; a live refresh token retries the mint once, a dead one
  drops the stored tokens and the rejection is confirmed.
- gatewayTicketFailure: a confirmed 401/403 is tagged isReauthRequired so
  startHermes latches it and marks the boot non-retryable.
- startHermes: latches are set before the first await in the failure path;
  updateBootProgress holds every update that is not a re-emit of the latched
  failure until a recovery path clears the latch.

Tests: composition + main.ts source pins (remote-reauth-latch.test.ts), unit
coverage for the new helpers, and a Playwright e2e that boots the real app
against a fake gateway with a dead session and proves the overlay latches
once (one mint, one refresh, retryable:false, Sign in stays clickable).
This commit is contained in:
Zeus-Deus
2026-08-26 18:43:38 +02:00
committed by kshitij
parent b7d353fef1
commit 8d22781b1c
13 changed files with 822 additions and 28 deletions

View File

@@ -0,0 +1,285 @@
/**
* E2E regression for issue #95701: a remote OAuth gateway whose stored
* session has been invalidated server-side must boot into ONE stable
* recovery overlay with a clickable Sign in action — not alternate between
* the connecting state and the overlay while the renderer's transient-boot
* retry loop re-drives a rejection that can never self-heal.
*
* The shape under test is the RFC 8252 native-bearer flow: the desktop holds
* a locally-unexpired access token, but the gateway rejects it (restarted
* with new signing state), and the refresh token is dead too. The gateway
* never rotates a native bearer server-side (see
* hermes_cli/dashboard_auth/middleware.py), so the desktop must try ONE
* rotation via /auth/native/refresh and then treat the rejection as
* confirmed: latch reauth, mark the boot non-retryable, hold the overlay.
*
* Contract asserted here, end to end through the real Electron main process
* and renderer:
*
* 1. The recovery overlay appears and then stays continuously visible —
* no sample over a window longer than the renderer's boot-retry backoff
* sees it hidden.
* 2. The gateway saw exactly one ticket mint and exactly one refresh
* attempt: the rejection was confirmed once, not hammered.
* 3. The boot-progress snapshot the renderer reads is the non-retryable,
* structured-401 verdict the reauth latch depends on.
* 4. The overlay offers the remote sign-in action (dead tokens were dropped,
* so Settings no longer reports the session as connected).
*
* Prerequisite: `npm run build` must have been run so dist/ exists.
*/
import * as fs from 'node:fs'
import * as http from 'node:http'
import type { AddressInfo } from 'node:net'
import * as path from 'node:path'
import { buildAppEnv, createSandbox, launchDesktop, type Sandbox, waitForBootFailure } from './fixtures'
import { allowErrorBanners, type ElectronApplication, expect, type Page, test } from './test'
// Longer than the renderer's first boot-retry delay (2s base, full jitter,
// #82679) by a wide margin, so a retry loop that hides the overlay cannot
// slip between two samples.
const STABILITY_WINDOW_MS = 15_000
const SAMPLE_INTERVAL_MS = 100
interface RecordedRequest {
method: string
path: string
bearer: boolean
}
interface ExpiredSessionGateway {
url: string
requests: RecordedRequest[]
count: (method: string, pathname: string) => number
close: () => Promise<void>
}
/**
* A gated gateway whose every authenticated session is dead. Public
* discovery works (status advertises the native flow, providers are
* OAuth-style so the desktop keeps the strict guard), but the ticket mint
* and the native refresh both answer the middleware's structured
* `session_expired` 401. Every request is recorded so the test can prove the
* rejection was confirmed exactly once.
*/
async function startExpiredSessionGateway(): Promise<ExpiredSessionGateway> {
const requests: RecordedRequest[] = []
const server = http.createServer((req, res) => {
const url = new URL(req.url ?? '/', 'http://127.0.0.1')
const authorization = req.headers.authorization
requests.push({
method: req.method ?? 'GET',
path: url.pathname,
bearer: typeof authorization === 'string' && authorization.startsWith('Bearer '),
})
const json = (status: number, body: unknown): void => {
res.writeHead(status, { 'Content-Type': 'application/json' })
res.end(JSON.stringify(body))
}
// Drain the body before answering so a POST never sees an early close.
req.resume()
req.on('end', () => {
if (url.pathname === '/api/status') {
json(200, { auth_required: true, auth_flows: ['native_pkce'], ok: true, version: '0.0.0-e2e-fake' })
return
}
if (url.pathname === '/api/auth/providers') {
json(200, { providers: [{ name: 'portal', supports_password: false }] })
return
}
if (url.pathname === '/api/auth/ws-ticket' || url.pathname === '/auth/native/refresh') {
json(401, {
error: 'session_expired',
detail: 'Unauthorized',
reason: 'invalid_or_expired_session',
login_url: '/login',
})
return
}
if (url.pathname.startsWith('/api/')) {
json(401, { error: 'unauthenticated', detail: 'Unauthorized', reason: 'no_cookie', login_url: '/login' })
return
}
json(404, { detail: 'not found' })
})
})
// No WebSocket can ever be authenticated here; refuse the upgrade fast.
server.on('upgrade', (_req, socket) => {
socket.destroy()
})
await new Promise<void>(resolve => server.listen(0, '127.0.0.1', resolve))
const { port } = server.address() as AddressInfo
return {
close: () =>
new Promise<void>(resolve => {
server.closeAllConnections?.()
server.close(() => resolve())
}),
count: (method, pathname) => requests.filter(r => r.method === method && r.path === pathname).length,
requests,
url: `http://127.0.0.1:${port}`,
}
}
/**
* Seed the sandbox the way a previously signed-in desktop leaves it: a saved
* remote OAuth connection and a stored native token set whose access token
* has NOT expired locally (so the desktop presents it as-is). The blob uses
* the plain encoding the desktop's secret reader accepts verbatim — an OS
* keyring is not available on the CI runner and is irrelevant to the contract.
*/
function seedExpiredNativeSession(sandbox: Sandbox, gatewayUrl: string): void {
fs.writeFileSync(
path.join(sandbox.userDataDir, 'connection.json'),
JSON.stringify({ mode: 'remote', remote: { url: gatewayUrl, authMode: 'oauth' }, profiles: {} }, null, 2),
{ encoding: 'utf8', mode: 0o600 },
)
const tokens = {
accessToken: 'e2e-stale-access-token',
refreshToken: 'e2e-stale-refresh-token',
expiresAt: Math.floor(Date.now() / 1000) + 3600,
provider: 'portal',
userId: 'e2e-user',
}
fs.writeFileSync(
path.join(sandbox.userDataDir, 'native-oauth-tokens.json'),
JSON.stringify({ [gatewayUrl]: { encoding: 'plain', value: JSON.stringify(tokens) } }, null, 2),
{ encoding: 'utf8', mode: 0o600 },
)
}
/**
* Sample the recovery overlay inside the page at a fixed cadence and report
* every sample in which it was NOT showing. The overlay is identified by its
* recovery actions, the same signal waitForBootFailure keys on; while the
* renderer re-drives a boot the overlay is unmounted (`running: true`) and
* the connecting surface paints instead.
*/
async function sampleOverlayStability(
page: Page,
windowMs: number,
intervalMs: number,
): Promise<{ samples: number; hidden: number; flips: number }> {
return page.evaluate(
async ({ windowMs: w, intervalMs: i }) => {
const overlayShowing = (): boolean => {
const text = document.body.textContent ?? ''
return (
text.includes('Sign out & sign in') ||
text.includes('Gateway settings') ||
text.includes('Use local gateway') ||
text.includes('Repair install')
)
}
const started = Date.now()
let samples = 0
let hidden = 0
let flips = 0
let last = overlayShowing()
while (Date.now() - started < w) {
await new Promise(resolve => setTimeout(resolve, i))
const showing = overlayShowing()
samples += 1
if (!showing) {
hidden += 1
}
if (showing !== last) {
flips += 1
last = showing
}
}
return { samples, hidden, flips }
},
{ windowMs, intervalMs },
)
}
let gateway: ExpiredSessionGateway | null = null
let sandbox: Sandbox | null = null
let app: ElectronApplication | null = null
test.afterAll(async () => {
await app?.close().catch(() => undefined)
await gateway?.close()
sandbox?.cleanup()
app = null
gateway = null
sandbox = null
})
test.describe('remote OAuth session rejected at cold boot (#95701)', () => {
test.beforeEach(() => {
// The boot deliberately fails; the "Desktop boot failed" toast is expected.
allowErrorBanners()
})
test('the recovery overlay latches once and its Sign in action stays put', async () => {
gateway = await startExpiredSessionGateway()
sandbox = createSandbox('reauth-latch')
seedExpiredNativeSession(sandbox, gateway.url)
const launched = await launchDesktop(buildAppEnv(sandbox))
app = launched.app
const page = launched.page
await waitForBootFailure(page, 60_000)
// 1. Stability: once the overlay is up it never blinks out again. A
// transient-boot retry (`running: true` re-emitted over the failure)
// unmounts it, which is exactly the flicker the issue describes.
const stability = await sampleOverlayStability(page, STABILITY_WINDOW_MS, SAMPLE_INTERVAL_MS)
expect(stability.samples).toBeGreaterThan(50)
expect(stability, 'recovery overlay must stay visible for the whole window').toMatchObject({ hidden: 0, flips: 0 })
// 2. The rejection was confirmed exactly once: one bearer ticket mint,
// one forced rotation attempt, and no further boot re-entry hammering
// the dead session for the rest of the window.
const ticketMints = gateway.requests.filter(r => r.method === 'POST' && r.path === '/api/auth/ws-ticket')
expect(ticketMints, 'exactly one ws-ticket mint').toHaveLength(1)
expect(ticketMints[0]?.bearer, 'the mint presented the stored native bearer').toBe(true)
expect(gateway.count('POST', '/auth/native/refresh'), 'exactly one native refresh attempt').toBe(1)
// 3. The renderer-visible verdict is the structured, non-retryable one.
const snapshot = await page.evaluate(() =>
(window as unknown as { hermesDesktop: { getBootProgress: () => Promise<Record<string, unknown>> } })
.hermesDesktop.getBootProgress(),
)
expect(snapshot).toMatchObject({ running: false, retryable: false, statusCode: 401 })
// A session that DID exist reads as expired, not as never signed in — the
// dead tokens are dropped on the way out, so this copy must be chosen
// from the pre-mint state.
expect(String(snapshot.error)).toMatch(/session has expired/i)
// 4. The overlay offers remote sign-in — the dead tokens were dropped, so
// the connection no longer reads as "connected" and the reauth branch
// renders instead of the local Retry/Repair buttons.
await expect(page.getByRole('button', { name: /sign out & sign in/i })).toBeVisible()
await expect(page.getByRole('button', { name: /sign out & sign in/i })).toBeEnabled()
})
})

View File

@@ -19,6 +19,7 @@ import { afterAll, describe, expect, it } from 'vitest'
import {
destroyKeepaliveAgents,
downloadAgentFor,
httpStatusError,
isIdempotentMethod,
isTransientTransportError,
jsonAgentFor,
@@ -381,3 +382,25 @@ describe('live: POST reset after server-side processing', () => {
}
}, 20_000)
})
describe('httpStatusError', () => {
it('carries the integer statusCode downstream classifiers read, plus the legacy "<status>: <body>" message', () => {
const err = httpStatusError(401, '{"error":"session_expired"}', 'Unauthorized')
expect(err).toBeInstanceOf(Error)
expect(err.statusCode).toBe(401)
expect(err.message).toBe('401: {"error":"session_expired"}')
})
it('falls back to the status message, then to an empty detail, when the body is empty', () => {
expect(httpStatusError(503, '', 'Service Unavailable').message).toBe('503: Service Unavailable')
expect(httpStatusError(403, '', undefined).message).toBe('403: ')
expect(httpStatusError(403, '', undefined).statusCode).toBe(403)
})
it('normalizes a missing/invalid status to 500 exactly like the `res.statusCode || 500` guard', () => {
expect(httpStatusError(undefined, 'boom').statusCode).toBe(500)
expect(httpStatusError(undefined, 'boom').message).toBe('500: boom')
expect(httpStatusError(0, 'boom').statusCode).toBe(500)
})
})

View File

@@ -167,9 +167,32 @@ async function withRetry(makeAttempt, options: any = {}) {
throw lastError
}
/**
* The one error shape the REST helpers throw for an HTTP >= 400 response.
*
* `statusCode` is the structured contract every downstream classifier reads —
* isGatewayAuthRejection (401/403 → reauth, never retried), isServerSideHttpError
* (502/503/504 → Cloud-down), ensureNativeAccessToken's dead-refresh-token
* check — and the "<status>: <body>" message keeps the legacy prefix readers
* working. fetchJson used to build a bare Error here, so a native-bearer 401
* reached the boot path as an anonymous transport failure: it was retried,
* then classified as transient, and the renderer's boot-retry loop flickered
* the Sign in overlay away (#95701). Shared by fetchJson, fetchPublicJson and
* the OAuth-session fetch so the three paths cannot drift apart again.
*/
function httpStatusError(statusCode, text, statusMessage?) {
const status = Number.isInteger(statusCode) && statusCode > 0 ? statusCode : 500
const detail = String(text || statusMessage || '')
const error: any = new Error(`${status}: ${detail}`)
error.statusCode = status
return error
}
export {
destroyKeepaliveAgents,
downloadAgentFor,
httpStatusError,
isIdempotentMethod,
isTransientTransportError,
jsonAgentFor,

View File

@@ -220,10 +220,13 @@ test('a credentialed 401 fails fast for reauth instead of reporting a dead sessi
assert.deepEqual(calls, [['probe', 'https://gateway.example/api/health']])
})
test('unsigned OAuth is a terminal reauth failure; needsOauthLogin alone is not', () => {
test('unsigned OAuth is a terminal reauth failure; a bare needsOauthLogin hint is not', () => {
// The unsigned-in throw must set isReauthRequired so startHermes latches.
// needsOauthLogin alone (ticket 401/403) stays a Sign-in hint, not a latch —
// a lapsed AT cookie can still rotate from a live RT on the next mint.
// A bare needsOauthLogin (the IPC-shaped hint) stays Sign-in copy, not a
// latch. A CONFIRMED ticket 401/403 is different: the gateway has already
// tried the AT/RT rotation (cookie) or the desktop has forced one (native)
// before that rejection reaches gatewayTicketFailure, which tags it
// isReauthRequired itself (#95701).
const unsigned = makeUnsignedOauthError() as any
assert.equal(unsigned.needsOauthLogin, true)

View File

@@ -206,8 +206,10 @@ export function makeReauthRequiredError(detail?: string): Error {
/**
* No native token and no live cookie: boot cannot self-heal. Must carry
* `isReauthRequired` so startHermes latches; `needsOauthLogin` alone only
* drives Sign in copy and would retry after #88070, hiding the overlay.
* `isReauthRequired` so startHermes latches; a bare `needsOauthLogin` (the
* IPC-shaped hint) only drives Sign in copy and would retry after #88070,
* hiding the overlay. A confirmed ticket-mint 401/403 carries the same tag
* (see gatewayTicketFailure, #95701).
*/
export function makeUnsignedOauthError(): Error {
const error = new Error(REMOTE_UNSIGNED_OAUTH_MESSAGE) as any

View File

@@ -6,6 +6,7 @@ import { isReauthRequiredError, makeUnsignedOauthError } from './backend-health'
import {
isHostKeyChangedBootFailure,
isRetryableRemoteBootFailure,
shouldHoldBootProgressForReauth,
shouldLatchBackendStartFailure,
shouldLatchHostKeyChangedFailure,
shouldLatchRemoteReauthFailure
@@ -70,8 +71,11 @@ test('a CONFIRMED reauth rejection is never auto-retried (missing capability, no
assert.equal(isRetryableRemoteBootFailure({ attemptedRemote: true, isReauth: true }), false)
})
test('unsigned OAuth latches and is never auto-retried; needsOauthLogin alone still retries', () => {
test('unsigned OAuth latches and is never auto-retried; a bare needsOauthLogin hint still retries', () => {
// Production composition in startHermes: isReauth = isReauthRequiredError(error).
// A bare `{ needsOauthLogin: true }` is the IPC-shaped hint, not a confirmed
// rejection; gatewayTicketFailure tags a confirmed 401/403 with
// isReauthRequired itself (#95701, see remote-reauth-latch.test.ts).
const unsigned = isReauthRequiredError(makeUnsignedOauthError())
const ticketHint = isReauthRequiredError({ needsOauthLogin: true })
@@ -147,3 +151,25 @@ test('every remote failure picks exactly one path: retry, reauth latch, or host-
}
}
})
test('FIX #95701: while a reauth rejection is latched, only re-emits of that failure reach the renderer', () => {
const latched = 'Your remote gateway session has expired. Sign in again.'
// The latched failure's own (re-)emit passes — it carries retryable:false.
assert.equal(shouldHoldBootProgressForReauth(latched, { error: latched }), false)
// Anything that would lift the overlay is held: a running phase from an
// attempt already in flight when the latch closed, a cleared error, or a
// sibling failure that would flip retryable back on.
assert.equal(shouldHoldBootProgressForReauth(latched, { error: null }), true)
assert.equal(shouldHoldBootProgressForReauth(latched, {}), true)
assert.equal(shouldHoldBootProgressForReauth(latched, { error: 'Could not reach the remote Hermes gateway' }), true)
})
test('FIX #95701: with no reauth latch every boot-progress update flows as before', () => {
for (const latch of [null, undefined, '']) {
assert.equal(shouldHoldBootProgressForReauth(latch, { error: null }), false)
assert.equal(shouldHoldBootProgressForReauth(latch, {}), false)
assert.equal(shouldHoldBootProgressForReauth(latch, { error: 'Desktop boot failed: spawn ENOENT' }), false)
}
})

View File

@@ -138,3 +138,35 @@ export function shouldLatchHostKeyChangedFailure(context: RemoteBootRetryContext
export function isRetryableRemoteBootFailure(context: RemoteBootRetryContext): boolean {
return context.attemptedRemote && !context.isReauth && context.isHostKeyChanged !== true
}
export interface BootProgressUpdateLike {
/** The failure text an error update carries; absent/null on progress updates. */
error?: unknown
}
/**
* Whether a boot-progress update must be dropped because a CONFIRMED remote
* reauth rejection is latched.
*
* `latchedMessage` is the message of the latched reauth failure (null when
* nothing is latched). While it is set, the recovery overlay with its Sign in
* button is the only correct surface until the user signs in, and any update
* that is not a re-emit of that same failure would lift it: a `running: true`
* phase or a cleared error from an attempt that was already in flight when the
* latch closed, or an unrelated failure from a sibling caller (which would
* also swap the retryable verdict back to true and re-arm the renderer's
* boot-retry loop — the flicker of #95701). Re-emits of the latched failure
* pass so the non-retryable verdict is never lost. Every recovery path
* (reset, repair, apply-config, confirmed sign-in) clears the latch BEFORE it
* re-drives boot, so this never holds a legitimate restart.
*/
export function shouldHoldBootProgressForReauth(
latchedMessage: string | null | undefined,
update: BootProgressUpdateLike
): boolean {
if (!latchedMessage) {
return false
}
return update.error !== latchedMessage
}

View File

@@ -1342,3 +1342,34 @@ test('OAuth ticket-mint 401 stays on the reauth path (never Cloud-down)', () =>
assert.equal((wrapped as any).needsOauthLogin, true)
assert.equal((wrapped as any).statusCode, 401)
})
test('FIX #95701: a confirmed 401/403 ticket rejection is tagged isReauthRequired so startHermes latches it', () => {
for (const statusCode of [401, 403]) {
const source = Object.assign(new Error(`${statusCode}: rejected`), { statusCode })
const wrapped = gatewayTicketFailure(source, 'auth copy', 'transport copy') as any
assert.equal(wrapped.message, 'auth copy')
assert.equal(wrapped.needsOauthLogin, true)
assert.equal(wrapped.isReauthRequired, true, `a ${statusCode} mint rejection cannot self-heal`)
assert.equal(wrapped.statusCode, statusCode)
}
// A pre-tagged rejection (needsOauthLogin from an upstream classifier) is
// confirmed the same way.
const tagged = gatewayTicketFailure({ needsOauthLogin: true }, 'auth copy', 'transport copy') as any
assert.equal(tagged.isReauthRequired, true)
})
test('FIX #95701: transport and server failures at the ticket mint stay retryable — never reauth', () => {
for (const source of [
Object.assign(new Error('503: unavailable'), { statusCode: 503 }),
new Error('Timed out connecting to Hermes backend after 8000ms'),
Object.assign(new Error('read ECONNRESET'), { code: 'ECONNRESET' })
]) {
const wrapped = gatewayTicketFailure(source, 'auth copy', 'transport copy') as any
assert.equal(wrapped.message, 'transport copy')
assert.equal(wrapped.needsOauthLogin, undefined)
assert.equal(wrapped.isReauthRequired, undefined)
}
})

View File

@@ -132,6 +132,15 @@ function gatewayTicketFailure(error, authMessage, transportMessage) {
if (needsOauthLogin) {
;(err as any).needsOauthLogin = true
// A rejected ticket mint is a CONFIRMED reauth failure, not a hint. The
// cookie path only sees a 401/403 after the gateway's transparent AT/RT
// rotation has already failed, and the native-bearer path only after
// mintGatewayWsTicket's forced /auth/native/refresh has. Nothing will
// change until the user signs in, so tag it the way startHermes latches
// (isReauthRequiredError): the boot is marked non-retryable and the
// overlay's Sign in button stops flickering away under the renderer's
// transient-boot retry loop (#95701).
;(err as any).isReauthRequired = true
}
// Preserve structured HTTP context when the source error carried an integer

View File

@@ -30,7 +30,7 @@ import {
} from 'electron'
import { classifyActiveRuntime } from './active-runtime-state'
import { destroyKeepaliveAgents, downloadAgentFor, jsonAgentFor, withRetry } from './api-transport'
import { destroyKeepaliveAgents, downloadAgentFor, httpStatusError, jsonAgentFor, withRetry } from './api-transport'
import { appIconCandidates, resolveAppIcon } from './app-icon'
import { stopBackendChild as stopBackendChildImpl, stopBackendTreesForUpdate } from './backend-child'
import {
@@ -68,6 +68,7 @@ import { isPidAliveWindows, waitForBackendRelease } from './backend-release-gate
import {
isHostKeyChangedBootFailure,
isRetryableRemoteBootFailure,
shouldHoldBootProgressForReauth,
shouldLatchBackendStartFailure,
shouldLatchHostKeyChangedFailure,
shouldLatchRemoteReauthFailure
@@ -259,7 +260,8 @@ import {
resolveGatedDownloadAuth,
resolveJsonBody,
resolveOauthRestAuth,
resolveReadinessProbeAuth
resolveReadinessProbeAuth,
shouldRotateNativeTokenAfterRejection
} from './native-auth-decisions'
import {
nativeRefreshUrl,
@@ -2237,7 +2239,28 @@ function abandonFirstRunSetupChoiceForRemoteApply() {
return resumedGatedConnection
}
// The latched reauth failure whose hold has already been logged, so a burst of
// dropped updates from one in-flight sibling attempt logs once, not per event.
let bootProgressHeldFor = null
function updateBootProgress(update, options: { allowDecrease?: boolean } = {}) {
// A latched CONFIRMED reauth rejection owns the boot surface until a
// recovery path clears it. Updates that are not a re-emit of that failure —
// a running:true phase or cleared error from an attempt already in flight
// when the latch closed, or an unrelated sibling failure that would flip
// retryable back on — must not reach the renderer, or the overlay's Sign in
// button flickers away again (#95701).
if (shouldHoldBootProgressForReauth(remoteReauthFailure ? remoteReauthFailure.message : null, update)) {
if (bootProgressHeldFor !== remoteReauthFailure) {
bootProgressHeldFor = remoteReauthFailure
rememberLog('[boot] remote reauth latched: holding the recovery overlay against a stale boot-progress update')
}
return
}
bootProgressHeldFor = null
const nextProgressRaw =
typeof update.progress === 'number' ? clampBootProgress(update.progress) : bootProgressState.progress
@@ -5374,7 +5397,7 @@ function fetchJson(url, token, options: any = {}) {
const text = Buffer.concat(chunks).toString('utf8')
if ((res.statusCode || 500) >= 400) {
reject(new Error(`${res.statusCode}: ${text || res.statusMessage}`))
reject(httpStatusError(res.statusCode, text, res.statusMessage))
return
}
@@ -5540,7 +5563,7 @@ function fetchPublicJson(url, options: any = {}) {
const text = Buffer.concat(chunks).toString('utf8')
if ((res.statusCode || 500) >= 400) {
reject(new Error(`${res.statusCode}: ${text || res.statusMessage}`))
reject(httpStatusError(res.statusCode, text, res.statusMessage))
return
}
@@ -7949,14 +7972,20 @@ function postJsonNoAuth(url: string, body: unknown, opts: any = {}) {
// Return a valid native access token for baseUrl, refreshing via
// /auth/native/refresh if the stored one is at/near expiry. Returns null when
// there are no tokens or the refresh is terminally rejected (caller re-logins).
async function ensureNativeAccessToken(baseUrl: string): Promise<string | null> {
// `forceRefresh` rotates even a locally-unexpired access token: the gateway
// never rotates a native bearer server-side, so after the gate rejects one
// the desktop must run the refresh itself before the rejection is confirmed.
async function ensureNativeAccessToken(
baseUrl: string,
options: { forceRefresh?: boolean } = {}
): Promise<string | null> {
const tokens = _loadNativeTokens(baseUrl)
if (!tokens) {
return null
}
if (!tokenNeedsRefresh(tokens, Math.floor(Date.now() / 1000))) {
if (!options.forceRefresh && !tokenNeedsRefresh(tokens, Math.floor(Date.now() / 1000))) {
return tokens.accessToken
}
@@ -8286,20 +8315,29 @@ async function mintGatewayWsTicket(baseUrl, headers = {}) {
const nativeAt = await ensureNativeAccessToken(baseUrl).catch(() => null)
if (nativeAt) {
const body = (await fetchJson(`${baseUrl}/api/auth/ws-ticket`, null, {
method: 'POST',
timeoutMs: 8_000,
bearer: nativeAt,
headers
})) as any
try {
return await mintGatewayWsTicketWithBearer(baseUrl, nativeAt, headers)
} catch (error) {
// The gate rejected a bearer the desktop still considered valid. It
// never rotates a native access token server-side (only cookie
// sessions get the transparent refresh; the native flow is told to
// call /auth/native/refresh itself), so run ONE forced rotation here:
// a live refresh token yields a fresh bearer and the mint is retried
// once; a dead one drops the stored set, and the original 401 stands
// as a CONFIRMED rejection that latches into the Sign in overlay
// instead of being replayed by every boot retry (#95701).
if (!shouldRotateNativeTokenAfterRejection(error)) {
throw error
}
const ticket = body?.ticket
const rotatedAt = await ensureNativeAccessToken(baseUrl, { forceRefresh: true }).catch(() => null)
if (!ticket || typeof ticket !== 'string') {
throw new Error('Gateway did not return a WS ticket.')
if (!rotatedAt || rotatedAt === nativeAt) {
throw error
}
return await mintGatewayWsTicketWithBearer(baseUrl, rotatedAt, headers)
}
return ticket
}
const body = (await fetchJsonViaOauthSession(`${baseUrl}/api/auth/ws-ticket`, {
@@ -8318,6 +8356,26 @@ async function mintGatewayWsTicket(baseUrl, headers = {}) {
})
}
// One bearer-authenticated ticket mint. Kept separate from the rotation
// decision in mintGatewayWsTicket so the retry-after-refresh leg presents the
// rotated bearer through exactly the same request shape as the first attempt.
async function mintGatewayWsTicketWithBearer(baseUrl, bearer, headers = {}) {
const body = (await fetchJson(`${baseUrl}/api/auth/ws-ticket`, null, {
method: 'POST',
timeoutMs: 8_000,
bearer,
headers
})) as any
const ticket = body?.ticket
if (!ticket || typeof ticket !== 'string') {
throw new Error('Gateway did not return a WS ticket.')
}
return ticket
}
// Build a fresh WS URL for the *current* connection. Critical for reconnects:
// OAuth WS tickets are single-use with a ~30s TTL, so the ticket baked into
// the cached connection's wsUrl is stale on the second connect. The renderer
@@ -10086,6 +10144,12 @@ async function buildRemoteConnection(
throw makeUnsignedOauthError()
}
// Snapshot BEFORE the mint: a confirmed native rejection drops the dead
// token set on its way out (mintGatewayWsTicket's forced rotation), and
// the failure copy must still say "session expired" for a session that
// did exist — "not signed in" is for a jar that never held one.
const hadNativeSession = hasNativeSession(baseUrl)
let ticket
try {
@@ -10106,7 +10170,7 @@ async function buildRemoteConnection(
throw gatewayTicketFailure(
error,
oauthTicketFailureAuthMessage(hasNativeSession(baseUrl)),
oauthTicketFailureAuthMessage(hadNativeSession),
'Could not reach the remote Hermes gateway while refreshing its WebSocket ticket. Try reconnecting.'
)
}
@@ -13287,14 +13351,15 @@ async function startHermes() {
const failedProcess = backendConnectionState.invalidate()
stopBackendChild(failedProcess)
await waitForBackendExit(failedProcess)
if (error instanceof FirstRunSetupResetError) {
await waitForBackendExit(failedProcess)
throw error
}
const message = error instanceof Error ? error.message : String(error)
const hostKeyChanged = isHostKeyChangedBootFailure(error)
const isReauth = isReauthRequiredError(error)
// Carry structured Cloud-down metadata through the boot-progress / IPC
// boundary when present, so the renderer overlay can key on it rather than
@@ -13329,10 +13394,19 @@ async function startHermes() {
// A confirmed reauth rejection latches separately: it can't self-heal, and
// leaving it unlatched hides the overlay's "Sign in" button on every retry.
if (shouldLatchRemoteReauthFailure({ attemptedRemote, isReauth: isReauthRequiredError(error) })) {
if (shouldLatchRemoteReauthFailure({ attemptedRemote, isReauth })) {
remoteReauthFailure = error instanceof Error ? error : new Error(message)
}
// Every latch above is set BEFORE this first yield back to the event loop.
// invalidate() already dropped the shared attempt promise, so a concurrent
// getConnection()/startHermes() caller arriving during the exit wait would
// otherwise start a brand-new attempt, re-emit running:true over the
// failure and re-drive the identical rejection. With the latch in place it
// short-circuits on the cached failure instead: the first confirmed
// rejection owns the transition into recovery (#95701).
await waitForBackendExit(failedProcess)
updateBootProgress(
{
error: message,
@@ -13347,7 +13421,7 @@ async function startHermes() {
// sign-in affordance.
retryable: isRetryableRemoteBootFailure({
attemptedRemote,
isReauth: isReauthRequiredError(error),
isReauth,
isHostKeyChanged: hostKeyChanged
}),
running: false,

View File

@@ -18,7 +18,8 @@ import {
resolveGatedDownloadAuth,
resolveJsonBody,
resolveOauthRestAuth,
resolveReadinessProbeAuth
resolveReadinessProbeAuth,
shouldRotateNativeTokenAfterRejection
} from './native-auth-decisions'
// --- 1. body encoding (guards the double-JSON.stringify 422) ---
@@ -171,3 +172,33 @@ test('resolveGatedDownloadAuth uses the session token for token and local modes'
assert.deepEqual(resolveGatedDownloadAuth('local', null, 'sess'), { kind: 'token', token: 'sess' })
assert.deepEqual(resolveGatedDownloadAuth(undefined, null, null), { kind: 'token', token: null })
})
// --- 7. forced native rotation after a bearer rejection (#95701) ---
test('shouldRotateNativeTokenAfterRejection: only a structured 401 earns the one forced refresh', () => {
// The gate never rotates a native bearer server-side, so a 401 on a
// locally-unexpired access token is ambiguous until /auth/native/refresh
// has run once.
assert.equal(
shouldRotateNativeTokenAfterRejection(Object.assign(new Error('401: expired'), { statusCode: 401 })),
true
)
assert.equal(shouldRotateNativeTokenAfterRejection({ statusCode: 401 }), true)
})
test('shouldRotateNativeTokenAfterRejection: 403, 5xx, transport, and anonymous errors never rotate', () => {
// 403 is a policy refusal for an identity the gate recognized — a fresh
// bearer for the same identity cannot change it.
assert.equal(
shouldRotateNativeTokenAfterRejection(Object.assign(new Error('403: forbidden'), { statusCode: 403 })),
false
)
assert.equal(shouldRotateNativeTokenAfterRejection(Object.assign(new Error('503: down'), { statusCode: 503 })), false)
assert.equal(shouldRotateNativeTokenAfterRejection(Object.assign(new Error('reset'), { code: 'ECONNRESET' })), false)
// The pre-fix fetchJson shape: a "401: ..." message with no statusCode says
// nothing structured about the credential and must not trigger rotation.
assert.equal(shouldRotateNativeTokenAfterRejection(new Error('401: {"error":"session_expired"}')), false)
assert.equal(shouldRotateNativeTokenAfterRejection(null), false)
assert.equal(shouldRotateNativeTokenAfterRejection(undefined), false)
assert.equal(shouldRotateNativeTokenAfterRejection('401'), false)
})

View File

@@ -233,3 +233,24 @@ export function oauthGuardMayHardFail(providers: unknown): boolean {
return !named.every(provider => provider.supportsPassword)
}
/**
* Whether a rejected native-bearer request should try ONE forced token
* rotation before the rejection counts as confirmed.
*
* The gateway gate never rotates a native access token server-side — only
* cookie sessions get the transparent AT-from-RT refresh; the native flow is
* told to call `/auth/native/refresh` itself (dashboard_auth/middleware.py).
* So a 401 on a bearer the desktop still considers unexpired is ambiguous
* until that refresh has had its say: a live refresh token yields a fresh
* bearer, a dead one confirms the session is gone (#95701). Only a structured
* 401 qualifies. A 403 is a policy refusal for an identity the gate DID
* recognize, and a rotated token for the same identity cannot change it; a
* transport failure or an anonymous message-only error says nothing about
* the credential and must keep its transient classification.
*/
export function shouldRotateNativeTokenAfterRejection(error: unknown): boolean {
const statusCode = Number(error && typeof error === 'object' ? (error as { statusCode?: unknown }).statusCode : NaN)
return statusCode === 401
}

View File

@@ -0,0 +1,234 @@
/**
* Regression suite for issue #95701: an expired remote OAuth session must
* boot into ONE latched recovery overlay, not flicker between the connecting
* state and the overlay while the renderer re-drives a rejection that can
* never self-heal.
*
* The chain that broke:
*
* fetchJson (native bearer) → bare Error("401: ...") — no statusCode
* withTransientRetries → not an auth rejection → hammered 3x
* gatewayTicketFailure → transport copy, no needsOauthLogin
* startHermes → isReauth=false → NOT latched, retryable:true
* renderer boot-retry loop → running:true hides the overlay, repeat
*
* The first half of this file composes the REAL modules exactly the way
* main.ts does, so the contract is proven on the code that ships rather than
* on a mock of it. The second half pins the main.ts wiring with the repo's
* source-assertion pattern (main.ts has no exports; see hardening.test.ts).
*/
import assert from 'node:assert/strict'
import fs from 'node:fs'
import path from 'node:path'
import { fileURLToPath } from 'node:url'
import { test } from 'vitest'
import { httpStatusError } from './api-transport'
import { isReauthRequiredError } from './backend-health'
import {
isRetryableRemoteBootFailure,
shouldHoldBootProgressForReauth,
shouldLatchRemoteReauthFailure
} from './backend-start-failure'
import { gatewayTicketFailure, isGatewayAuthRejection, withTransientRetries } from './connection-config'
import { shouldRotateNativeTokenAfterRejection } from './native-auth-decisions'
// --- composition: the real modules, in production order ------------------
test('FIX #95701: a native-bearer 401 is a confirmed, non-retryable reauth rejection end to end', async () => {
// What fetchJson now throws for the gate's structured session_expired 401.
const bearerRejection = httpStatusError(401, '{"error":"session_expired","reason":"invalid_or_expired_session"}')
assert.equal(isGatewayAuthRejection(bearerRejection), true)
// mintGatewayWsTicket's transient-retry wrapper fails immediately: a dead
// session is never hammered.
let attempts = 0
const mintError = await withTransientRetries(
async () => {
attempts += 1
throw bearerRejection
},
{ sleep: async () => {} }
).then(
() => null,
(error: unknown) => error
)
assert.equal(attempts, 1)
assert.equal(mintError, bearerRejection)
// buildRemoteConnection wraps the rejection for the boot path.
const wrapped = gatewayTicketFailure(mintError, 'session expired — sign in', 'could not reach gateway') as any
assert.equal(wrapped.message, 'session expired — sign in')
assert.equal(wrapped.needsOauthLogin, true)
assert.equal(wrapped.statusCode, 401)
// startHermes's own composition: isReauth = isReauthRequiredError(error).
const isReauth = isReauthRequiredError(wrapped)
assert.equal(isReauth, true)
assert.equal(shouldLatchRemoteReauthFailure({ attemptedRemote: true, isReauth }), true)
assert.equal(
isRetryableRemoteBootFailure({ attemptedRemote: true, isReauth }),
false,
'the boot progress must be non-retryable so the renderer never re-drives the boot'
)
})
test('FIX #95701: the pre-fix error shape is exactly what made the rejection look transient', async () => {
// A "401: ..." message with no structured statusCode — what fetchJson used
// to throw. Every classifier below is shape-based on purpose, so this is
// the regression the structured error prevents.
const anonymous = new Error('401: {"error":"session_expired"}')
assert.equal(isGatewayAuthRejection(anonymous), false)
assert.equal(shouldRotateNativeTokenAfterRejection(anonymous), false)
let attempts = 0
await withTransientRetries(
async () => {
attempts += 1
throw anonymous
},
{ attempts: 3, sleep: async () => {} }
).catch(() => undefined)
assert.equal(attempts, 3, 'an anonymous 401 was retried like a transport blip')
const wrapped = gatewayTicketFailure(anonymous, 'auth copy', 'transport copy')
assert.equal(wrapped.message, 'transport copy')
assert.equal(isReauthRequiredError(wrapped), false)
assert.equal(isRetryableRemoteBootFailure({ attemptedRemote: true, isReauth: isReauthRequiredError(wrapped) }), true)
})
test('FIX #95701: a bearer 401 earns exactly one forced rotation; a 403 or a 5xx does not', () => {
assert.equal(shouldRotateNativeTokenAfterRejection(httpStatusError(401, 'expired')), true)
assert.equal(shouldRotateNativeTokenAfterRejection(httpStatusError(403, 'forbidden')), false)
assert.equal(shouldRotateNativeTokenAfterRejection(httpStatusError(503, 'down')), false)
// A 403 is still a confirmed rejection for the boot path — it just skips
// the pointless rotation on the way there.
const forbidden = gatewayTicketFailure(httpStatusError(403, 'forbidden'), 'auth copy', 'transport copy')
assert.equal(isReauthRequiredError(forbidden), true)
assert.equal(isRetryableRemoteBootFailure({ attemptedRemote: true, isReauth: true }), false)
})
test('FIX #95701: once latched, the boot surface ignores everything but the latched failure', () => {
const latched = gatewayTicketFailure(httpStatusError(401, 'expired'), 'session expired — sign in', 'transport')
// The latching emit itself (startHermes: updateBootProgress({ error: message, retryable: false })).
assert.equal(shouldHoldBootProgressForReauth(latched.message, { error: latched.message }), false)
// A sibling attempt's progress phase (advanceBootProgress → error: null, running: true).
assert.equal(shouldHoldBootProgressForReauth(latched.message, { error: null }), true)
// A sibling attempt's unrelated transport failure (would flip retryable back on).
assert.equal(shouldHoldBootProgressForReauth(latched.message, { error: 'Desktop boot failed: transport' }), true)
})
// --- main.ts wiring ------------------------------------------------------
const here = path.dirname(fileURLToPath(import.meta.url))
const mainSource = fs.readFileSync(path.join(here, 'main.ts'), 'utf8').replace(/\r\n/g, '\n')
/** Source of one top-level function of main.ts, from its header to the next top-level function. */
function mainFunction(header: string): string {
const start = mainSource.indexOf(header)
assert.notEqual(start, -1, `${header} must exist in main.ts`)
const candidates = ['\nfunction ', '\nasync function ']
.map(marker => mainSource.indexOf(marker, start + header.length))
.filter(index => index !== -1)
const end = candidates.length > 0 ? Math.min(...candidates) : mainSource.length
return mainSource.slice(start, end)
}
test('main.ts: fetchJson and fetchPublicJson throw the structured httpStatusError, never a bare Error', () => {
for (const header of ['function fetchJson(', 'function fetchPublicJson(']) {
const body = mainFunction(header)
assert.match(body, /reject\(httpStatusError\(res\.statusCode, text, res\.statusMessage\)\)/, header)
assert.doesNotMatch(
body,
/new Error\(`\$\{res\.statusCode\}:/,
`${header} must not rebuild the anonymous status error`
)
}
})
test('main.ts: the native ticket mint forces one refresh before the rejection is confirmed', () => {
const mint = mainFunction('async function mintGatewayWsTicket(')
assert.match(mint, /if \(!shouldRotateNativeTokenAfterRejection\(error\)\) \{\s*throw error/)
assert.match(mint, /ensureNativeAccessToken\(baseUrl, \{ forceRefresh: true \}\)/)
assert.match(mint, /return await mintGatewayWsTicketWithBearer\(baseUrl, rotatedAt, headers\)/)
const ensure = mainFunction('async function ensureNativeAccessToken(')
assert.match(ensure, /if \(!options\.forceRefresh && !tokenNeedsRefresh\(/)
// A dead refresh token drops the stored set (the overlay then reads "not
// connected" and offers Sign in) — this branch only works because the
// refresh POST now carries a structured statusCode.
assert.match(ensure, /error\.statusCode === 401\) \{\s*_clearNativeTokens\(baseUrl\)/)
})
test('main.ts: startHermes latches the failure before its first yield back to the event loop', () => {
const setPromise = mainSource.indexOf('backendConnectionState.setPromise(connectionAttempt, connectionPromise)')
assert.notEqual(setPromise, -1)
const catchStart = mainSource.lastIndexOf('.catch(async error => {', setPromise)
assert.notEqual(catchStart, -1)
const failurePath = mainSource.slice(catchStart, setPromise)
const invalidate = failurePath.indexOf('backendConnectionState.invalidate()')
const resetGuard = failurePath.indexOf('error instanceof FirstRunSetupResetError')
const reauthLatch = failurePath.indexOf('remoteReauthFailure = error instanceof Error ? error : new Error(message)')
const localLatch = failurePath.indexOf('backendStartFailure = error instanceof Error ? error : new Error(message)')
const exitWait = failurePath.lastIndexOf('await waitForBackendExit(failedProcess)')
const emit = failurePath.indexOf('updateBootProgress(')
for (const [label, index] of Object.entries({ invalidate, resetGuard, reauthLatch, localLatch, exitWait, emit })) {
assert.notEqual(index, -1, `${label} must exist in the startHermes failure path`)
}
// No await between invalidate() (which drops the shared attempt promise)
// and the latch assignments — a concurrent caller must hit the latch, not
// start a fresh attempt that re-emits running:true over the overlay.
const beforeLatch = failurePath.slice(invalidate, Math.max(reauthLatch, localLatch))
const awaitsBeforeLatch = beforeLatch.match(/\bawait\b/g) ?? []
assert.equal(
awaitsBeforeLatch.length,
1,
'the only await before the latches is the FirstRunSetupResetError early-exit branch'
)
assert.ok(resetGuard < reauthLatch, 'a first-run reset is never latched as a failure')
assert.ok(reauthLatch < exitWait && localLatch < exitWait, 'latches are set before the exit wait yields')
assert.ok(exitWait < emit, 'the failure is emitted after the child has exited, as before')
})
test('main.ts: updateBootProgress holds every non-latched update while the reauth latch is set', () => {
const body = mainFunction('function updateBootProgress(')
const hold = body.indexOf(
'shouldHoldBootProgressForReauth(remoteReauthFailure ? remoteReauthFailure.message : null, update)'
)
const assign = body.indexOf('bootProgressState = {')
assert.notEqual(hold, -1, 'the hold must key on the latched failure message')
assert.notEqual(assign, -1)
assert.ok(hold < assign, 'the hold runs before the state is touched or broadcast')
assert.match(body.slice(hold, assign), /return\s*\}/, 'a held update returns without broadcasting')
})