hermes-agent/docs/chronos-managed-cron-contract.md

Chronos managed-cron — agent ↔ NAS wire contract

Status: authoritative wire spec for the Chronos cron provider. Audience: the NAS-side implementer of the agent-cron endpoints (nous-account-service) and anyone debugging the managed-cron path.

Chronos lets a hosted Hermes gateway scale to zero while idle and still fire cron jobs. Instead of an in-process 60-second ticker, the agent asks NAS to arm exactly one external one-shot per job at that job's real next-fire time. NAS calls the agent back at fire time over an authenticated webhook; the agent runs the job and re-arms the next one-shot. Between fires the agent process can be fully stopped — it wakes only on a genuine fire.

The external scheduler NAS uses to implement the one-shots is an internal NAS implementation detail. The agent never talks to it, never holds its credentials, and never names it. The agent only knows the three NAS endpoints below.

create/update/pause/resume/remove a cron job (agent side)
  │
  ▼
ChronosCronScheduler.reconcile()        ── agent computes next_run_at
  │  POST {portal}/api/agent-cron/provision   (auth: agent's Nous access token)
  ▼
NAS arms a one-shot for fire_at         ── NAS owns the scheduler + its creds
  │
  ⏰ at fire_at
  ▼
scheduler → POST {portal}/api/agent-cron/relay   (auth: scheduler signature, NAS-verified)
  │
  ▼
NAS mints a short-lived agent-audience JWT (purpose=cron_fire)
  │  POST {agent_callback_url}/api/cron/fire        (auth: that JWT)
  ▼
agent verifies the NAS JWT → store CAS claim → run_one_job → re-arm next one-shot

Trust model (read this first)

Hop	Who calls whom	Auth mechanism	Verified by
1	agent → NAS (provision/cancel/list)	the agent's existing Nous Portal access token (Bearer) — for a hosted agent this is the bootstrap-session token NAS planted in auth.json (client hermes-cli-vps), NOT an agent:* client token	NAS (its normal agent-token path)
2	scheduler → NAS (relay)	the scheduler's request signature	NAS (the signature path it already has)
3	NAS → agent (/api/cron/fire)	a short-lived NAS-minted JWT (aud=agent:{instance_id}, purpose=cron_fire)	agent (PyJWT against NAS JWKS)

Which token, exactly (hop 1). A hosted agent never holds an agent:{instance_id} OAuth client credential — that shape is minted only by the interactive dashboard auth-code grant (a browser user). For all of its own outbound portal calls the agent uses the bootstrap-session access token (resolve_nous_access_token), minted under the bootstrap-only client hermes-cli-vps and seeded into the container on first boot. NAS therefore must resolve the calling agent's instance id from EITHER an agent:{id} client (self-hosted/dashboard callers) OR — for the bootstrap token — from AgentInstance.bootstrapSessionId matching the token's session id (sid), org-scoped. The fire JWT minted at hop 3 still carries aud=agent:{instance_id} regardless. (Gating hop 1 on an agent:* client alone 403s every real hosted-agent provision — see src/server/agent-cron/instance-auth.ts.)

Why NAS-mediated rather than scheduler→agent direct: the scheduler signs with NAS's keys, which the agent does not (and should not) hold. The agent can only verify a NAS-minted token — a trust path it already has. This keeps all scheduler credentials inside NAS. (Full rationale: the plan's DQ-4.)

No new secret is introduced on the agent: hop 1 reuses the token the agent already uses for the portal, and hop 3 reuses the NAS-JWT verification the agent already performs.

---

Endpoint 1 — POST /api/agent-cron/provision (agent → NAS)

Arm (or re-arm, idempotently) exactly one one-shot for a job.

• Auth: Authorization: Bearer <agent Nous access token>. NAS validates via its normal agent-token path and scopes the row to the calling agent/org.
• Request body:

  {
    "job_id": "ab12cd34",
    "fire_at": "2026-06-18T12:34:56+00:00",
    "agent_callback_url": "https://agent-xyz.fly.dev",
    "dedup_key": "ab12cd34:2026-06-18T12:34:56+00:00"
  }
  

  • fire_at — ISO 8601, agent-computed. May be sub-minute in the future; NAS must honor second-granularity (the agent owns the time, so there is no 1-minute scheduler floor).
  • agent_callback_url — the agent's own publicly-reachable base URL. NAS POSTs {agent_callback_url}/api/cron/fire at fire time.
  • dedup_key — "{job_id}:{fire_at}". NAS upserts by (agent_id, job_id) so re-arming the same fire is idempotent (no duplicate one-shots). A new fire_at for the same job_id replaces the prior arm.
• Action: arm one one-shot to fire at fire_at, destined for the NAS relay route (Endpoint 3) — NOT the agent directly, so NAS stays in the loop to mint the agent JWT. Persist (agent_id, job_id, schedule_id, agent_callback_url).
• Response: 200 {"schedule_id": "<opaque>"}.

Endpoint 2 — POST /api/agent-cron/cancel (agent → NAS)

• Auth: same as Endpoint 1.
• Body: {"job_id": "ab12cd34"}.
• Action: cancel the armed one-shot for (agent_id, job_id) and delete the row. Idempotent — cancelling an unknown job is a 200 no-op.
• Response: 200 {"ok": true}.

Endpoint 3 — POST /api/agent-cron/relay (scheduler → NAS, the fire relay)

• Auth: the scheduler's request signature, verified by NAS with the signature path it already has. This is the trust boundary for the fire — a forged relay call must be rejected here.
• Action:
  1. Look up (agent_id, job_id) → agent_callback_url from the persisted row.
  2. Mint a short-lived JWT: aud = "agent:{instance_id}", iss = {portal_url}, purpose = "cron_fire", small exp (≈60–120s), signed with NAS's normal asymmetric signing key (published via JWKS).
  3. POST {agent_callback_url}/api/cron/fire with Authorization: Bearer <that JWT> and body {"job_id": "...", "fire_at": "..."}.
  4. Treat a non-2xx agent response as a retryable failure (let the scheduler retry the relay). The agent's store CAS de-dupes a double fire, so retries are safe.
• Response to the scheduler: 2xx once the agent POST is accepted (202), so the scheduler does not retry a delivered fire.

---

Inbound POST /api/cron/fire (NAS → agent) — agent side, already implemented

This is the agent endpoint NAS calls in Endpoint 3 step 3. Two hops on hosted deployments:

1. Dashboard app (hermes_cli/web_server.py) — the agent's only public HTTP surface (the Fly proxy exposes exactly one port, the dashboard's). It is in PUBLIC_API_PATHS so the dashboard cookie gate lets the bearer-JWT callback through to the verifier. The dashboard verifies the JWT, resolves the job's profile, then forwards the fire to hop 2 on loopback with the NAS bearer preserved — it does NOT execute the job itself.
2. Gateway APIServerAdapter (gateway/platforms/api_server.py, loopback bind, default port 8642) — re-verifies the JWT (defense in depth) and runs the job with the gateway's live platform adapters, which is what makes delivery work for relay-fronted logical platforms and E2EE rooms (the standalone send path can serve neither). Self-host API-server deployments that expose the api_server directly hit hop 2 without hop 1.

Gateway unreachable from hop 1 (scale-to-zero wake still booting, restart window, api_server disabled) → the dashboard returns 503 and NAS retries (non-2xx = retryable, below); the store CAS de-dupes the eventual double fire. There is deliberately no in-dashboard execution fallback. The verifier is plugins/cron/chronos/verify.py.

• Auth: Authorization: Bearer <NAS-minted JWT>. The agent verifies:
  • signature against the NAS JWKS (cron.chronos.nas_jwks_url),
  • aud == cron.chronos.expected_audience (this agent's agent:{instance_id}),
  • iss == cron.chronos.portal_url,
  • exp / nbf (30s leeway),
  • purpose == "cron_fire" — a general agent JWT (no/other purpose) is rejected so it can't be replayed against this endpoint.
• Body: {"job_id": "ab12cd34", "fire_at": "..."} (only job_id is used).
• Behavior:
  • invalid/missing/forged/expired/wrong-aud/wrong-purpose token → 401, no execution.
  • missing job_id → 400.
  • valid → 202 {"status": "accepted", "job_id": "..."} immediately, and the job runs in the background. 202-before-run means a long agent turn never trips the relay's HTTP timeout.
• At-most-once: the agent claims the job with a store-level compare-and-set (claim_job_for_fire) before running. A relay/scheduler retry that arrives while the first fire is in flight (or after it completed) loses the claim and does not double-run.

---

At-most-once & re-arm semantics

• Recurring (cron/interval): on fire, the agent advances next_run_at (under its store lock) as part of the claim, runs the job, then re-provisions a one-shot for the new next_run_at. A duplicate relay for the old fire_at finds the claim taken / time advanced and is dropped.
• One-shot (30m, +90s, etc.): fires once; mark_job_run marks it completed. No re-arm.
• repeat.times = N: mark_job_run deletes the job at the limit, so get_job returns None after the final fire → the agent does not re-arm → the schedule stops cleanly with no orphaned one-shot.
• Multi-replica agents: the store CAS makes the fire at-most-once across N gateway replicas sharing one HERMES_HOME — exactly one replica runs each fire.

Reconcile (self-healing)

The agent reconciles desired (jobs.json) vs armed on:

• start() (gateway boot / wake),
• every successful job mutation (on_jobs_changed),
• piggybacked after each fire (re-arm).

Reconcile arms missing/changed-time jobs and cancels orphans. A missed provision (transient NAS error) self-heals on the next reconcile. There is no periodic wake of a sleeping agent — that would negate scale-to-zero.

Config (agent side)

All non-secret (cron.chronos.* in config.yaml); the agent holds no scheduler credentials. For hosted agents NAS sets these at provision time:

key	meaning
cron.provider	"chronos" to activate (empty = built-in ticker)
cron.chronos.portal_url	NAS base URL (also the expected JWT iss)
cron.chronos.callback_url	the agent's own public base URL for NAS→agent fires
cron.chronos.expected_audience	this agent's JWT aud (agent:{instance_id})
cron.chronos.nas_jwks_url	NAS JWKS for verifying the fire JWT

If callback_url / portal_url is blank or the agent has no Nous login, is_available() returns False and the resolver falls back to the built-in in-process ticker — cron never loses its trigger.

Escape hatch (not default)

The inbound /api/cron/fire verifier is pluggable (get_fire_verifier()). If relay volume through NAS ever saturates, a direct scheduler→agent mode with a per-job NAS-minted cron-key can replace the NAS-JWT verifier with no change to the webhook handler. NAS-mediated (this contract) is the default.