hermes-admin/hermes-agent
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

chore(plugin-catalog): model-drift-watch disclosure at merge

Browse Source
This commit is contained in:
teknium1
2026-10-04 04:26:27 -07:00
committed by Teknium
parent 071ddfc80c
commit ec39149895
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
plugin-catalog/model-drift-watch.yaml
View File

@@ -1,7 +1,7 @@
name: model-drift-watch
repo: https://github.com/TakeshiTGAL/hermes-plugin-model-drift
sha: 0102786ba4812e8581f3f6234a453eb26137fa52
description: "Scheduled check that the model you use in Hermes has not quietly got worse: 34 fixed, exactly graded questions (reasoning, instruction following, Japanese, plus your own) run through Hermes cron, each run tested against the same model's own baseline with an exact per-question test and a same-day confirmation run, and a message only for a confirmed drop, a changed served model or a failed measurement. Not yet tried against a real LLM API (checked with a local mock model and real Hermes cron). Disclosure: each check calls your own model through ctx.llm on your bill, capped per check by the plugin (default 120,000 tokens / $0.50; the scheduled agent turn is extra); price lookup goes through Hermes's pricing code; 10 optional code questions, off by default, would run model-written Python in a child process; creates one Hermes cron job when you run `schedule` (remove with `unschedule`); writes run history to plugin-data/model-drift-watch; no telemetry."
description: "Scheduled check that the model you use in Hermes has not quietly got worse: 34 fixed, exactly graded questions (reasoning, instruction following, Japanese, plus your own) run through Hermes cron, each run tested against the same model's own baseline with an exact per-question test and a same-day confirmation run, and a message only for a confirmed drop, a changed served model or a failed measurement. Not yet tried against a real LLM API (checked with a local mock model and real Hermes cron). Disclosure — each check calls your own model through ctx.llm on your bill, capped per check by the plugin (default 120,000 tokens / $0.50, no daily cap; the agent can start checks itself via the model_drift tool, and the scheduled agent turn is extra); price lookup goes through Hermes's pricing code (may fetch OpenRouter/models.dev prices); 10 optional code questions, off by default, run model-written Python in a limited child process (not a hostile-code sandbox) including in unattended cron runs when enabled; `schedule` creates one Hermes cron job that survives plugin removal (run `unschedule` first); writes run history to plugin-data/model-drift-watch; no telemetry."
maintainer: TakeshiTGAL
tier: community
category: automation