From 071ddfc80c2cb8454ef7a9287d554ba8ce13399e Mon Sep 17 00:00:00 2001 From: TakeshiTGAL Date: Sat, 3 Oct 2026 07:48:59 +0900 Subject: [PATCH] feat(plugin-catalog): add model-drift-watch --- plugin-catalog/model-drift-watch.yaml | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 plugin-catalog/model-drift-watch.yaml diff --git a/plugin-catalog/model-drift-watch.yaml b/plugin-catalog/model-drift-watch.yaml new file mode 100644 index 0000000000..24a7895614 --- /dev/null +++ b/plugin-catalog/model-drift-watch.yaml @@ -0,0 +1,17 @@ +name: model-drift-watch +repo: https://github.com/TakeshiTGAL/hermes-plugin-model-drift +sha: 0102786ba4812e8581f3f6234a453eb26137fa52 +description: "Scheduled check that the model you use in Hermes has not quietly got worse: 34 fixed, exactly graded questions (reasoning, instruction following, Japanese, plus your own) run through Hermes cron, each run tested against the same model's own baseline with an exact per-question test and a same-day confirmation run, and a message only for a confirmed drop, a changed served model or a failed measurement. Not yet tried against a real LLM API (checked with a local mock model and real Hermes cron). Disclosure: each check calls your own model through ctx.llm on your bill, capped per check by the plugin (default 120,000 tokens / $0.50; the scheduled agent turn is extra); price lookup goes through Hermes's pricing code; 10 optional code questions, off by default, would run model-written Python in a child process; creates one Hermes cron job when you run `schedule` (remove with `unschedule`); writes run history to plugin-data/model-drift-watch; no telemetry." +maintainer: TakeshiTGAL +tier: community +category: automation +docs_url: https://github.com/TakeshiTGAL/hermes-plugin-model-drift#readme +version: "0.1.0" +requires_hermes: ">=0.21.5" +platforms: [] +capabilities: + provides_tools: + - model_drift + provides_hooks: [] + provides_middleware: [] + requires_env: []