hermes-admin/hermes-agent
1
0
Fork 0
Code Issues Pull Requests Actions 4 Packages Projects Releases Wiki Activity

plugin-catalog: bump evalroute to v0.6.2

Browse Source 
Supersedes the v0.6.1 bump on this branch. 0.6.2 pins the evalroute library at
>=0.9,<0.10 and adds two Disclosure clauses: "hermes evalroute install-encoder"
(run by hand) downloads a 146 MB text-classification model from the Hub and
runs it locally; "hermes evalroute export-cases" (run by hand) writes the
user's own pinned route rows, task text included, to a local file and uploads
nothing. Plugin code unchanged.
This commit is contained in:
James
2026-10-04 22:03:35 -07:00
committed by Teknium
parent 713ecd8b46
commit cf8a413808
1 changed files with 8 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
plugin-catalog/evalroute.yaml
View File

@@ -1,19 +1,19 @@
name: evalroute
repo: https://github.com/keppy/hermes-plugin-evalroute
sha: 3cbb7298cfc5b19e0331c5f5020a1605744c873a
description: "Route a task to the right (model, reasoning effort) arm before you start. Classifies the task into a lane, prints a provenance-carrying route card, and installs per-model reasoning-effort defaults. Strong rules run locally; weak-signal classification calls the host LLM and consumes tokens. The separate harness and shim can make paid provider calls when explicitly run; no plugin-owned credentials. Disclosure — weak-signal task classification sends the task text to your active Hermes model via the host LLM facade (token cost). The model-callable tool, /route, /rate and observed /model and /reasoning commands append task text and command args to a local ledger at <HERMES_HOME>/evalroute/labels.jsonl. \"hermes evalroute install-routes\" rewrites agent.reasoning_overrides in config.yaml when you run it. The bundled harness and Hermes shim are never run by the plugin. When run by hand they make paid provider calls, execute model-generated code on the host via shell checkers, and run hermes -z with approvals bypassed. Disclosure — \"hermes evalroute sync\" (run by hand) downloads the route table from the Hugging Face dataset keppy/evalroute-flywheel into HERMES_HOME and route cards then show that dataset's text; \"hermes evalroute dispatch\" (run by hand) spawns a single-query hermes chat on the routed model (Nous provider by default) with the brief as its query, inheriting the current environment except HERMES_YOLO_MODE (the worker never inherits --yolo; its approvals follow approvals.single_query_mode, default deny), and writes its report and a .dispatch.json sidecar beside the brief. Disclosure — \"hermes evalroute contribute\" (run by hand only) refuses to run unless evalroute.contribute: true is set in config.yaml, and then uploads redacted outcome rows only — lane, routed arm, confirmed arm, verdict, method, facet counts, ISO week and an HMAC-SHA256 task hash; never task text, notes, paths or the salt — to the user's own path (contributed/<hf-username>/) in the Hugging Face dataset keppy/evalroute-flywheel using the user's existing huggingface_hub login; \"--dry-run\" prints the exact rows and sends nothing; the per-install salt stays local in <HERMES_HOME>/evalroute/contribute.salt. As of 0.6.1 the plugin is a thin adapter: all behaviour lives in the PyPI package evalroute (>=0.8,<0.9), declared as a Python dependency that Hermes installs on your confirmation."
sha: 65d5a6a2544487a56e4cab1bf4a3ecf7444b8b2d
description: 'Route a task to the right (model, reasoning effort) arm before you start. Classifies the task into a lane, prints a provenance-carrying route card, and installs per-model reasoning-effort defaults. Strong rules run locally; weak-signal classification calls the host LLM and consumes tokens. The separate harness and shim can make paid provider calls when explicitly run; no plugin-owned credentials. Disclosure — weak-signal task classification sends the task text to your active Hermes model via the host LLM facade (token cost). The model-callable tool, /route, /rate and observed /model and /reasoning commands append task text and command args to a local ledger at <HERMES_HOME>/evalroute/labels.jsonl. "hermes evalroute install-routes" rewrites agent.reasoning_overrides in config.yaml when you run it. The bundled harness and Hermes shim are never run by the plugin. When run by hand they make paid provider calls, execute model-generated code on the host via shell checkers, and run hermes -z with approvals bypassed. Disclosure — "hermes evalroute sync" (run by hand) downloads the route table from the Hugging Face dataset keppy/evalroute-flywheel into HERMES_HOME and route cards then show that dataset''s text; "hermes evalroute dispatch" (run by hand) spawns a single-query hermes chat on the routed model (Nous provider by default) with the brief as its query, inheriting the current environment except HERMES_YOLO_MODE (the worker never inherits --yolo; its approvals follow approvals.single_query_mode, default deny), and writes its report and a .dispatch.json sidecar beside the brief. Disclosure — "hermes evalroute contribute" (run by hand only) refuses to run unless evalroute.contribute: true is set in config.yaml, and then uploads redacted outcome rows only — lane, routed arm, confirmed arm, verdict, method, facet counts, ISO week and an HMAC-SHA256 task hash; never task text, notes, paths or the salt — to the user''s own path (contributed/<hf-username>/) in the Hugging Face dataset keppy/evalroute-flywheel using the user''s existing huggingface_hub login; "--dry-run" prints the exact rows and sends nothing; the per-install salt stays local in <HERMES_HOME>/evalroute/contribute.salt. Disclosure — "hermes evalroute install-encoder <dir|owner/name>" (run by hand only) downloads a small text-classification model (the default id keppy/evalroute-lane-encoder is a 146 MB ModernBERT fine-tune on the Hugging Face Hub) into <HERMES_HOME>/evalroute/encoder/ and from then on "hermes evalroute route" runs it locally, offline, to classify the task when the keyword rules are weak, instead of the LLM call; it needs torch and transformers in the plugin environment (pip extra evalroute[encoder]), which the plugin does not install; "install-encoder --remove" deletes it. "hermes evalroute export-cases" (run by hand only) writes your --lane-pinned route rows, task text included, to a local file you name, for training your own encoder; stdout carries counts only and nothing is uploaded. As of 0.6.2 the plugin is a thin adapter: all behaviour lives in the PyPI package evalroute (>=0.9,<0.10), declared as a Python dependency that Hermes installs on your confirmation.'
maintainer: keppy
tier: community
category: tools
requires_hermes: ">=0.21"
version: "0.6.1"
requires_hermes: '>=0.21'
version: "0.6.2"
docs_url: https://github.com/keppy/hermes-plugin-evalroute#readme
capabilities:
provides_tools:
- evalroute_route
- evalroute_route
provides_hooks:
- pre_llm_call
- pre_command
- post_llm_call
- pre_llm_call
- pre_command
- post_llm_call
provides_middleware: []
requires_env: []