hermes-admin/hermes-agent
1
0
Fork 0
Code Issues Pull Requests Actions 28 Packages Projects Releases Wiki Activity

fix: video generation tools no longer let the agent pick the model

Browse Source 
video_generate advertised an optional `model` argument (and the xAI edit/extend
tools a model override) so the LLM could route a single call to a different
model family — a different endpoint and billing tier — than the one the user
selected in `hermes tools`. image_generate never exposed this, and #83080 asked
to extend it there; the ruling is the opposite: models do not choose models.

The `model` property is gone from the static and dynamic video_generate schema
and from xai_video_edit / xai_video_extend; a `model` smuggled into the call is
ignored and the configured `video_gen.model` (then the provider default) is what
reaches the request. Config-side selection (`video_gen.model`,
`video_gen.<provider>.model`, `<PROVIDER>_VIDEO_MODEL`) is unchanged, and the
xAI plugin's explicit-model branch is no longer reachable from the tool layer.

Refs #83080
This commit is contained in:
teknium1
2026-09-19 12:22:44 -07:00
parent 00570550f3
commit 19b29df13b
8 changed files with 28 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
plugins/video_gen/xai/__init__.py
View File

@@ -168,8 +168,8 @@ class XAIVideoGenProvider(VideoGenProvider):
seed: Optional[int] = None, **kwargs: Any,
) -> Dict[str, Any]:
return _run_xai_video(
"generation", _generate_xai_video_async, prompt=prompt, model=model,
explicit_model=bool(kwargs.get("_model_override_explicit")), image_url=image_url,
# ``model`` is the configured video_gen.model; the agent has no per-request override (#83080).
"generation", _generate_xai_video_async, prompt=prompt, model=model, explicit_model=False, image_url=image_url,
reference_image_urls=reference_image_urls, duration=duration, aspect_ratio=aspect_ratio, resolution=resolution,
)

11
tests/plugins/video_gen/test_xai_plugin_integration.py
View File

@@ -122,17 +122,6 @@ class TestXAIPayload:
assert payload["model"] == "grok-imagine-video-1.5"
assert payload["image"]["url"].startswith("data:image/png;base64,")
def test_explicit_model_override_is_honored_for_image(self, xai_provider):
provider, captured = xai_provider
provider.generate(
"animate this",
image_url="https://example.com/cat.png",
model="grok-imagine-video",
_model_override_explicit=True,
)
payload = _last_post(captured)["json"]
assert payload["model"] == "grok-imagine-video"
def test_reference_images_payload(self, xai_provider):
provider, captured = xai_provider
provider.generate(

2
tests/tools/test_video_generate_schema.py
View File

@@ -247,7 +247,7 @@ class TestDynamicParamGating(unittest.TestCase):
props = VIDEO_GENERATE_SCHEMA["parameters"]["properties"]
self.assertEqual(
sorted(props),
["aspect_ratio", "duration", "model", "prompt", "resolution"],
["aspect_ratio", "duration", "prompt", "resolution"],
)

39
tests/tools/test_video_generation_tool_surface_matrix.py
View File

@@ -236,43 +236,24 @@ def test_xai_text_only_via_tool_surface(matrix_env):
# ─────────────────────────────────────────────────────────────────────────
# tool-level `model` arg overrides config
# models do not choose models (#83080 ruling): the configured video_gen.model is the only selector
# ─────────────────────────────────────────────────────────────────────────
def test_tool_model_arg_overrides_config(matrix_env):
"""When the tool call passes model=, it wins over video_gen.model in config."""
def test_model_is_never_an_agent_choice(matrix_env):
"""No generation tool advertises a ``model`` parameter, and a ``model`` smuggled into the call
is ignored: the configured ``video_gen.model`` is what reaches the provider request."""
import tools.video_generation_tool as vt
import tools.xai_video_tools as xt
home, fal_calls, _ = matrix_env
# Config picks pixverse-v6, but tool call says veo3.1
result = _invoke_tool(
home,
{"video_gen": {"provider": "fal", "model": "pixverse-v6"}},
{"prompt": "a dog", "model": "veo3.1"},
)
assert result["success"] is True
assert result["model"] == "veo3.1"
# Outbound endpoint reflects the override, not config
assert fal_calls[0]["endpoint"] == "fal-ai/veo3.1"
assert result["model"] == "pixverse-v6"
assert fal_calls[0]["endpoint"] == "fal-ai/pixverse/v6/text-to-video"
def test_tool_model_arg_with_image_url_routes_to_override_image_endpoint(matrix_env):
"""model= override on text+image goes to the override family's image endpoint."""
home, fal_calls, _ = matrix_env
result = _invoke_tool(
home,
{"video_gen": {"provider": "fal", "model": "pixverse-v6"}},
{
"prompt": "animate this",
"image_url": "https://example.com/i.png",
"model": "kling-v3-4k",
},
)
assert result["success"] is True
assert result["model"] == "kling-v3-4k"
assert fal_calls[0]["endpoint"] == "fal-ai/kling-video/v3/4k/image-to-video"
# Kling 4K uses start_image_url
assert fal_calls[0]["arguments"].get("start_image_url") == "https://example.com/i.png"
assert "image_url" not in fal_calls[0]["arguments"]
schemas = [vt._build_dynamic_video_schema(), xt.XAI_VIDEO_EDIT_SCHEMA, xt.XAI_VIDEO_EXTEND_SCHEMA]
assert all("model" not in schema["parameters"]["properties"] for schema in schemas)

19
tools/video_generation_tool.py
View File

@@ -59,14 +59,8 @@ VIDEO_GENERATE_SCHEMA: Dict[str, Any] = {
"description": "Output resolution.",
"default": DEFAULT_RESOLUTION,
},
"model": {
"type": "string",
"description": (
"Optional model override; defaults to the configured "
"``video_gen.model``. Unknown models are rejected."
),
},
# Capability-gated args are added by _build_dynamic_video_schema; never statically.
# No ``model`` here: the backend/model is user configuration (``video_gen.model``), never an
# agent choice (#83080 ruling). Capability-gated args are added by _build_dynamic_video_schema; never statically.
},
# NOTE (schema diet, #95681): image_url / reference_image_urls / negative_prompt / audio / seed /
# upscale are added per-capability by _build_dynamic_video_schema.
@@ -181,7 +175,6 @@ def _handle_video_generate(args: Dict[str, Any], **_kw: Any) -> str:
"audio": _coerce_bool(args.get("audio")),
"seed": _coerce_int(args.get("seed")),
"upscale": _coerce_bool(args.get("upscale"))}
model_override = (args.get("model") or "").strip() or None
# Soft validation — providers do their own; our surface never accepts image-only.
if not prompt:
@@ -195,11 +188,10 @@ def _handle_video_generate(args: Dict[str, Any], **_kw: Any) -> str:
if provider is None:
return _missing_provider_error(configured)
# Explicit arg wins, then config, then provider default.
model = model_override or _read_configured_video_model() or provider.default_model()
# Config, then provider default; a ``model`` in args is ignored (models do not choose models).
model = _read_configured_video_model() or provider.default_model()
kwargs: Dict[str, Any] = {
"model": model, "_model_override_explicit": bool(model_override),
"image_url": image_url, "reference_image_urls": reference_image_urls, **optional}
"model": model, "image_url": image_url, "reference_image_urls": reference_image_urls, **optional}
# Drop None entries so providers see clean defaults.
kwargs = {k: v for k, v in kwargs.items() if v is not None}
pname = getattr(provider, "name", "?")
@@ -372,7 +364,6 @@ def _build_dynamic_video_schema() -> Dict[str, Any]:
"- audio: native stereo audio is generated with every video "
"(always on; no toggle) — describe the desired sound in the "
"prompt")
properties["model"] = static_props["model"]
return _schema("\n".join(parts), properties)

5
tools/xai_video_tools.py
View File

@@ -44,7 +44,6 @@ _VIDEO_URL_PARAM = {
"`public_url` from a prior xAI Imagine result."
),
}
_MODEL_PARAM = {"type": "string", "description": "Optional xAI Imagine model override."}
def _xai_video_schema(name: str, verb: str, noun: str, prompt_verb: str, extra: Dict[str, Any]) -> Dict[str, Any]:
@@ -65,7 +64,6 @@ def _xai_video_schema(name: str, verb: str, noun: str, prompt_verb: str, extra:
},
"video_url": _VIDEO_URL_PARAM,
**extra,
"model": _MODEL_PARAM,
},
"required": ["prompt", "video_url"],
},
@@ -106,8 +104,7 @@ def _run_xai_video_tool(args: Dict[str, Any], op: str, run, **extra: Any) -> str
"error_type": "provider_not_configured",
"provider": "xai",
})
model = _clean_string(args.get("model"))
return json.dumps(run(prompt=prompt, video_url=video_url, model=model, **extra))
return json.dumps(run(prompt=prompt, video_url=video_url, **extra))
def _handle_xai_video_edit(args: Dict[str, Any], **_kw: Any) -> str:

14
website/docs/developer-guide/video-gen-provider-plugin.md
View File

@@ -179,7 +179,8 @@ The tool exposes one schema across every backend. Providers ignore parameters th
| `negative_prompt` | Content to avoid (Pixverse/Kling only) |
| `audio` | Native audio (Veo3 / Pixverse pricing tier) |
| `seed` | Reproducibility |
| `model` | Override the active model/family |
There is deliberately no `model` parameter: the backend and model are user configuration (`video_gen.provider` / `video_gen.model`), never an agent choice. Your `generate()` still receives `model=` — it is the configured model, resolved by the tool layer.
The provider's `capabilities()` advertises which of these are honored. The agent sees the active backend's capabilities in the tool description, dynamically rebuilt when the user changes backend via `hermes tools`.
@@ -208,11 +209,12 @@ The user picks `veo3.1` once in `hermes tools`. The agent never thinks about end
For per-instance model knobs (see `plugins/video_gen/fal/__init__.py`):
1. `model=` keyword from the tool call
2. `<PROVIDER>_VIDEO_MODEL` env var
3. `video_gen.<provider>.model` in `config.yaml`
4. `video_gen.model` in `config.yaml` (when it's one of your IDs)
5. Provider's `default_model()`
1. `<PROVIDER>_VIDEO_MODEL` env var
2. `video_gen.<provider>.model` in `config.yaml`
3. `video_gen.model` in `config.yaml` (when it's one of your IDs)
4. Provider's `default_model()`
The `model=` keyword your `generate()` receives is the outcome of this resolution — a `model` in the agent's tool call is ignored, so the LLM cannot switch backends or billing tiers on its own.
## Response shape

2
website/docs/reference/tools-reference.md
View File

@@ -327,7 +327,7 @@ Backends ship as plugins under `plugins/video_gen/<name>/`:
- **OpenRouter** — every generative model on OpenRouter's video API (Veo 3.1, Sora 2 Pro, Kling 3, Seedance 2, Wan 3, Hailuo 3, Grok Imagine, FLUX 3 Video, …); text-to-video, image-to-video and reference-to-video; catalog and per-model limits fetched live (requires `OPENROUTER_API_KEY`, billed to your OpenRouter credit).
- **DeepInfra** — live `video-gen` catalog over the OpenAI-compatible videos endpoint (requires `DEEPINFRA_API_KEY`).
The single `video_generate` tool covers both modalities — pass `image_url` to animate a still, omit it to generate from text alone. The active backend auto-routes to the right endpoint. The tool's description is rebuilt at session start to reflect the active backend's actual capabilities (modalities, aspect ratios, resolutions, duration range, max reference images, audio support). See [Video Generation Provider Plugins](../developer-guide/video-gen-provider-plugin.md) for backend authoring.
The single `video_generate` tool covers both modalities — pass `image_url` to animate a still, omit it to generate from text alone. The active backend auto-routes to the right endpoint. As with `image_generate`, the model is user-configured (`video_gen.model`) and not selectable by the agent — none of the video tools take a `model` argument. The tool's description is rebuilt at session start to reflect the active backend's actual capabilities (modalities, aspect ratios, resolutions, duration range, max reference images, audio support). See [Video Generation Provider Plugins](../developer-guide/video-gen-provider-plugin.md) for backend authoring.
| Tool | Description | Requires environment |
|------|-------------|----------------------|