fix(url_safety): dial a declared local-proxy fake-ip block instead of blocking it
A TUN proxy in fake-ip mode (Mihomo/Clash `fake-ip`, Surge enhanced) answers DNS with an address from its own block — `198.18.0.0/15` by default — for every name outside its filter. The SSRF guard resolves names with the local resolver and reads that sentinel as a private destination, so on such a host every fetch fails before the request leaves the machine: `web_extract`, gateway media downloads (`is_safe_url` has ~20 call sites, including the Feishu/WeCom/Telegram/Slack/Discord attachment paths) and the browser relay all report "URL targets a private or internal network address". The existing hostname allowlist (`_TRUSTED_PRIVATE_IP_HOSTS`, the QQ multimedia case) does not generalise to this: the sentinel is a property of the host's resolver, not of one name. Fix: `security.fake_ip_ranges` declares the CIDR blocks the local proxy owns. Answers inside a declared block are dialable even with private-IP blocking on. Empty by default, so no existing host changes behaviour. Loopback, RFC 1918, link-local, CGNAT and the cloud-metadata floor are untouched. Pre-flight and connect-time checks both go through `_resolved_ip_block_reason`, so one exemption covers the class instead of the ~20 call sites. Tests: `TestDeclaredFakeIpSentinelRanges` in tests/tools/test_url_safety.py — red on the unfixed module (declared sentinel rejected, "Blocked request to private/internal address during connect: example.com -> 198.18.0.55"), green after. The pre-existing benchmark/QQ hostname tests are unchanged and still pass (66 passed). Docs: website/docs/user-guide/security.md + zh-Hans translation. The key is registered in hermes_cli/config_defaults.py so `hermes config set security.fake_ip_ranges` is not flagged as unknown.
This commit is contained in:
@@ -1624,6 +1624,10 @@ DEFAULT_CONFIG = {
|
||||
"personalities": {},
|
||||
"security": { # Security: pre-exec scanning via tirith plus related guards.
|
||||
"allow_private_urls": False, # allow requests to private/internal IPs (OpenWrt, VPNs)
|
||||
# CIDR blocks a local TUN proxy answers DNS with (Mihomo/Clash fake-ip, Surge enhanced).
|
||||
# Answers inside these blocks are the proxy's sentinels, not internal hosts, so the guard
|
||||
# dials them instead of rejecting them as private. Empty = normal private-address verdict.
|
||||
"fake_ip_ranges": [],
|
||||
"redact_secrets": True,
|
||||
# Persisted acknowledgement for unattended model overrides whose tier lets the vendor train
|
||||
# on prompts. The startup guard still warns every run; cost guards are unaffected.
|
||||
|
||||
@@ -19,6 +19,7 @@ from tools.url_safety import (
|
||||
_is_blocked_ip,
|
||||
_global_allow_private_urls,
|
||||
_reset_allow_private_cache,
|
||||
_reset_fake_ip_cache,
|
||||
)
|
||||
|
||||
import ipaddress
|
||||
@@ -496,3 +497,44 @@ class TestRedirectTargetFromResponse:
|
||||
next_request=_FakeNextRequest("http://10.0.0.1/meta"),
|
||||
)
|
||||
assert redirect_target_from_response(resp) == "http://10.0.0.1/meta"
|
||||
|
||||
|
||||
class TestDeclaredFakeIpSentinelRanges:
|
||||
"""A local TUN proxy answers DNS with a fake-ip block — declared in ``security.fake_ip_ranges``.
|
||||
|
||||
On such a host every name outside the proxy's filter resolves into that block, so keying the
|
||||
guard on the resolver's answer blocked every outbound fetch (web_extract, platform attachment
|
||||
downloads, the browser relay) while the request never reached the network at all. The exemption
|
||||
is per-host opt-in and scoped to the declared block — an undeclared host keeps the ordinary
|
||||
private-address verdict (see TestProxyEnvironmentDnsDelegation).
|
||||
"""
|
||||
|
||||
@pytest.fixture
|
||||
def declared(self, monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
"hermes_cli.config.read_raw_config",
|
||||
lambda: {"security": {"fake_ip_ranges": ["198.18.0.0/15"]}},
|
||||
)
|
||||
_reset_fake_ip_cache()
|
||||
yield
|
||||
_reset_fake_ip_cache()
|
||||
|
||||
def test_undeclared_host_is_unaffected(self):
|
||||
with _resolves_to("198.18.0.23"):
|
||||
assert is_safe_url("https://example.com/file.jpg") is False
|
||||
|
||||
def test_declared_sentinel_is_dialable_with_private_blocking_on(self, declared):
|
||||
with _resolves_to("198.18.1.125"):
|
||||
assert is_safe_url("https://example.com/") is True
|
||||
|
||||
def test_declared_sentinel_passes_the_connect_time_check_too(self, declared):
|
||||
with _resolves_to("198.18.0.55"):
|
||||
assert _resolved_http_connect_ips("example.com", 443, "https") == ["198.18.0.55"]
|
||||
|
||||
def test_declaration_does_not_excuse_real_private_answers(self, declared):
|
||||
with _resolves_to("192.168.99.99"):
|
||||
assert is_safe_url("https://example.com/") is False
|
||||
|
||||
def test_metadata_floor_outranks_the_declaration(self, declared):
|
||||
with _resolves_to("169.254.169.254"):
|
||||
assert is_safe_url("http://example.com/") is False
|
||||
|
||||
@@ -1,7 +1,10 @@
|
||||
"""URL safety checks — blocks requests to private/internal network addresses (SSRF).
|
||||
|
||||
``security.allow_private_urls: true`` disables private-IP blocking (DNS that resolves public
|
||||
names to private ranges); cloud metadata hostnames/IPs are **always** blocked. DNS rebinding
|
||||
names to private ranges); cloud metadata hostnames/IPs are **always** blocked. A local TUN proxy
|
||||
that answers DNS with a fake-ip block (Mihomo/Clash fake-ip, Surge enhanced) declares that block
|
||||
in ``security.fake_ip_ranges`` so its sentinel answers are dialable instead of looking private;
|
||||
the list is empty by default, so the sentinel stays blocked for everyone else. DNS rebinding
|
||||
(TOCTOU) is closed for Hermes-owned httpx paths by ``create_ssrf_safe_[async_]client()``, which
|
||||
re-apply the policy at TCP connect and dial the validated IP while preserving Host/SNI. Redirect
|
||||
bypass is mitigated by response hooks re-validating each target (``redirect_target_from_response``).
|
||||
@@ -120,6 +123,7 @@ _CGNAT_NETWORK = ipaddress.ip_network("100.64.0.0/10")
|
||||
|
||||
# Global toggle cache (process lifetime; see _global_allow_private_urls).
|
||||
_allow_private_resolved, _cached_allow_private = False, False
|
||||
_fake_ip_resolved, _cached_fake_ip_ranges = False, ()
|
||||
|
||||
|
||||
def _global_allow_private_urls() -> bool:
|
||||
@@ -160,6 +164,50 @@ def _reset_allow_private_cache() -> None:
|
||||
_allow_private_resolved = _cached_allow_private = False
|
||||
|
||||
|
||||
def _resolve_fake_ip_ranges() -> tuple:
|
||||
"""CIDR blocks this host's local proxy answers DNS with (``security.fake_ip_ranges``).
|
||||
|
||||
A TUN proxy in fake-ip mode answers every non-filtered name with an address from its own
|
||||
block; that answer is the proxy's sentinel, not an internal host's, so treating it as a
|
||||
private target blocks every outbound fetch on such a host (web_extract, platform attachment
|
||||
downloads, the browser relay). Empty by default: no host gets the exemption unless it
|
||||
declares one, and the sentinel range keeps the ordinary private-address verdict otherwise.
|
||||
"""
|
||||
try:
|
||||
from hermes_cli.config import read_raw_config
|
||||
block = read_raw_config().get("security", {})
|
||||
raw = block.get("fake_ip_ranges") if isinstance(block, dict) else None
|
||||
if isinstance(raw, str):
|
||||
raw = [part.strip() for part in raw.split(",")]
|
||||
if not isinstance(raw, (list, tuple)):
|
||||
return ()
|
||||
networks = []
|
||||
for entry in raw:
|
||||
if not str(entry).strip():
|
||||
continue
|
||||
try:
|
||||
networks.append(ipaddress.ip_network(str(entry).strip(), strict=False))
|
||||
except ValueError:
|
||||
logger.warning("Ignoring unparseable security.fake_ip_ranges entry: %r", entry)
|
||||
return tuple(networks)
|
||||
except Exception:
|
||||
return () # config unavailable (tests, early import) — keep the secure default
|
||||
|
||||
|
||||
def _global_fake_ip_ranges() -> tuple:
|
||||
"""Process-lifetime cache, same shape as the allow_private toggle."""
|
||||
global _fake_ip_resolved, _cached_fake_ip_ranges
|
||||
if not _fake_ip_resolved:
|
||||
_fake_ip_resolved, _cached_fake_ip_ranges = True, _resolve_fake_ip_ranges()
|
||||
return _cached_fake_ip_ranges
|
||||
|
||||
|
||||
def _reset_fake_ip_cache() -> None:
|
||||
"""Reset the cached sentinel ranges — only for tests."""
|
||||
global _fake_ip_resolved, _cached_fake_ip_ranges
|
||||
_fake_ip_resolved, _cached_fake_ip_ranges = False, ()
|
||||
|
||||
|
||||
def _normalize_hostname(host: Optional[str]) -> str:
|
||||
return (host or "").strip().lower().rstrip(".")
|
||||
|
||||
@@ -189,6 +237,21 @@ def _is_always_blocked_ip(ip: _IPAddress) -> bool:
|
||||
return ip in _ALWAYS_BLOCKED_IPS or any(ip in net for net in _ALWAYS_BLOCKED_NETWORKS)
|
||||
|
||||
|
||||
def _is_local_proxy_sentinel(ip: _IPAddress) -> bool:
|
||||
"""True when *ip* is in a fake-ip block this host declared in ``security.fake_ip_ranges``.
|
||||
|
||||
The dial still goes to the local proxy, which resolves and connects to the real target, so
|
||||
exempting a declared block grants no reach an attacker lacks through the proxy's own DNS.
|
||||
Undeclared ranges keep the ordinary private-address verdict.
|
||||
"""
|
||||
networks = _global_fake_ip_ranges()
|
||||
if not networks:
|
||||
return False
|
||||
if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None:
|
||||
ip = ip.ipv4_mapped
|
||||
return any(ip in net for net in networks)
|
||||
|
||||
|
||||
def _is_blocked_ip(ip: _IPAddress) -> bool:
|
||||
"""Return True if the IP should be blocked for SSRF protection."""
|
||||
# IPv4-mapped IPv6 (``::ffff:x.x.x.x``) is classified by its embedded IPv4.
|
||||
@@ -248,6 +311,10 @@ def _resolved_ip_block_reason(ip: _IPAddress, allow_private: bool) -> Optional[s
|
||||
if _is_always_blocked_ip(ip):
|
||||
return "cloud metadata address"
|
||||
if not allow_private and _is_blocked_ip(ip):
|
||||
# A fake-ip sentinel is the local proxy's own address, not an internal target — dialable either way.
|
||||
if _is_local_proxy_sentinel(ip):
|
||||
logger.debug("Allowing local-proxy fake-ip sentinel address: %s", ip)
|
||||
return None
|
||||
return "private/internal address"
|
||||
return None
|
||||
|
||||
|
||||
@@ -746,6 +746,27 @@ When on, web tools, the browser, vision URL fetches, and gateway media downloads
|
||||
|
||||
The host-substring guard (which blocks lookalike Unicode domain tricks even when the underlying IP is public) stays on regardless of this setting.
|
||||
|
||||
#### Local proxy fake-ip ranges
|
||||
|
||||
A TUN proxy in fake-ip mode (Mihomo/Clash `fake-ip`, Surge enhanced mode) answers DNS with an
|
||||
address from its own block — `198.18.0.0/15` (RFC 2544 benchmarking) by default — for every name
|
||||
outside its filter. Those answers are the proxy's sentinel, not an internal host, so the
|
||||
private-IP guard otherwise rejects every outbound fetch on such a host: `web_extract`, platform
|
||||
attachment downloads and the browser relay all fail with *URL targets a private or internal
|
||||
network address* while the request never reaches the network. Declare the block to let the
|
||||
sentinel through:
|
||||
|
||||
```yaml
|
||||
security:
|
||||
fake_ip_ranges:
|
||||
- 198.18.0.0/15
|
||||
```
|
||||
|
||||
Empty by default, and narrower than `allow_private_urls`: only the declared blocks get the
|
||||
exemption, they should be ranges the local proxy owns (the dial still goes to the proxy, which
|
||||
resolves the real target itself), and loopback, RFC 1918, link-local, CGNAT and cloud-metadata
|
||||
destinations stay blocked.
|
||||
|
||||
### Tirith Pre-Exec Security Scanning
|
||||
|
||||
Hermes integrates [tirith](https://github.com/sheeki03/tirith) for content-level command scanning before execution. Tirith detects threats that pattern matching alone misses:
|
||||
|
||||
@@ -523,6 +523,24 @@ security:
|
||||
|
||||
主机子字符串防护(即使底层 IP 是公共的,也能阻止 Unicode 同形字域名欺骗)无论此设置如何均保持开启。
|
||||
|
||||
#### 本地代理的 fake-ip 地址段
|
||||
|
||||
以 fake-ip 模式工作的 TUN 代理(Mihomo/Clash `fake-ip`、Surge 增强模式)会对不在其过滤器内的
|
||||
每个域名返回自己地址段中的地址——默认是 `198.18.0.0/15`(RFC 2544 基准测试段)。这些地址是代理
|
||||
的哨兵地址,而不是内网主机,因此私网 IP 守卫会在这种机器上拦掉全部出网抓取:`web_extract`、平台
|
||||
附件下载、浏览器链路都会以 *URL targets a private or internal network address* 失败,而请求根本
|
||||
没有发出。声明该地址段即可放行哨兵地址:
|
||||
|
||||
```yaml
|
||||
security:
|
||||
fake_ip_ranges:
|
||||
- 198.18.0.0/15
|
||||
```
|
||||
|
||||
默认为空,且比 `allow_private_urls` 更窄:只有被声明的地址段获得豁免,且应当是本地代理自己拥有的
|
||||
地址段(连接仍然发往代理,由代理自行解析真实目标),回环、RFC 1918、链路本地、CGNAT 和云元数据
|
||||
目标依然被拦截。
|
||||
|
||||
### Tirith 预执行安全扫描
|
||||
|
||||
Hermes 集成了 [tirith](https://github.com/sheeki03/tirith) 用于在执行前进行内容级命令扫描。Tirith 能检测单纯模式匹配所遗漏的威胁:
|
||||
|
||||
Reference in New Issue
Block a user