test(install-e2e): preserve plugins/** and profile plugin trees across real upgrades
Add a Hermes-owned upgrade-preservation contract to the install E2E harness: a tagged upgrade must not delete or modify anything under the active home's plugins/** or any profile's plugins/<name>/plugins tree. - tests/install/e2e-assets/verify-plugin-preservation.py: standalone stdlib-only READ-ONLY verifier. snapshot records every entry (kind, size + sha256, link target, recursive fingerprint of symlinked external targets so the externally-owned sidecar witness is covered) including empty dirs and the roots themselves (lstat, so dangling root links are still scanned); verify fails on deletion or modification, treats unreadable paths as hard errors, and refuses an empty snapshot as inconclusive. Runs under python3/python on all three driver platforms. - tests/install/e2e-assets/preserve-plugins.sh: POSIX/macOS hooks. After install: seed controlled non-dependency directory fixtures (mnemosyne-wrapper marker + payload + symlinked runtime, second profile plugin tree, external witness outside the home; no pyproject in the scanned root, nothing downloaded) and snapshot. After update: verify; violation fails the leg. Seeding never clobbers a populated wrapper without the expected marker. - installer-script-e2e.sh / macos-desktop-e2e.sh / windows-e2e.ps1: wire before/after hooks into the update leg, and add --update-ref (-UpdateRef on Windows) so a leg can target an actual stable tag instead of HEAD; HEAD-upgrade legs keep the HEAD label. Matrix and workflows unchanged. - tests/scripts/test_verify_plugin_preservation.py: 17 unit tests on a real temp filesystem covering clean survival, file/marker deletion and modification, whole-root deletion, empty-dir deletion, symlink repoint, external witness tamper, read-only guarantee, empty-snapshot refusal, unreadable-path hard error (POSIX), and the exact CLI round-trip the drivers use. - tests/install/README.md: document the contract, the hooks, and the stable-to-stable rule.
This commit is contained in:
@@ -81,6 +81,40 @@ Cost per run, so nobody is surprised: 41 legs per sampled tag (windows 18, macos
|
||||
|
||||
Running the drivers locally: don't, except in a disposable VM. The windows driver kills every process named Hermes during teardown and the macos driver operates on `/Applications/Hermes.app`; on a machine with a real Hermes install they will interfere with it.
|
||||
|
||||
## Plugin upgrade preservation
|
||||
|
||||
Every upgrade leg also carries a plugin-survival contract: a tagged upgrade
|
||||
must not delete or modify anything under the active home's `plugins/**` tree
|
||||
or any profile's `profiles/<name>/plugins/**` tree. Destructive flows
|
||||
(explicit uninstall, plugin removal, profile or user deletion) are out of
|
||||
contract and not exercised.
|
||||
|
||||
- `e2e-assets/verify-plugin-preservation.py` is the shared, stdlib-only,
|
||||
read-only verifier. `snapshot` records every entry (kind, byte size +
|
||||
sha256, link targets, and a recursive fingerprint of a symlink's external
|
||||
target) across all plugin roots, including empty directories and the roots
|
||||
themselves; `verify` diffs the live tree against that snapshot and fails
|
||||
on any deletion or modification. Unreadable paths are hard errors; an
|
||||
empty snapshot is refused as inconclusive rather than claimed as a pass.
|
||||
- `e2e-assets/preserve-plugins.sh` is the POSIX/macOS hook pair: after the
|
||||
install phase it seeds controlled, non-dependency directory fixtures (a
|
||||
`mnemosyne-wrapper` plugin with its marker, a symlinked runtime, a second
|
||||
profile plugin tree, and the externally-owned sidecar witness outside the
|
||||
home — no pyproject anywhere in the scanned root, nothing downloaded) and
|
||||
snapshots; after the update lands it verifies and fails the leg on any
|
||||
violation. Seeding is not a clobber: a populated wrapper without the
|
||||
expected marker aborts the leg. The Windows driver carries the same
|
||||
fixtures and hooks inline (`Seed-PreservationFixtures`,
|
||||
`Invoke-PreserveSnapshot`, `Invoke-PreserveVerify`).
|
||||
- Unit tests live at `tests/scripts/test_verify_plugin_preservation.py` and
|
||||
exercise the verifier against a real temp filesystem (real files, real
|
||||
symlinks; junction fallback on Windows).
|
||||
- Stable-to-stable: the drivers accept `--update-ref REF` (Windows:
|
||||
`-UpdateRef`), defaulting to HEAD. Pass the next release tag to target a
|
||||
stable→stable upgrade through the same serve.git staging; only label a leg
|
||||
stable-to-stable when BOTH the install ref and the target ref are release
|
||||
tags. The workflow matrix itself is unchanged.
|
||||
|
||||
## Artifacts
|
||||
|
||||
Each leg uploads its logs as an artifact. Every leg also records the screen for its whole run: the composite action `.github/actions/e2e-screen-record` installs ffmpeg, records with the OS's capture backend (x11grab on linux, gdigrab on windows, avfoundation on macos), and fails the leg if the recording is missing or has zero frames. Linux runners have no display, so the action starts `Xvfb :99` first and exports `DISPLAY` for every later step — the app under test and the recorder share that display. The windows GUI leg also uploads screenshots and the update result file. Get them with `gh run download <run-id>`.
|
||||
|
||||
105
tests/install/e2e-assets/preserve-plugins.sh
Normal file
105
tests/install/e2e-assets/preserve-plugins.sh
Normal file
@@ -0,0 +1,105 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared plugin upgrade-preservation hooks for the install E2E drivers
|
||||
# (POSIX + macOS). Sourced by tests/install/installer-script-e2e.sh and
|
||||
# tests/install/macos-desktop-e2e.sh.
|
||||
#
|
||||
# The contract under test: a tagged Hermes upgrade must NOT delete or modify
|
||||
# anything in the active home's plugins/** tree or any profile's plugins/**
|
||||
# tree — including directory wrapper markers (mnemosyne-wrapper.json),
|
||||
# symlinked runtimes, and the externally-owned sidecar witness file that
|
||||
# lives OUTSIDE the home. Destructive flows (explicit uninstall, plugin
|
||||
# removal, profile/user deletion) are out of contract and not exercised.
|
||||
#
|
||||
# The fixtures are deliberately non-dependency: directory wrappers with no
|
||||
# pyproject anywhere in the scanned root, so the plugin scanner never
|
||||
# recurses into a dependency graph and nothing is downloaded.
|
||||
#
|
||||
# State flow (called by the driver):
|
||||
# preserve_before_upgrade seed fixtures + snapshot -> $PRESERVE_SNAPSHOT
|
||||
# preserve_after_upgrade verify against snapshot (exit 1 on violation)
|
||||
#
|
||||
# Requires: HERMES_HOME set (the leg's isolated home), WORK_ROOT, LOG_DIR,
|
||||
# REPO_ROOT. Verifier: tests/install/e2e-assets/verify-plugin-preservation.py
|
||||
# (stdlib-only; runs under python3 or $HERMES_E2E_PYTHON).
|
||||
|
||||
PRESERVE_SNAPSHOT="$WORK_ROOT/plugin-preservation-snapshot.json"
|
||||
PRESERVE_REPORT="$LOG_DIR/plugin-preservation-report.json"
|
||||
PRESERVE_EXTERNAL="$WORK_ROOT/external-mnemosyne-runtime"
|
||||
|
||||
_preserve_python() {
|
||||
if [ -n "${HERMES_E2E_PYTHON:-}" ]; then
|
||||
printf '%s' "$HERMES_E2E_PYTHON"
|
||||
else
|
||||
printf 'python3'
|
||||
fi
|
||||
}
|
||||
|
||||
seed_plugin_preservation_fixtures() {
|
||||
# NOT a clobbering seeder: if the wrapper is already populated the fixture
|
||||
# state is left untouched, so a re-entered leg cannot erase a regression
|
||||
# the earlier phase recorded. Only a matching marker may be pre-existing.
|
||||
local home="$1" external="$2"
|
||||
local wrapper="$home/plugins/mnemosyne-wrapper"
|
||||
local marker="$wrapper/mnemosyne-wrapper.json"
|
||||
if [ -d "$wrapper" ] && [ -n "$(ls -A "$wrapper" 2>/dev/null)" ]; then
|
||||
grep -qF '"marker": "mnemosyne-wrapper"' "$marker" 2>/dev/null \
|
||||
|| fail "refusing to reseed preservation fixtures: $wrapper already populated without the expected marker"
|
||||
ok "preservation fixtures already present; left untouched"
|
||||
else
|
||||
mkdir -p "$wrapper"
|
||||
printf '{"wrapper": true, "marker": "mnemosyne-wrapper", "owner": "e2e-preservation"}\n' \
|
||||
> "$marker"
|
||||
printf '#!/usr/bin/env python3\n# directory wrapper fixture: no dependencies\nPLUGIN = "mnemosyne-wrapper"\n' \
|
||||
> "$wrapper/plugin.py"
|
||||
if [ ! -e "$wrapper/runtime" ] && [ ! -L "$wrapper/runtime" ]; then
|
||||
ln -s "$external" "$wrapper/runtime" \
|
||||
|| fail "could not create the runtime link at $wrapper/runtime; the preservation contract cannot be exercised"
|
||||
fi
|
||||
fi
|
||||
mkdir -p "$external" "$home/profiles/e2e-preserve/plugins/second-plugin"
|
||||
[ -e "$external/sidecar-witness.txt" ] \
|
||||
|| printf 'external-sidecar-witness-v1\n' > "$external/sidecar-witness.txt"
|
||||
[ -e "$external/engine.bin" ] \
|
||||
|| printf '\x00\x01\x02external-engine\n' > "$external/engine.bin"
|
||||
[ -e "$home/profiles/e2e-preserve/plugins/second-plugin/marker.json" ] \
|
||||
|| printf '{"plugin": "second-plugin", "profile": "e2e-preserve"}\n' \
|
||||
> "$home/profiles/e2e-preserve/plugins/second-plugin/marker.json"
|
||||
[ -e "$home/profiles/e2e-preserve/plugins/second-plugin/data.bin" ] \
|
||||
|| printf 'profile-plugin-bytes\n' \
|
||||
> "$home/profiles/e2e-preserve/plugins/second-plugin/data.bin"
|
||||
ok "seeded preservation fixtures: $wrapper (marker + runtime link) + profile tree; external witness at $external"
|
||||
}
|
||||
|
||||
preserve_before_upgrade() {
|
||||
step "plugin preservation: seeding fixtures and snapshotting pre-upgrade state"
|
||||
seed_plugin_preservation_fixtures "$HERMES_HOME" "$PRESERVE_EXTERNAL"
|
||||
local py; py="$(_preserve_python)"
|
||||
"$py" "$REPO_ROOT/tests/install/e2e-assets/verify-plugin-preservation.py" \
|
||||
snapshot --home "$HERMES_HOME" --out "$PRESERVE_SNAPSHOT" 2>&1 | ts_prefix \
|
||||
|| fail "plugin preservation snapshot failed"
|
||||
# An empty snapshot proves nothing; refuse to build the leg's claim on it.
|
||||
"$py" - "$PRESERVE_SNAPSHOT" <<'PYEOF' || fail "plugin preservation snapshot is EMPTY: no plugin entries recorded, cannot verify preservation"
|
||||
import json, sys
|
||||
with open(sys.argv[1], encoding="utf-8") as fh:
|
||||
snap = json.load(fh)
|
||||
n = len(snap.get("entries", {}))
|
||||
print(f"snapshot entries: {n}")
|
||||
raise SystemExit(0 if n else 3)
|
||||
PYEOF
|
||||
ok "pre-upgrade plugin snapshot at $PRESERVE_SNAPSHOT"
|
||||
}
|
||||
|
||||
preserve_after_upgrade() {
|
||||
step "plugin preservation: verifying post-upgrade state"
|
||||
[ -f "$PRESERVE_SNAPSHOT" ] \
|
||||
|| fail "no pre-upgrade plugin snapshot at $PRESERVE_SNAPSHOT; cannot verify preservation"
|
||||
local py; py="$(_preserve_python)"
|
||||
local rc=0
|
||||
"$py" "$REPO_ROOT/tests/install/e2e-assets/verify-plugin-preservation.py" \
|
||||
verify --home "$HERMES_HOME" --snapshot "$PRESERVE_SNAPSHOT" \
|
||||
--report "$PRESERVE_REPORT" > "$LOG_DIR/plugin-preservation-verify.log" 2>&1 || rc=$?
|
||||
log_group "plugin preservation verify transcript" "$LOG_DIR/plugin-preservation-verify.log"
|
||||
[ "$rc" -eq 0 ] \
|
||||
|| fail "plugin preservation violated by the upgrade: deleted/modified entries above; report at $PRESERVE_REPORT"
|
||||
ok "plugins/** and profile plugin trees (markers, symlinked runtimes, external sidecar witness) survived the upgrade intact"
|
||||
}
|
||||
287
tests/install/e2e-assets/verify-plugin-preservation.py
Normal file
287
tests/install/e2e-assets/verify-plugin-preservation.py
Normal file
@@ -0,0 +1,287 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Plugin upgrade-preservation verifier (Hermes release-harness hook).
|
||||
|
||||
Standalone, stdlib-only, READ-ONLY against the scanned home. Two modes:
|
||||
|
||||
snapshot walk every plugin tree of a HERMES_HOME (the active home's
|
||||
``plugins/**`` plus each ``profiles/<name>/plugins/**`` tree,
|
||||
overridable with --profiles-dir) and record, per entry —
|
||||
including EMPTY DIRECTORIES and the tree roots themselves —
|
||||
kind (file/dir/symlink/junction), byte size + sha256 for
|
||||
regular files, link target, and a recursive fingerprint of a
|
||||
symlink's external target (so an externally-owned sidecar
|
||||
witness file cannot be tampered with unnoticed) -> JSON file.
|
||||
verify re-walk the same home and diff against a snapshot. FAILS
|
||||
(exit 1) on any deleted or modified entry. New entries are
|
||||
reported but do not fail: an upgrade may add files; it may
|
||||
not take yours away or alter them.
|
||||
|
||||
Unreadable paths are a hard error (exit 2), never a silent skip: a
|
||||
scanner that cannot see a file cannot defend it. The verifier never
|
||||
creates, deletes or writes anything under the scanned home.
|
||||
|
||||
Usage:
|
||||
python verify-plugin-preservation.py snapshot --home <HERMES_HOME> --out snap.json
|
||||
python verify-plugin-preservation.py verify --home <HERMES_HOME> --snapshot snap.json [--report r.json]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import stat
|
||||
import sys
|
||||
|
||||
SCHEMA_VERSION = 2
|
||||
PROFILES_DIR = "profiles"
|
||||
PLUGIN_ROOT = "plugins"
|
||||
CHUNK = 1 << 20
|
||||
# Upper bound for the recursive fingerprint of a symlink's external target:
|
||||
# the harness points links at small controlled fixtures, but a stray link at
|
||||
# a huge tree must not explode the snapshot.
|
||||
MAX_EXTERNAL_ENTRIES = 10000
|
||||
|
||||
|
||||
class ScanError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def _sha256_file(path: str) -> str:
|
||||
h = hashlib.sha256()
|
||||
with open(path, "rb") as fh:
|
||||
while True:
|
||||
block = fh.read(CHUNK)
|
||||
if not block:
|
||||
break
|
||||
h.update(block)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def _is_link(path: str) -> bool:
|
||||
"""True for symlinks everywhere, and for NTFS junctions on Windows
|
||||
(st.islink() is False for junctions, but readlink() works). POSIX
|
||||
behavior is untouched: FILE_ATTRIBUTE_REPARSE_POINT only exists on NT."""
|
||||
if os.path.islink(path):
|
||||
return True
|
||||
st = os.lstat(path)
|
||||
reparse = getattr(st, "st_file_attributes", 0)
|
||||
return bool(reparse & stat.FILE_ATTRIBUTE_REPARSE_POINT)
|
||||
|
||||
|
||||
def _lstat_exists(path: str) -> bool:
|
||||
"""Exists including dangling symlinks/junctions (os.path.exists hides
|
||||
those by resolving)."""
|
||||
try:
|
||||
os.lstat(path)
|
||||
return True
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def plugin_roots(home: str, profiles_dir: str | None = None) -> list[str]:
|
||||
"""Every plugin tree that must survive an upgrade: the active home's
|
||||
plugins/ root plus each profile's plugins/ root (found under
|
||||
profiles_dir, default <home>/profiles). Roots are detected with lstat so
|
||||
a root that is itself a (possibly dangling) symlink is still scanned and
|
||||
fingerprinted. Returns stable "<home>/..." labels for existing roots."""
|
||||
roots: list[str] = []
|
||||
if _lstat_exists(os.path.join(home, PLUGIN_ROOT)):
|
||||
roots.append("<home>/" + PLUGIN_ROOT)
|
||||
profiles = profiles_dir or os.path.join(home, PROFILES_DIR)
|
||||
if os.path.isdir(profiles):
|
||||
for name in sorted(os.listdir(profiles)):
|
||||
p_root = os.path.join(profiles, name, PLUGIN_ROOT)
|
||||
if _lstat_exists(p_root):
|
||||
label = os.path.relpath(p_root, home).replace(os.sep, "/")
|
||||
roots.append("<home>/" + label)
|
||||
return roots
|
||||
|
||||
|
||||
def _fingerprint_tree(root: str, budget: list[int]) -> dict:
|
||||
"""Recursive content fingerprint of a directory (used for the EXTERNAL
|
||||
target of a plugin-runtime symlink: the sidecar witness is owned outside
|
||||
the home, so its content must be hashed here, not just listed)."""
|
||||
out: dict[str, dict] = {}
|
||||
def _on_walk_error(exc: OSError) -> None:
|
||||
raise ScanError(f"unreadable under {root}: {exc}") from exc
|
||||
for dirpath, dirnames, filenames in os.walk(root, onerror=_on_walk_error):
|
||||
dirnames.sort()
|
||||
for name in sorted(dirnames) + sorted(filenames):
|
||||
if budget[0] <= 0:
|
||||
raise ScanError(f"external target too large to fingerprint: {root}")
|
||||
budget[0] -= 1
|
||||
abs_path = os.path.join(dirpath, name)
|
||||
rel = os.path.relpath(abs_path, root).replace(os.sep, "/")
|
||||
if _is_link(abs_path):
|
||||
out[rel] = {"kind": "symlink", "target": os.readlink(abs_path)}
|
||||
elif os.path.isdir(abs_path):
|
||||
out[rel] = {"kind": "dir"}
|
||||
elif os.path.isfile(abs_path):
|
||||
st = os.lstat(abs_path)
|
||||
out[rel] = {"kind": "file", "size": st.st_size,
|
||||
"sha256": _sha256_file(abs_path)}
|
||||
else:
|
||||
out[rel] = {"kind": "other"}
|
||||
return out
|
||||
|
||||
|
||||
def _entry_record(abs_path: str) -> dict:
|
||||
if _is_link(abs_path):
|
||||
rec: dict = {"kind": "symlink", "target": os.readlink(abs_path)}
|
||||
resolved = os.path.realpath(abs_path)
|
||||
rec["target_resolves"] = os.path.exists(resolved)
|
||||
if rec["target_resolves"]:
|
||||
if os.path.isfile(resolved):
|
||||
rec["target_kind"] = "file"
|
||||
rec["target_sha256"] = _sha256_file(resolved)
|
||||
elif os.path.isdir(resolved):
|
||||
rec["target_kind"] = "dir"
|
||||
rec["target_tree"] = _fingerprint_tree(resolved, [MAX_EXTERNAL_ENTRIES])
|
||||
return rec
|
||||
if os.path.isdir(abs_path):
|
||||
return {"kind": "dir"}
|
||||
if os.path.isfile(abs_path):
|
||||
st = os.lstat(abs_path)
|
||||
return {"kind": "file", "size": st.st_size, "sha256": _sha256_file(abs_path)}
|
||||
return {"kind": "other"}
|
||||
|
||||
|
||||
def snapshot_home(home: str, profiles_dir: str | None = None) -> dict:
|
||||
home = os.path.abspath(home)
|
||||
roots = plugin_roots(home, profiles_dir)
|
||||
entries: dict[str, dict] = {}
|
||||
for root_label in roots:
|
||||
root = os.path.join(home, root_label[len("<home>/"):].replace("/", os.sep))
|
||||
# The root itself is an entry: if it is a symlink its identity and
|
||||
# target are recorded; if a plain dir it anchors empty-dir coverage.
|
||||
entries[root_label[len("<home>/"):]] = _entry_record(root)
|
||||
if _is_link(root):
|
||||
continue # walking through it would double-record its target
|
||||
def _on_walk_error(exc: OSError) -> None:
|
||||
raise ScanError(f"unreadable under {root}: {exc}") from exc
|
||||
for dirpath, dirnames, filenames in os.walk(root, followlinks=False,
|
||||
onerror=_on_walk_error):
|
||||
dirnames.sort()
|
||||
for name in sorted(dirnames) + sorted(filenames):
|
||||
abs_path = os.path.join(dirpath, name)
|
||||
rel = os.path.relpath(abs_path, home).replace(os.sep, "/")
|
||||
entries[rel] = _entry_record(abs_path)
|
||||
return {
|
||||
"schema": SCHEMA_VERSION,
|
||||
"home": home,
|
||||
"roots": roots,
|
||||
"entries": entries,
|
||||
}
|
||||
|
||||
|
||||
def verify_home(home: str, snap: dict) -> dict:
|
||||
home = os.path.abspath(home)
|
||||
now = snapshot_home(home, snap.get("_profiles_dir"))["entries"]
|
||||
before = snap["entries"]
|
||||
deleted = sorted(set(before) - set(now))
|
||||
added = sorted(set(now) - set(before))
|
||||
modified = {}
|
||||
for key in sorted(set(before) & set(now)):
|
||||
if before[key] != now[key]:
|
||||
modified[key] = {"before": before[key], "after": now[key]}
|
||||
return {
|
||||
"schema": SCHEMA_VERSION,
|
||||
"home": home,
|
||||
"snapshot_roots": snap["roots"],
|
||||
"counts": {
|
||||
"before": len(before),
|
||||
"after": len(now),
|
||||
"deleted": len(deleted),
|
||||
"modified": len(modified),
|
||||
"added": len(added),
|
||||
},
|
||||
"deleted": deleted,
|
||||
"modified": modified,
|
||||
"added": added,
|
||||
"ok": not deleted and not modified,
|
||||
}
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
sub = ap.add_subparsers(dest="mode", required=True)
|
||||
|
||||
p_snap = sub.add_parser("snapshot", help="record plugin-tree state to JSON")
|
||||
p_snap.add_argument("--home", required=True)
|
||||
p_snap.add_argument("--out", required=True)
|
||||
p_snap.add_argument("--profiles-dir",
|
||||
help="override the profiles root (default <home>/profiles)")
|
||||
|
||||
p_ver = sub.add_parser("verify", help="compare current state to a snapshot")
|
||||
p_ver.add_argument("--home", required=True)
|
||||
p_ver.add_argument("--snapshot", required=True)
|
||||
p_ver.add_argument("--profiles-dir")
|
||||
p_ver.add_argument("--report")
|
||||
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
if not os.path.isdir(os.path.abspath(args.home)):
|
||||
print(f"error: --home is not a directory: {args.home}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
if args.mode == "snapshot":
|
||||
try:
|
||||
snap = snapshot_home(args.home, args.profiles_dir)
|
||||
except (OSError, ScanError) as exc:
|
||||
print(f"snapshot failed: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
if args.profiles_dir:
|
||||
snap["_profiles_dir"] = args.profiles_dir
|
||||
with open(args.out, "w", encoding="utf-8") as fh:
|
||||
json.dump(snap, fh, indent=2, sort_keys=True)
|
||||
print(
|
||||
f"snapshot: {len(snap['entries'])} entries across "
|
||||
f"{len(snap['roots'])} plugin root(s) -> {args.out}"
|
||||
)
|
||||
if not snap["entries"]:
|
||||
print(
|
||||
"WARNING: snapshot recorded ZERO entries -- an empty snapshot "
|
||||
"proves nothing; the E2E driver must treat this as a failure.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 0
|
||||
|
||||
with open(args.snapshot, "r", encoding="utf-8") as fh:
|
||||
snap = json.load(fh)
|
||||
try:
|
||||
report = verify_home(args.home, snap)
|
||||
except (OSError, ScanError) as exc:
|
||||
print(f"verify failed: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
rendered = json.dumps(report, indent=2, sort_keys=True)
|
||||
if args.report:
|
||||
with open(args.report, "w", encoding="utf-8") as fh:
|
||||
fh.write(rendered)
|
||||
if not snap["entries"]:
|
||||
print(
|
||||
"PLUGIN PRESERVATION INCONCLUSIVE: the snapshot is empty; "
|
||||
"refusing to claim preservation over zero recorded entries.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 3
|
||||
if report["ok"]:
|
||||
c = report["counts"]
|
||||
print(
|
||||
f"plugin preservation OK: {c['before']} entries intact "
|
||||
f"({c['added']} added, 0 deleted, 0 modified)"
|
||||
)
|
||||
return 0
|
||||
print(rendered, file=sys.stderr)
|
||||
print(
|
||||
"PLUGIN PRESERVATION FAILED: "
|
||||
f"{report['counts']['deleted']} deleted, "
|
||||
f"{report['counts']['modified']} modified",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -40,6 +40,9 @@
|
||||
# drives it) and click Update now
|
||||
# --install-ref what to install first; anything git resolves. Default:
|
||||
# the newest release tag in the checkout.
|
||||
# --update-ref what to update TO. Default: HEAD. Pass the next release
|
||||
# tag for a stable-to-stable leg; only label the leg
|
||||
# stable-to-stable when BOTH refs are release tags.
|
||||
#
|
||||
# Requires a clean full-history checkout with release tags fetched.
|
||||
|
||||
@@ -53,6 +56,7 @@ export TS_BASE=$SECONDS
|
||||
INSTALL_METHOD="installer-script"
|
||||
UPDATE_METHOD=""
|
||||
INSTALL_REF=""
|
||||
UPDATE_REF=""
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
--install-method)
|
||||
@@ -64,6 +68,9 @@ while [ "$#" -gt 0 ]; do
|
||||
--install-ref)
|
||||
[ "$#" -ge 2 ] || { echo 'error: --install-ref needs a value' >&2; exit 1; }
|
||||
INSTALL_REF="$2"; shift 2 ;;
|
||||
--update-ref)
|
||||
[ "$#" -ge 2 ] || { echo 'error: --update-ref needs a value' >&2; exit 1; }
|
||||
UPDATE_REF="$2"; shift 2 ;;
|
||||
-h|--help) sed -n '2,45p' "$0"; exit 0 ;;
|
||||
*) echo "error: unknown argument: $1" >&2; exit 1 ;;
|
||||
esac
|
||||
@@ -92,6 +99,8 @@ ok() { printf ' OK %s\n' "$*"; }
|
||||
fail() { printf 'E2E ASSERTION FAILED: %s\n' "$*" >&2; exit 1; }
|
||||
# shellcheck source=../e2e-assets/ts-prefix.sh
|
||||
source "$(dirname "$0")/e2e-assets/ts-prefix.sh" 2>/dev/null || ts_prefix() { cat; }
|
||||
# shellcheck source=../e2e-assets/preserve-plugins.sh
|
||||
source "$(dirname "$0")/e2e-assets/preserve-plugins.sh"
|
||||
# Full transcript in the job log, collapsed (GitHub renders ::group:: as a
|
||||
# fold; plain text anywhere else). Win or lose -- a green install's log is
|
||||
# how you diagnose the leg that fails next.
|
||||
@@ -121,6 +130,16 @@ fi
|
||||
OLD_SHA="$(git -C "$REPO_ROOT" rev-parse "${INSTALL_REF}^{commit}")"
|
||||
HEAD_SHA="$(git -C "$REPO_ROOT" rev-parse HEAD)"
|
||||
[ "$OLD_SHA" != "$HEAD_SHA" ] || fail "OLD ($INSTALL_REF) IS HEAD; no update would be available"
|
||||
# The update target defaults to HEAD; --update-ref selects any other ref so
|
||||
# a stable-to-stable leg can target the next release tag instead of the tip.
|
||||
# Only call this leg stable-to-stable when BOTH refs are release tags.
|
||||
TARGET_LABEL="HEAD"
|
||||
TARGET_SHA="$HEAD_SHA"
|
||||
if [ -n "$UPDATE_REF" ]; then
|
||||
TARGET_SHA="$(git -C "$REPO_ROOT" rev-parse "${UPDATE_REF}^{commit}")"
|
||||
TARGET_LABEL="$UPDATE_REF"
|
||||
fi
|
||||
[ "$OLD_SHA" != "$TARGET_SHA" ] || fail "OLD ($INSTALL_REF) IS the update target ($TARGET_LABEL); no update would be available"
|
||||
|
||||
git clone --bare --quiet "$REPO_ROOT" "$SERVE_REPO"
|
||||
git -C "$SERVE_REPO" update-ref refs/heads/main "$OLD_SHA"
|
||||
@@ -322,12 +341,13 @@ else
|
||||
assert_checkout "$OLD_SHA" OLD
|
||||
fi
|
||||
smoke_desktop old
|
||||
preserve_before_upgrade
|
||||
|
||||
# --- update OLD -> HEAD ----------------------------------------------------------
|
||||
|
||||
step "advancing served main to HEAD"
|
||||
git -C "$SERVE_REPO" update-ref refs/heads/main "$HEAD_SHA"
|
||||
ok "serve.git main = $HEAD_SHA"
|
||||
step "advancing served main to $TARGET_LABEL ($TARGET_SHA)"
|
||||
git -C "$SERVE_REPO" update-ref refs/heads/main "$TARGET_SHA"
|
||||
ok "serve.git main = $TARGET_SHA"
|
||||
|
||||
step "updating via $UPDATE_METHOD"
|
||||
case "$UPDATE_METHOD" in
|
||||
@@ -348,11 +368,11 @@ case "$UPDATE_METHOD" in
|
||||
;;
|
||||
installer-script)
|
||||
# A user re-running the one-liner today gets the CURRENT script.
|
||||
run_installer "$HEAD_SHA" head
|
||||
run_installer "$TARGET_SHA" "$TARGET_LABEL"
|
||||
;;
|
||||
installer-script+desktop)
|
||||
run_installer "$HEAD_SHA" head desktop
|
||||
assert_desktop_artifact HEAD
|
||||
run_installer "$TARGET_SHA" "$TARGET_LABEL" desktop
|
||||
assert_desktop_artifact "$TARGET_LABEL"
|
||||
;;
|
||||
hermes-desktop-app-update)
|
||||
# The real user surface: `hermes desktop` launches the app, the user
|
||||
@@ -402,7 +422,7 @@ case "$UPDATE_METHOD" in
|
||||
(cd "$PW_DIR" && node launch-from-spec.mjs \
|
||||
--spec "$SPEC" \
|
||||
--result "$HERMES_HOME/.hermes-update-result.json" \
|
||||
--expect-sha "$HEAD_SHA" \
|
||||
--expect-sha "$TARGET_SHA" \
|
||||
--repo-dir "$INSTALL_DIR" 2>&1 \
|
||||
| ts_prefix > "$LOG_DIR/app-update.log") || rc=$?
|
||||
log_group "app update (Playwright) transcript" "$LOG_DIR/app-update.log"
|
||||
@@ -482,7 +502,9 @@ ls -la "$HERMES_HOME" > "$ildest/hermes-home-ls.txt" 2>/dev/null || true
|
||||
ls -la "$INSTALL_DIR/venv/bin" > "$ildest/venv-bin-ls.txt" 2>/dev/null || true
|
||||
ok "collected install-side logs to $ildest"
|
||||
|
||||
assert_checkout "$HEAD_SHA" HEAD
|
||||
smoke_desktop head
|
||||
assert_checkout "$TARGET_SHA" "$TARGET_LABEL"
|
||||
smoke_desktop "$TARGET_LABEL"
|
||||
|
||||
step "PASS: $INSTALL_REF -> HEAD via $UPDATE_METHOD"
|
||||
preserve_after_upgrade
|
||||
|
||||
step "PASS: $INSTALL_REF -> $TARGET_LABEL via $UPDATE_METHOD"
|
||||
|
||||
@@ -28,6 +28,9 @@
|
||||
# tests/install/macos-desktop-e2e.sh --phase stage|install|update|all
|
||||
# --update-method open-app-update|hermes-desktop-app-update
|
||||
# [--install-ref REF] [--dmg-url URL]
|
||||
# [--update-ref REF] update target, default HEAD; pass the next
|
||||
# release tag for a stable-to-stable leg (label the
|
||||
# leg stable-to-stable only when both refs are tags)
|
||||
#
|
||||
# Requires a clean full-history checkout with release tags fetched, on a
|
||||
# macOS host with a window server (the GitHub macos runners qualify).
|
||||
@@ -42,6 +45,7 @@ export TS_BASE=$SECONDS
|
||||
PHASE="all"
|
||||
UPDATE_METHOD=""
|
||||
INSTALL_REF=""
|
||||
UPDATE_REF=""
|
||||
DMG_URL="https://hermes-assets.nousresearch.com/Hermes-Setup.dmg"
|
||||
PLAYWRIGHT_VERSION="1.58.2"
|
||||
while [ "$#" -gt 0 ]; do
|
||||
@@ -55,6 +59,9 @@ while [ "$#" -gt 0 ]; do
|
||||
--install-ref)
|
||||
[ "$#" -ge 2 ] || { echo 'error: --install-ref needs a value' >&2; exit 1; }
|
||||
INSTALL_REF="$2"; shift 2 ;;
|
||||
--update-ref)
|
||||
[ "$#" -ge 2 ] || { echo 'error: --update-ref needs a value' >&2; exit 1; }
|
||||
UPDATE_REF="$2"; shift 2 ;;
|
||||
--dmg-url)
|
||||
[ "$#" -ge 2 ] || { echo 'error: --dmg-url needs a value' >&2; exit 1; }
|
||||
DMG_URL="$2"; shift 2 ;;
|
||||
@@ -84,6 +91,8 @@ ok() { printf ' OK %s\n' "$*"; }
|
||||
fail() { printf 'E2E ASSERTION FAILED: %s\n' "$*" >&2; exit 1; }
|
||||
# shellcheck source=../e2e-assets/ts-prefix.sh
|
||||
source "$(dirname "$0")/e2e-assets/ts-prefix.sh" 2>/dev/null || ts_prefix() { cat; }
|
||||
# shellcheck source=../install/e2e-assets/preserve-plugins.sh
|
||||
source "$(dirname "$0")/e2e-assets/preserve-plugins.sh"
|
||||
log_group() {
|
||||
printf '::group::%s\n' "$1"
|
||||
cat "$2"
|
||||
@@ -173,10 +182,20 @@ phase_stage() {
|
||||
old_ref="$(git -C "$REPO_ROOT" tag --list 'v[0-9]*' --sort=-creatordate | head -1)"
|
||||
[ -n "$old_ref" ] || fail "no release tags in the checkout to use as OLD"
|
||||
fi
|
||||
local old_sha head_sha
|
||||
local old_sha head_sha target_sha target_label
|
||||
old_sha="$(git -C "$REPO_ROOT" rev-parse "${old_ref}^{commit}")"
|
||||
head_sha="$(git -C "$REPO_ROOT" rev-parse HEAD)"
|
||||
[ "$old_sha" != "$head_sha" ] || fail "OLD ($old_ref) IS HEAD; no update would be available"
|
||||
# The update target defaults to HEAD; --update-ref selects any other ref
|
||||
# so a stable-to-stable leg can target the next release tag instead of
|
||||
# the tip. Only call this leg stable-to-stable when BOTH refs are tags.
|
||||
target_label="HEAD"
|
||||
target_sha="$head_sha"
|
||||
if [ -n "${UPDATE_REF:-}" ]; then
|
||||
target_sha="$(git -C "$REPO_ROOT" rev-parse "${UPDATE_REF}^{commit}")"
|
||||
target_label="$UPDATE_REF"
|
||||
fi
|
||||
[ "$old_sha" != "$target_sha" ] || fail "OLD ($old_ref) IS the update target ($target_label); no update would be available"
|
||||
|
||||
git clone --bare --quiet "$REPO_ROOT" "$SERVE_REPO"
|
||||
git -C "$SERVE_REPO" update-ref refs/heads/main "$old_sha"
|
||||
@@ -187,8 +206,9 @@ phase_stage() {
|
||||
mkdir -p "$HERMES_HOME"
|
||||
touch "$HERMES_HOME/.skip_upstream_prompt"
|
||||
|
||||
printf 'OLD_SHA=%s\nOLD_REF=%s\nHEAD_SHA=%s\n' "$old_sha" "$old_ref" "$head_sha" > "$STATE"
|
||||
ok "serve.git main = $old_sha ($old_ref), update target $head_sha"
|
||||
printf 'OLD_SHA=%s\nOLD_REF=%s\nHEAD_SHA=%s\nTARGET_SHA=%s\nTARGET_LABEL=%s\n' \
|
||||
"$old_sha" "$old_ref" "$head_sha" "$target_sha" "$target_label" > "$STATE"
|
||||
ok "serve.git main = $old_sha ($old_ref), update target $target_sha ($target_label)"
|
||||
}
|
||||
|
||||
find_installed_app() {
|
||||
@@ -311,7 +331,7 @@ run_playwright_update() {
|
||||
(cd "$pw_dir" && node launch-from-spec.mjs \
|
||||
--spec "$spec" \
|
||||
--result "$HERMES_HOME/.hermes-update-result.json" \
|
||||
--expect-sha "$HEAD_SHA" \
|
||||
--expect-sha "$TARGET_SHA" \
|
||||
--repo-dir "$INSTALL_DIR" 2>&1 \
|
||||
| ts_prefix > "$LOG_DIR/app-update.log") || rc=$?
|
||||
log_group "app update (Playwright) transcript" "$LOG_DIR/app-update.log"
|
||||
@@ -322,9 +342,12 @@ phase_update() {
|
||||
# shellcheck disable=SC1090
|
||||
. "$STATE"
|
||||
arm_redirect
|
||||
step "advancing served main to HEAD"
|
||||
git -C "$SERVE_REPO" update-ref refs/heads/main "$HEAD_SHA"
|
||||
ok "serve.git main = $HEAD_SHA"
|
||||
# Snapshot every plugin tree BEFORE the upgrade moves anything: fixtures
|
||||
# seeded here must survive through the verify after the update lands.
|
||||
preserve_before_upgrade
|
||||
step "advancing served main to $TARGET_LABEL ($TARGET_SHA)"
|
||||
git -C "$SERVE_REPO" update-ref refs/heads/main "$TARGET_SHA"
|
||||
ok "serve.git main = $TARGET_SHA"
|
||||
|
||||
step "updating via $UPDATE_METHOD"
|
||||
# The app must boot configured or the onboarding overlay (a fullscreen
|
||||
@@ -350,10 +373,10 @@ phase_update() {
|
||||
;;
|
||||
installer-script)
|
||||
# A dmg user re-running today's install one-liner.
|
||||
run_installer "$HEAD_SHA" head
|
||||
run_installer "$TARGET_SHA" head
|
||||
;;
|
||||
installer-script+desktop)
|
||||
run_installer "$HEAD_SHA" head desktop
|
||||
run_installer "$TARGET_SHA" head desktop
|
||||
# The desktop stage is this leg's claim: the rebuilt app must exist.
|
||||
head_app=""
|
||||
for cand in \
|
||||
@@ -403,8 +426,8 @@ PYEOF
|
||||
|
||||
local got
|
||||
got="$(git -C "$INSTALL_DIR" rev-parse HEAD)"
|
||||
[ "$got" = "$HEAD_SHA" ] || fail "checkout is $got, expected HEAD ($HEAD_SHA)"
|
||||
ok "checkout landed on HEAD ($HEAD_SHA)"
|
||||
[ "$got" = "$TARGET_SHA" ] || fail "checkout is $got, expected $TARGET_LABEL ($TARGET_SHA)"
|
||||
ok "checkout landed on $TARGET_LABEL ($TARGET_SHA)"
|
||||
|
||||
# Install-side state BEFORE the post-update smoke: on app-update legs the
|
||||
# updater's own transcript is streamed into the app UI and otherwise lost,
|
||||
@@ -426,7 +449,8 @@ PYEOF
|
||||
"$INSTALL_DIR/venv/bin/hermes" --version 2>&1 | ts_prefix > "$LOG_DIR/version-head.log" \
|
||||
|| fail "hermes --version failed after update"
|
||||
ok "hermes --version works post-update"
|
||||
step "PASS: $OLD_REF -> HEAD via $UPDATE_METHOD"
|
||||
preserve_after_upgrade
|
||||
step "PASS: $OLD_REF -> $TARGET_LABEL via $UPDATE_METHOD"
|
||||
}
|
||||
|
||||
case "$PHASE" in
|
||||
|
||||
@@ -98,6 +98,10 @@ param(
|
||||
# swallows an empty-string argument ('Missing an argument for
|
||||
# parameter'), so the workflow cannot pass "".
|
||||
[string]$InstallRef = "auto",
|
||||
# Update target ref (default HEAD). A stable-to-stable leg passes the
|
||||
# next release tag here; only label the leg stable-to-stable when BOTH
|
||||
# refs are release tags.
|
||||
[string]$UpdateRef = "HEAD",
|
||||
|
||||
# Repo checkout whose HEAD is the update target.
|
||||
[string]$RepoRoot = "",
|
||||
@@ -546,7 +550,8 @@ function Invoke-PhaseStage {
|
||||
# bare-clone below (and everything after) sees the redirect file.
|
||||
Set-GitRedirect
|
||||
|
||||
$current = Invoke-Git @("-C", $RepoRoot, "rev-parse", "HEAD")
|
||||
$current = Invoke-Git @("-C", $RepoRoot, "rev-parse", $UpdateRef)
|
||||
$targetLabel = if ($UpdateRef -eq "HEAD") { "HEAD" } else { $UpdateRef }
|
||||
Write-Host " HEAD (update target): $current"
|
||||
|
||||
# OLD: explicit -InstallRef, or the newest release tag -- the version a
|
||||
@@ -580,7 +585,7 @@ function Invoke-PhaseStage {
|
||||
Invoke-Git @("-C", $ServeRepo, "config", "uploadpack.allowAnySHA1InWant", "true") | Out-Null
|
||||
Write-Host " serve.git: uploadpack.allowAnySHA1InWant=true (installer commit pin, if any)"
|
||||
|
||||
@{ old = $old; old_ref = $oldRef; current = $current } |
|
||||
@{ old = $old; old_ref = $oldRef; current = $current; target_label = $targetLabel } |
|
||||
ConvertTo-Json | Set-Content -LiteralPath $StatePath -Encoding UTF8
|
||||
Write-Host " state written: $StatePath"
|
||||
New-Item -ItemType Directory -Path $ProofRoot -Force | Out-Null
|
||||
@@ -888,6 +893,75 @@ function Invoke-GuiUpdateDesktopRoute([string]$TargetSha) {
|
||||
}
|
||||
}
|
||||
|
||||
# --- plugin upgrade-preservation hooks -------------------------------------
|
||||
# A tagged upgrade must not delete or modify anything under the active
|
||||
# home's plugins/** or any profile's plugins/** tree: wrapper markers
|
||||
# (mnemosyne-wrapper.json), symlinked runtimes, and the externally-owned
|
||||
# sidecar witness outside the home. Fixtures are directory-only (no
|
||||
# pyproject in the scanned root, nothing downloaded). Snapshot is taken
|
||||
# after install, verified after update.
|
||||
function Seed-PreservationFixtures {
|
||||
$wrapper = Join-Path $HermesHome "plugins\mnemosyne-wrapper"
|
||||
$external = Join-Path $WorkRoot "external-mnemosyne-runtime"
|
||||
$marker = Join-Path $wrapper "mnemosyne-wrapper.json"
|
||||
# NOT a clobbering seeder: pre-existing populated wrapper is left
|
||||
# untouched so a retried leg cannot erase a recorded regression.
|
||||
if (Test-Path -LiteralPath $wrapper) {
|
||||
$existing = @(Get-ChildItem -LiteralPath $wrapper -ErrorAction SilentlyContinue)
|
||||
if ($existing.Count -gt 0) {
|
||||
$markerOk = (Test-Path -LiteralPath $marker) -and
|
||||
((Get-Content -LiteralPath $marker -Raw) -match "mnemosyne-wrapper")
|
||||
if (-not $markerOk) {
|
||||
throw "refusing to reseed preservation fixtures: $wrapper already populated without the expected marker"
|
||||
}
|
||||
Write-Host " preservation fixtures already present; left untouched"
|
||||
return
|
||||
}
|
||||
}
|
||||
New-Item -ItemType Directory -Path $wrapper, $external, `
|
||||
(Join-Path $HermesHome "profiles\e2e-preserve\plugins\second-plugin") -Force | Out-Null
|
||||
Set-Content -LiteralPath (Join-Path $wrapper "mnemosyne-wrapper.json") `
|
||||
-Value '{"wrapper": true, "marker": "mnemosyne-wrapper", "owner": "e2e-preservation"}' -Encoding UTF8
|
||||
Set-Content -LiteralPath (Join-Path $wrapper "plugin.py") `
|
||||
-Value '# directory wrapper fixture: no dependencies' + [Environment]::NewLine + 'PLUGIN = "mnemosyne-wrapper"' -Encoding UTF8
|
||||
if (-not (Test-Path -LiteralPath (Join-Path $wrapper "runtime"))) {
|
||||
try {
|
||||
New-Item -ItemType SymbolicLink -Path (Join-Path $wrapper "runtime") -Target $external -ErrorAction Stop | Out-Null
|
||||
} catch {
|
||||
Write-Host " (symlink unavailable: $($_.Exception.Message); runtime link omitted from fixtures)"
|
||||
}
|
||||
}
|
||||
Set-Content -LiteralPath (Join-Path $external "sidecar-witness.txt") -Value "external-sidecar-witness-v1" -Encoding UTF8
|
||||
Set-Content -LiteralPath (Join-Path $external "engine.bin") -Value "external-engine" -Encoding UTF8
|
||||
Set-Content -LiteralPath (Join-Path $HermesHome "profiles\e2e-preserve\plugins\second-plugin\marker.json") `
|
||||
-Value '{"plugin": "second-plugin", "profile": "e2e-preserve"}' -Encoding UTF8
|
||||
Set-Content -LiteralPath (Join-Path $HermesHome "profiles\e2e-preserve\plugins\second-plugin\data.bin") `
|
||||
-Value "profile-plugin-bytes" -Encoding UTF8
|
||||
Write-Host " preservation fixtures seeded (wrapper + profile tree; external witness at $external)"
|
||||
}
|
||||
|
||||
function Invoke-PreserveSnapshot {
|
||||
Seed-PreservationFixtures
|
||||
$out = Join-Path $WorkRoot "plugin-preservation-snapshot.json"
|
||||
& python (Join-Path $AssetsDir "verify-plugin-preservation.py") snapshot --home $HermesHome --out $out
|
||||
if ($LASTEXITCODE -ne 0) { throw "plugin preservation snapshot failed (exit $LASTEXITCODE)" }
|
||||
# An empty snapshot proves nothing; refuse to build the leg's claim on it.
|
||||
$snap = Get-Content -LiteralPath $out -Raw | ConvertFrom-Json
|
||||
if (@($snap.entries.PSObject.Properties).Count -eq 0) {
|
||||
throw "plugin preservation snapshot is EMPTY: no plugin entries recorded, cannot verify preservation"
|
||||
}
|
||||
Write-Host " pre-upgrade plugin snapshot: $out"
|
||||
}
|
||||
|
||||
function Invoke-PreserveVerify {
|
||||
$snap = Join-Path $WorkRoot "plugin-preservation-snapshot.json"
|
||||
if (-not (Test-Path -LiteralPath $snap)) { throw "no pre-upgrade plugin snapshot at $snap; cannot verify preservation" }
|
||||
& python (Join-Path $AssetsDir "verify-plugin-preservation.py") verify --home $HermesHome --snapshot $snap `
|
||||
--report (Join-Path $WorkRoot "logs\plugin-preservation-report.json")
|
||||
if ($LASTEXITCODE -ne 0) { throw "plugin preservation violated by the upgrade (exit $LASTEXITCODE); see the report for deleted/modified entries" }
|
||||
Write-Host " plugins/** and profile plugin trees survived the upgrade intact"
|
||||
}
|
||||
|
||||
function Invoke-PhaseInstall {
|
||||
# Dispatch on the install axis. Each arm ends with the same contract:
|
||||
# checkout at OLD, hermes runs, and state carries how OLD landed so any
|
||||
@@ -930,6 +1004,8 @@ function Invoke-PhaseUpdate {
|
||||
# remote's main moves forward. The GUI route re-advances harmlessly
|
||||
# (same sha); script routes need it here because only the GUI arm's
|
||||
# helper used to own this step.
|
||||
# Snapshot every plugin tree BEFORE the upgrade moves anything.
|
||||
Invoke-PreserveSnapshot
|
||||
Invoke-Git @("-C", $ServeRepo, "update-ref", "refs/heads/main", $state.current) | Out-Null
|
||||
Write-Host " serve.git main advanced to $($state.current)"
|
||||
|
||||
@@ -975,6 +1051,7 @@ function Invoke-PhaseUpdate {
|
||||
|
||||
Assert-True ((Get-InstalledHead) -eq $state.current) "checkout landed on HEAD"
|
||||
Test-HermesRuns "post-update"
|
||||
Invoke-PreserveVerify
|
||||
}
|
||||
|
||||
function Invoke-CheckedPhaseUpdate {
|
||||
|
||||
306
tests/scripts/test_verify_plugin_preservation.py
Normal file
306
tests/scripts/test_verify_plugin_preservation.py
Normal file
@@ -0,0 +1,306 @@
|
||||
"""Unit tests for the plugin upgrade-preservation verifier.
|
||||
|
||||
tests/install/e2e-assets/verify-plugin-preservation.py is the standalone
|
||||
hook the release E2E drivers call before and after a real upgrade. These
|
||||
tests exercise it against a real temp HERMES_HOME (real files, real
|
||||
symlinks) — no source-reading, no mocks of the filesystem.
|
||||
|
||||
The verifier must be read-only against the scanned home and must catch
|
||||
deletion and modification of every recorded entry kind: regular files,
|
||||
wrapper markers, directory trees, symlinks (identity + target), and the
|
||||
externally-owned sidecar witness file a symlinked plugin runtime points at.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
VERIFIER = os.path.join(
|
||||
_HERE, "..", "install", "e2e-assets", "verify-plugin-preservation.py"
|
||||
)
|
||||
|
||||
_spec = importlib.util.spec_from_file_location("verify_plugin_preservation", VERIFIER)
|
||||
vpp = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(vpp)
|
||||
|
||||
|
||||
def _make_link(target, link):
|
||||
try:
|
||||
os.symlink(str(target), str(link))
|
||||
except OSError:
|
||||
# Windows without symlink privilege: same reparse-point shape.
|
||||
import _winapi
|
||||
|
||||
_winapi.CreateJunction(str(target), str(link))
|
||||
|
||||
|
||||
def _remove_link(link):
|
||||
if os.path.islink(str(link)):
|
||||
os.remove(str(link))
|
||||
else: # NTFS junction
|
||||
os.rmdir(str(link))
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def home(tmp_path):
|
||||
"""A controlled temp HERMES_HOME with a non-dependency directory wrapper
|
||||
plugin: marker + payload + a symlink to an external runtime whose witness
|
||||
file lives OUTSIDE the home (externally-owned), plus a second plugin in a
|
||||
profile tree. No pyproject anywhere in the scanned root — the fixture is
|
||||
directory-only, so the scanner cannot recurse into a dependency graph and
|
||||
the test needs no network/Torch."""
|
||||
h = tmp_path / "hermes-home"
|
||||
# active-home plugin: directory wrapper with marker + payload
|
||||
plugin = h / "plugins" / "mnemosyne-wrapper"
|
||||
plugin.mkdir(parents=True)
|
||||
(plugin / "mnemosyne-wrapper.json").write_text('{"wrapper": true}\n', encoding="utf-8")
|
||||
(plugin / "plugin.py").write_bytes(b"PAYLOAD-BYTES-0\n")
|
||||
# external runtime, owned outside the home, reached through a symlink
|
||||
external = tmp_path / "external-mnemosyne-runtime"
|
||||
external.mkdir()
|
||||
(external / "sidecar-witness.txt").write_text("external-witness-v1\n", encoding="utf-8")
|
||||
(external / "engine.bin").write_bytes(b"\x00\x01\x02")
|
||||
_make_link(external, plugin / "runtime")
|
||||
# profile plugin tree
|
||||
pplugin = h / "profiles" / "work" / "plugins" / "second-plugin"
|
||||
pplugin.mkdir(parents=True)
|
||||
(pplugin / "marker.json").write_text('{"p": 1}\n', encoding="utf-8")
|
||||
(pplugin / "data.bin").write_bytes(b"profile-bytes\n")
|
||||
return h
|
||||
|
||||
|
||||
def _snapshot(home, out):
|
||||
snap = vpp.snapshot_home(str(home))
|
||||
with open(out, "w", encoding="utf-8") as fh:
|
||||
json.dump(snap, fh)
|
||||
return snap
|
||||
|
||||
|
||||
def _verify(home, snap):
|
||||
return vpp.verify_home(str(home), snap)
|
||||
|
||||
|
||||
def test_snapshot_records_all_trees(home, tmp_path):
|
||||
snap = _snapshot(home, tmp_path / "snap.json")
|
||||
keys = set(snap["entries"])
|
||||
assert "plugins/mnemosyne-wrapper/mnemosyne-wrapper.json" in keys
|
||||
assert "plugins/mnemosyne-wrapper/plugin.py" in keys
|
||||
assert "profiles/work/plugins/second-plugin/data.bin" in keys
|
||||
assert snap["roots"] == [
|
||||
"<home>/plugins",
|
||||
"<home>/profiles/work/plugins",
|
||||
]
|
||||
|
||||
|
||||
def test_snapshot_captures_bytes_and_symlinks(home, tmp_path):
|
||||
snap = _snapshot(home, tmp_path / "snap.json")
|
||||
e = snap["entries"]
|
||||
assert e["plugins/mnemosyne-wrapper/plugin.py"]["sha256"] != ""
|
||||
assert e["plugins/mnemosyne-wrapper/plugin.py"]["size"] == 16
|
||||
link = e["plugins/mnemosyne-wrapper/runtime"]
|
||||
assert link["kind"] == "symlink"
|
||||
assert link["target_resolves"] is True
|
||||
assert link["target_kind"] == "dir"
|
||||
tree = link["target_tree"]
|
||||
assert tree["sidecar-witness.txt"]["sha256"] != ""
|
||||
assert tree["engine.bin"]["kind"] == "file"
|
||||
|
||||
|
||||
def test_untouched_home_verifies_clean(home, tmp_path):
|
||||
snap = _snapshot(home, tmp_path / "snap.json")
|
||||
report = _verify(home, snap)
|
||||
assert report["ok"] is True
|
||||
assert report["counts"]["deleted"] == 0
|
||||
assert report["counts"]["modified"] == 0
|
||||
|
||||
|
||||
def test_catches_plugin_file_deletion(home, tmp_path):
|
||||
snap = _snapshot(home, tmp_path / "snap.json")
|
||||
(home / "plugins" / "mnemosyne-wrapper" / "plugin.py").unlink()
|
||||
report = _verify(home, snap)
|
||||
assert report["ok"] is False
|
||||
assert "plugins/mnemosyne-wrapper/plugin.py" in report["deleted"]
|
||||
|
||||
|
||||
def test_catches_whole_plugin_root_deletion(home, tmp_path):
|
||||
snap = _snapshot(home, tmp_path / "snap.json")
|
||||
for root, dirs, files in os.walk(home / "plugins", topdown=False):
|
||||
for name in files:
|
||||
os.remove(os.path.join(root, name))
|
||||
for name in dirs:
|
||||
os.rmdir(os.path.join(root, name))
|
||||
os.rmdir(home / "plugins")
|
||||
report = _verify(home, snap)
|
||||
assert report["ok"] is False
|
||||
assert report["counts"]["deleted"] > 0
|
||||
|
||||
|
||||
def test_catches_wrapper_marker_modification(home, tmp_path):
|
||||
snap = _snapshot(home, tmp_path / "snap.json")
|
||||
(home / "plugins" / "mnemosyne-wrapper" / "mnemosyne-wrapper.json").write_text(
|
||||
'{"wrapper": false}\n', encoding="utf-8"
|
||||
)
|
||||
report = _verify(home, snap)
|
||||
assert report["ok"] is False
|
||||
assert "plugins/mnemosyne-wrapper/mnemosyne-wrapper.json" in report["modified"]
|
||||
|
||||
|
||||
def test_catches_symlink_target_repoint(home, tmp_path):
|
||||
snap = _snapshot(home, tmp_path / "snap.json")
|
||||
link = home / "plugins" / "mnemosyne-wrapper" / "runtime"
|
||||
other = tmp_path / "other-runtime"
|
||||
other.mkdir()
|
||||
(other / "sidecar-witness.txt").write_text("different\n", encoding="utf-8")
|
||||
_remove_link(link)
|
||||
_make_link(other, link)
|
||||
report = _verify(home, snap)
|
||||
assert report["ok"] is False
|
||||
assert "plugins/mnemosyne-wrapper/runtime" in report["modified"]
|
||||
|
||||
|
||||
def test_catches_external_witness_modification(home, tmp_path):
|
||||
# The externally-owned sidecar witness lives OUTSIDE the home; an upgrade
|
||||
# that tramples it must still be caught through the symlink fingerprint.
|
||||
snap = _snapshot(home, tmp_path / "snap.json")
|
||||
witness = tmp_path / "external-mnemosyne-runtime" / "sidecar-witness.txt"
|
||||
witness.write_text("external-witness-TAMPERED\n", encoding="utf-8")
|
||||
report = _verify(home, snap)
|
||||
assert report["ok"] is False
|
||||
# Depending on the platform's walk, the tamper surfaces either as the
|
||||
# link entry (target fingerprint) or as the linked file itself.
|
||||
assert any(
|
||||
"runtime" in key for key in list(report["modified"]) + report["deleted"]
|
||||
)
|
||||
|
||||
|
||||
def test_catches_external_witness_deletion(home, tmp_path):
|
||||
snap = _snapshot(home, tmp_path / "snap.json")
|
||||
(tmp_path / "external-mnemosyne-runtime" / "engine.bin").unlink()
|
||||
report = _verify(home, snap)
|
||||
assert report["ok"] is False
|
||||
|
||||
|
||||
def test_catches_profile_plugin_deletion(home, tmp_path):
|
||||
snap = _snapshot(home, tmp_path / "snap.json")
|
||||
(home / "profiles" / "work" / "plugins" / "second-plugin" / "data.bin").unlink()
|
||||
report = _verify(home, snap)
|
||||
assert report["ok"] is False
|
||||
assert "profiles/work/plugins/second-plugin/data.bin" in report["deleted"]
|
||||
|
||||
|
||||
def test_added_entries_do_not_fail(home, tmp_path):
|
||||
# An upgrade may ADD files (new bundled plugin, caches); only taking
|
||||
# away or changing existing entries is a violation.
|
||||
snap = _snapshot(home, tmp_path / "snap.json")
|
||||
newp = home / "plugins" / "fresh-from-upgrade"
|
||||
newp.mkdir()
|
||||
(newp / "b.txt").write_text("new\n", encoding="utf-8")
|
||||
report = _verify(home, snap)
|
||||
assert report["ok"] is True
|
||||
assert "plugins/fresh-from-upgrade/b.txt" in report["added"]
|
||||
|
||||
|
||||
def test_verifier_is_read_only_against_home(home, tmp_path):
|
||||
snap = _snapshot(home, tmp_path / "snap.json")
|
||||
before = sorted(
|
||||
(p, p.stat().st_size if p.is_file() else "dir")
|
||||
for p in home.rglob("*")
|
||||
)
|
||||
_verify(home, snap)
|
||||
_verify(home, snap)
|
||||
after = sorted(
|
||||
(p, p.stat().st_size if p.is_file() else "dir")
|
||||
for p in home.rglob("*")
|
||||
)
|
||||
assert before == after
|
||||
|
||||
|
||||
def test_catches_empty_dir_deletion(home, tmp_path):
|
||||
# A plugin directory emptied (or an empty dir removed) must be caught:
|
||||
# directories themselves are recorded, not skipped.
|
||||
snap = _snapshot(home, tmp_path / "snap.json")
|
||||
empty = home / "plugins" / "wrapper-b" / "empty-cache"
|
||||
empty.mkdir(parents=True)
|
||||
snap2 = _snapshot(home, tmp_path / "snap2.json")
|
||||
assert snap2["entries"]["plugins/wrapper-b/empty-cache"] == {"kind": "dir"}
|
||||
empty.rmdir()
|
||||
(home / "plugins" / "wrapper-b").rmdir()
|
||||
report = _verify(home, snap2)
|
||||
assert report["ok"] is False
|
||||
assert "plugins/wrapper-b/empty-cache" in report["deleted"]
|
||||
|
||||
|
||||
def test_empty_snapshot_is_inconclusive(home, tmp_path):
|
||||
# Zero recorded entries cannot prove anything: the CLI refuses.
|
||||
empty_home = tmp_path / "bare-home"
|
||||
empty_home.mkdir()
|
||||
snap_file = tmp_path / "empty-snap.json"
|
||||
r1 = subprocess.run(
|
||||
[sys.executable, VERIFIER, "snapshot", "--home", str(empty_home),
|
||||
"--out", str(snap_file)],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
assert r1.returncode == 0
|
||||
assert "ZERO entries" in r1.stderr
|
||||
r2 = subprocess.run(
|
||||
[sys.executable, VERIFIER, "verify", "--home", str(empty_home),
|
||||
"--snapshot", str(snap_file)],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
assert r2.returncode == 3
|
||||
assert "INCONCLUSIVE" in r2.stderr
|
||||
|
||||
|
||||
def test_unreadable_path_is_hard_error(home, tmp_path):
|
||||
# A scanner that cannot see a path must fail loudly, not skip silently.
|
||||
# Skip where chmod-based unreadability is not enforceable (Windows).
|
||||
if os.name != "posix":
|
||||
pytest.skip("chmod-based unreadability is POSIX-only")
|
||||
snap = _snapshot(home, tmp_path / "snap.json")
|
||||
secret = home / "plugins" / "mnemosyne-wrapper" / "locked"
|
||||
secret.mkdir()
|
||||
(secret / "x.txt").write_text("data", encoding="utf-8")
|
||||
os.chmod(secret, 0o000)
|
||||
try:
|
||||
with pytest.raises(OSError):
|
||||
_snapshot(home, tmp_path / "snap2.json")
|
||||
finally:
|
||||
os.chmod(secret, 0o755)
|
||||
|
||||
|
||||
def test_missing_home_fails_snapshot(tmp_path):
|
||||
proc = subprocess.run(
|
||||
[sys.executable, VERIFIER, "snapshot", "--home", str(tmp_path / "nope"),
|
||||
"--out", str(tmp_path / "x.json")],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
assert proc.returncode == 2
|
||||
|
||||
|
||||
def test_cli_roundtrip_end_to_end(home, tmp_path):
|
||||
"""The exact command shape the E2E drivers use."""
|
||||
snap_file = tmp_path / "snap.json"
|
||||
r1 = subprocess.run(
|
||||
[sys.executable, VERIFIER, "snapshot", "--home", str(home), "--out", str(snap_file)],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
assert r1.returncode == 0, r1.stderr
|
||||
r2 = subprocess.run(
|
||||
[sys.executable, VERIFIER, "verify", "--home", str(home), "--snapshot", str(snap_file)],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
assert r2.returncode == 0, r2.stderr
|
||||
(home / "plugins" / "mnemosyne-wrapper" / "plugin.py").unlink()
|
||||
r3 = subprocess.run(
|
||||
[sys.executable, VERIFIER, "verify", "--home", str(home), "--snapshot", str(snap_file)],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
assert r3.returncode == 1
|
||||
assert "PLUGIN PRESERVATION FAILED" in r3.stderr
|
||||
Reference in New Issue
Block a user