Eval harness

Offline fixture recall and precision checks that keep stack isolation and planted-weakness detectors honest.

scripts/eval-audit.test.ts is an offline smoke harness. It drives the real secure-mcp server in memory against the committed fixtures and checks two properties that keep the product honest:

Recall floor — a planted weakness in a fixture must surface as a candidate finding in the expected detector family and category.
Precision smoke — a known-clean or differently-stacked fixture must not spam cross-stack detector families, recommend the wrong knowledge packs, or cite clean files as findings.

Run it

pnpm exec tsx --test scripts/eval-audit.test.ts

The full pnpm test suite includes this harness. No network and no spawned process are required; it uses an in-memory MCP transport with SECURE_MCP_ALLOWED_ROOTS scoped to fixtures/.

The label contract

Expectations live in scripts/eval-fixtures.ts (not in fixture source). Each fixture declares:

Field	Meaning
`stacks_include`	Detected stack labels that must be present
`required_rule_families`	Detector families that must appear (recall floor)
`required_categories`	Finding categories that must appear (recall floor)
`forbidden_rule_families`	Detector families that must not appear
`forbidden_packs`	Recommended pack ids that must not appear
`clean_files`	Fixture-relative paths that must not be cited
`expect_zero_findings`	The fixture must produce no candidates at all

Interpreting failures

A recall failure (missing required rule_family ...) means a known weakness stopped surfacing as a candidate. This is a detector regression and must be investigated, not silenced by lowering the floor.
A precision failure (unexpected forbidden rule_family/pack, or a clean file cited) means stack isolation regressed or a false positive appeared.
tiny-app is a mixed fixture on purpose (it contains ios/Secrets.swift), so Swift families may legitimately appear. Its forbidden set only excludes the Expo detector families and the expo-rn pack.

These are smoke numbers, not a benchmark badge. Do not put recall or precision figures into README marketing until the harness runs on real repositories and the numbers are reproducible across stacks.