Smoke tests and the eval suite - PolicyCodex Support

Run top to bottom after a deploy, or start at the layer a report points to. $PCX_API_KEY is a real org API key (never reuse demo keys outside the demo estate).

1. Worker is up

curl -s https://mcp.policycodex.cloud/
# → route listing; anything else is a worker/DNS problem

2. REST auth + data path

curl -s -H "authorization: Bearer $PCX_API_KEY" \
  "https://mcp.policycodex.cloud/v1/statements?status=enforced" | head
# 401 → bad/revoked key · empty list → the KEY'S org has no enforced statements

3. The compliance brain (LLM round-trip)

The canned probes in demo/diffs/ are known-bad and known-good against the standard seeded corpus:

# known-bad: MUST return a blocking finding citing a statement slug
python3 -c 'import json; print(json.dumps({"diff": open("demo/diffs/violating.diff").read()}))' |
  curl -s -X POST "https://mcp.policycodex.cloud/v1/check" \
    -H "authorization: Bearer $PCX_API_KEY" -H "content-type: application/json" \
    -d @- | python3 -m json.tool

# known-good control: MUST return {"findings": []}
python3 -c 'import json; print(json.dumps({"diff": open("demo/diffs/compliant.diff").read()}))' |
  curl -s -X POST "https://mcp.policycodex.cloud/v1/check" \
    -H "authorization: Bearer $PCX_API_KEY" -H "content-type: application/json" \
    -d @- | python3 -m json.tool

Reading the result:

Probes with no repo in the body write no service_checks row — they never pollute Coverage/Reporting. Include repo/branch and they record like a CI run.
statements_considered is the size of the key’s org’s enforced corpus. A surprising number means you are holding the wrong key.
To audit a real check, replay its exact input: gh pr diff <n> --repo <owner>/<repo> piped through the same command.

4. The golden eval suite

The formalised version of step 3 — run it after any model change in Settings → LLM:

pnpm eval:compliance                    # all cases in evals/compliance/
pnpm eval:compliance hardcoded-secret   # one case by name

Each case is a diff.patch (plus optional tree.txt) and an expect.json — either {"clean": true} or must_find regexes resolved against the target org’s enforced statement text at runtime. No matching statement in the org → the case SKIPs (portable across orgs); a missed expectation FAILs as a false negative; extra findings never fail (review them — the model finding more real issues is not a defect). A weekly GitHub workflow (eval-compliance.yml) runs the same suite with the POLICYCODEX_EVAL_API_KEY repo secret.

5. Audit chain integrity

curl -s "https://mcp.policycodex.cloud/v1/audit/export?format=json" \
  -H "authorization: Bearer $PCX_API_KEY" > /tmp/audit.json
pnpm verify:audit /tmp/audit.json
# ✅ chain valid: N entries, seq 1..N — anything else is an incident

The verifier recomputes every entry_hash from content alone; it trusts nothing but the hashes it carries.

6. Email pipeline (growth)

pnpm test:emails          # render mode: validates templates + variable bindings,
                          # sends nothing
pnpm test:emails send     # real events to Resend sandbox addresses only

7. Demo estate

The console’s health grid is the smoke test (eight tiles). From a terminal, the equivalent is demo-api’s health action, or:

curl -s "https://mcp.policycodex.cloud/v1/domains" -H "Authorization: Bearer $ACME_DEMO_KEY"
# → 3 domains with RFC counts; 14 enforced statements expected at start-of-demo

Workspace definition of done

Any change: pnpm lint and pnpm test pass, and the thin-slice demo path still works end-to-end. That last clause is what the demo estate’s S1–S6 arc exercises.

​1. Worker is up

​2. REST auth + data path

​3. The compliance brain (LLM round-trip)

​4. The golden eval suite

​5. Audit chain integrity

​6. Email pipeline (growth)

​7. Demo estate

​Workspace definition of done