version: "1.0.1" name: dev-testing description: Capsem testing policy and workflow. Use whenever running tests, writing new tests, or verifying changes work. Covers the three test tiers (unit, smoke, full), TDD red-green-refactor, adversarial security testing, coverage policy, and the mandatory end-to-end VM validation. For VM-specific tests see dev-testing-vm, for hypervisor tests see dev-testing-hypervisor, for frontend tests see dev-testing-frontend.

Testing

Read tests/README.md before adding or moving test fixtures. Test-only config belongs under tests/fixtures/, not root config/.

Test tiers

Three tiers, fast to thorough. Every change must pass all three before it ships.

just test is the single source of truth. There is no "fast" tier that skips integration tests -- that's how the "Connection refused" bug shipped while tests said green. Individual test-* recipes exist for targeted debugging but just test is the gate.

The full gate is a candidate boundary, not the edit loop. During TDD, reproduce the failure with the smallest focused test, run that test red/green, and batch adjacent parity fixes before paying for the complete gate. After a failed qualification, do not restart a 50-minute just test for each small edit. Run it once when the forward-fix candidate is ready; any later production or gate change invalidates that candidate and requires one new complete run.

Release CI invariant

Every stable and nightly release must execute just test in CI for the exact versioned, untagged candidate SHA before an immutable tag exists. A local gate, a prior or nearby commit's green workflow, a matching title with a different headSha, or a release-specific subset is not evidence and must never unblock tag creation or publication. release-qualification.yaml must call just test so additions to the canonical gate automatically become mandatory.

Temporary hosted-CI exception: the full gate runs once on Linux because GitHub-hosted macOS lacks the nested Virtualization.framework access needed by Capsem VM tests and Colima, and no physical macOS runner is registered. Keep this limitation explicitly commented in release-qualification.yaml. The tagged workflow must verify the exact successful qualification before macOS and Linux package jobs fan out. Restore a parallel macOS just test job when a physical runner becomes available.

Both macOS and Linux must install their exact publishable native packages, including their real post-install scripts, before publication. Notarization and the public stable-to-nightly switch/upgrade glow-up remain mandatory afterward as the end-to-end deployed-release test. Do not duplicate just test after tagging or packaging.

Linux proof is host-aware: a cross-built non-host package receives structural validation in qualification and exact native installation in its tagged release job. CAPSEM_REQUIRE_LINUX_DEB_PROOF=1 must not reject that non-host package before the host package reaches its mandatory KVM proof. The hosted arm64 runner does not expose /dev/kvm, so it proves exact package/service operation while the x86_64 runner additionally owns the guest-shell marker.

Expensive harnesses need a cheap clean-environment bootstrap proof at the start of just test. The Linux install rail must build the real install-test image, use a container-owned UV_PROJECT_ENVIRONMENT, and prove python -m pytest launches before audits, builds, VMs, or package assembly consume hours. Keep an ordering contract for this Stage 0 proof. It is fail-fast infrastructure validation only: the later Docker/systemd install E2E remains mandatory and must still exercise the installed package and post-install behavior.

Local/CI execution parity

Ironbank parity rule

The Ironbank parity rule is that every portable release gate must be owned by just test. A release candidate is not qualified until the complete just test passes locally and exact-SHA CI runs that same recipe successfully. Specialized CI jobs may provide faster feedback or platform evidence, but they must call the same checked-in recipe or script already exercised by just test; they cannot be the only owner of a portable requirement.

Treat just test as a strict superset, not merely a collection of similar assertions. Anything a CI workflow builds locally-portably must be built and tested by just test through the same production primitive. CI is allowed to run only the slice it needs; the local canonical gate is not allowed to omit that slice. A workflow-only build, even when another test validates its input schema, is an Ironbank violation because the produced artifact is unaccounted for.

Run scripts/check-hardcoded-release-selections.sh at the start of just test. This source guard is a release contract, not a style check: user-facing and profile-scoped requests must obtain profile ids from arguments or the installed catalog, package rails must materialize the catalog, native installers must use packaged manifest metadata without a stable/nightly fallback, and exact-SHA qualification must also match the requested channel. Extend the guarded profile terms during renames; keep code, co-work, cowork, terminal, termional, and gui until every migration path is complete. Because it runs before the expensive test stages, the guard must use only declared bootstrap dependencies. Its scanner is Python-standard-library only and has an executable regression with rg absent; never reintroduce a developer-machine-only search command into this fail-fast boundary.

This includes workspace/runtime tests, Rust and Python coverage floors, capsem-doctor and Ironbank acceptance, benchmarks, artifact completeness, frontend/docs/marketing/release-site validation, and the Docker/systemd Linux package install and guest-shell proof. It also includes the full profile-owned VM asset matrix: just test calls just test-assets, which rebuilds every checked-in profile for arm64 and x86_64 through just build-kernel and just build-rootfs, validates the release payload and manifest, and boots each host-architecture result to a real guest-shell marker. Truly non-portable boundaries remain explicit final gates: Apple signing/notarization, hosted KVM, Cloudflare publication, and the physical-Mac public install plus VZ guest shell.

Every portable release-critical CI path must be executable locally through the same production entrypoint. Do not create a local lookalike that merely checks similar commands. If CI calls a just recipe or checked-in script, local proof must call that same recipe or script; if a requirement is implemented as a shared shell function, both paths must execute that function.

Generated release graphs and their materialized profile artifacts must come from the same source snapshot. The local gate qualifies uncommitted candidate bytes from the worktree; it must never generate descriptors from the worktree and then fetch artifact bytes from HEAD. Production release workflows keep using an immutable git ref. Guard both modes with functional tests.

Artifact accounting is literal: macOS-local just test builds the real release-mode .pkg, builds both Linux release-mode .deb architectures, and runs the production host-SBOM generator over those exact packages. Generated settings are regenerated under a before/after idempotence gate. A fixture-only package test or a generator source inspection does not account for the artifact that a release workflow will publish.

Run Linux-only build, doctor, package, and service prerequisites in Docker from just test whenever the host is macOS. Match the CI architecture, command names, environment variables, permissions, and service manager as closely as the container permits. Add a contract test that ties the workflow command to the local entrypoint, plus an executable container regression for the failed requirement. A CI-only failure is evidence of a missing parity gate: add the local reproducer before rerunning CI.

Linux test containers must run the test process as a non-root user unless the CI process is also root. Root invalidates permission-denied regressions and is therefore a parity failure, not a harmless container implementation detail.

The canonical gate also has a runtime budget. Measure full local and CI stage durations and keep meaningful headroom below the runner's observed lifetime; the workflow's declared timeout is not proof that the host will live that long. Disk headroom is part of the same budget. Large immutable packages and VM blobs that are already present in the candidate workspace must use hardlink-first same-filesystem staging on both macOS and Linux, with a tested cross-filesystem copy fallback. Add a constrained-disk executable regression that makes an accidental full copy fail with ENOSPC; a source assertion or an unconstrained happy-path run cannot guard a multi-gigabyte late-stage copy. Record disk use before every expensive artifact lane and before the final install/glow-up tail so capacity failures happen before hours of qualification. Parallel Docker gates also own a daemon-space preflight: measure free space, reclaim only unused builder cache when below the documented reserve, and fail before launching lanes if the reserve remains unavailable. Preserve each lane's complete log and wait for the logging pipeline itself, so failure diagnostics cannot race a still-flushing process substitution. Two runs terminating at the same wall-clock age are a deterministic budget failure until disproved, not random infrastructure. Before another CI attempt, reproduce the expensive rail locally and remove the critical-path bottleneck. Parallelize only independent work with isolated workspaces, Docker tags, output roots, and cleanup ownership, and add a regression that asserts that isolation. Docker tags are daemon-global across worktrees. Automatic gates must never run docker image prune --all/-a, and an internal primitive used by concurrent lanes must not invoke garbage collection. A newly tagged cached image can have an old creation timestamp and be deleted by age-filtered prune -a from another lane or checkout. Run cleanup only at an owning outer boundary, prune dangling images and unused builder cache, and emit captured Docker stderr on failure. Any preflight that mutates the container VM or daemon must have a hard wall-clock timeout and an executable test that waits for process completion. Printed success output is not completion proof. In particular, never set the Colima VM clock from a privileged Docker container: date can exit while the Docker client remains blocked in cleanup. Use the bounded host-side Colima clock primitive.

When an unavoidable platform boundary prevents local execution, name it in the release skill and retain the nearest deterministic local proof. Hardware and external-service gates still require exact-SHA CI evidence; macOS VZ behavior still requires the final physical Mac install and guest-shell proof. Examples include Apple signing/notarization, hosted-runner KVM, Cloudflare publication, and physical-Mac VZ. Never silently omit a gate because the current host cannot run it.

TDD workflow

Write tests first:

1. Write failing tests that capture expected behavior
2. Verify they fail for the right reason
3. Write minimal implementation to pass them
4. Refactor

Without a failing test first, it's easy to write tests that pass by accident or don't actually verify the behavior you intended.

Functional slice proof matrix

Every non-trivial feature slice needs evidence in all of these categories before it can be called done. A green unit suite or a benchmark is not a substitute for functional or end-to-end proof.

If a category is genuinely impossible or deliberately deferred, record it as missing with a reason, owner, and follow-up task. Silent deferral is the bug. "Covered by later E2E" is not enough unless the tracker names the later test and the current milestone is explicitly scoped as internal-only.

For policy, MITM, MCP, telemetry, networking, filesystem, process lifecycle, or sandbox-boundary work, the functional slice matrix is mandatory. The tests should prove not only that the happy path succeeds, but also that enforcement happens at the intended boundary: a blocked MCP tool does not dispatch, a blocked return does not leak, a denied URL does not reach the network, a malformed frame does not poison the stream, and telemetry records the truth.

Ironbank ledger tests

Use /ironbank for release-critical VM, network, model, MCP, credential broker, package-manager, doctor, benchmark, and security acceptance proof. Ironbank lives in tests/ironbank/ and is full black-box: tests are written from public contracts, CLI help, docs, generated schemas, hermetic fixtures, route responses, logs, DB rows, and installed package metadata. Do not inspect Rust/product internals to decide expected behavior.

Ironbank cannot use:

• skip, skipif, slow, optional markers, or public-network dependencies
• status-code-only replay
• row-exists checks
• parser-only assertions
• manual OAuth/client runs as release proof

One deterministic stimulus must prove the whole ledger path: client result, parsed facts, CEL/security decision, detection/enforcement rows, protocol DB rows, structured logs, status counters, UDS route, HTTP route, and UI-facing JSON. Every emitted DB/log/route field is exact-value asserted, covered by a typed invariant, or explicitly marked not applicable. Unknown fields fail the test until the field ledger is updated.

Package-manager tests prove function. Installing zstd, for example, means compressing known bytes, decompressing them, and comparing the exact output; not just checking dpkg output.

Logged-data DB ownership

Telemetry and security ledgers are database-owned. Service routes, UI handlers, MCP helpers, and benchmark harnesses must not build their own logged-data projection caches and must not open SQLite directly. They may own query intent (for example the fields a route needs), but they call the logger DB object to execute it. The logger DB object owns connection threads, mem/disk table layout, write buffering, flush, reload-from-disk behavior, WAL tuning, and future FTS5/search tables.

Do not move route-specific SQL into DbWriter or turn the DB layer into a pile of route helper methods such as stats_detail_payload() just to hide SQL. The boundary is execution and storage mechanics:

db.ready().await?;
db.query(sql, params).await?;
db.write(event).await?;

db.write(event).await means the DB object accepted the event into its producer buffer. Tests that assert read-after-write rows must use the DB flush barrier or shutdown/reopen. Do not paper over visibility with sleeps, route projections, or direct SQLite readers.

Empty table means empty result. Missing table or column means the schema contract is broken and must fail loudly; never add compatibility branches that treat missing ledger shape as empty data.

Regression tests must guard the boundary. If a route needs ledger data, add a test that proves the route uses the DB object and a source guard that rejects raw rusqlite opens, direct DbReader::open, and service-owned projection state in production route code. Add a companion guard that prevents route-specific DB writer methods or missing-schema fallbacks from being introduced.

Mock server boundary

crates/capsem-mock-server is the single reusable local fixture server for benchmarks, doctor, protocol recording/replay, gateway/integration tests, and Ironbank. It owns mock protocol responses and deterministic local upstream behavior. Tests may launch it through scripts/mock_server.py, tests/helpers/mock_server.py, or CAPSEM_MOCK_SERVER_BASE_URL.

Do not add another local HTTP/MCP/OAuth/model mock server for a feature. Extend the shared mock server and its fixtures instead, then assert the route through the relevant black-box test.

Parallel tests as dogfooding (n=4 is non-negotiable)

just test runs the python suite under pytest -n 4 --dist=loadfile. Four real VMs boot simultaneously. This is the canary, not just a speed-up. We ship Capsem as a multi-VM sandbox for AI agents -- if our own test suite cannot safely boot 4 concurrent VMs, real users running an agent farm will hit the exact same bug. Treat any concurrency flake as a Capsem-side bug, not a test-tuning problem:

• "Suspend timed out" under load -> service IPC handling is racy, not "bump the timeout"
• "Session did not become ready" -> Apple VZ resource serialization, VirtioFS lock contention, or service handling concurrent provisions; investigate, don't suppress
• Two tests both want the same VM name -> name-collision bug in validate_vm_name / registry, not "isolate test names better"
• Stale socket between tests -> service didn't reap a child cleanly, real production bug

Anti-patterns when a test flakes under -n 4:

• Adding time.sleep() to "let things settle" -- masking a race
• Bumping the per-test timeout -- buying time for a real bug to manifest in prod instead of CI
• Marking the test serial so it runs alone -- defeating the dogfooding signal

The exception is a true timing or benchmark probe whose assertion is the measured number. Those tests must already be marked serial and just test runs them immediately after the -n 4 canary. That is not a flake escape hatch: it prevents another benchmark file from stealing the same Apple VZ launch budget and corrupting the number we are trying to publish.

The host has plenty of headroom (48 GB RAM, 14 cores; 4 VMs at 2 GB / 2 CPU each = 8 GB / 8 cores). If concurrency surfaces a flake, fix the product, then re-run. Bumping -n higher (8, 12) is the natural follow-on once n=4 is stable -- real users will run more.

Orphan processes across runs are a product bug (not a test bug)

If a previous just test -n 4 run was interrupted (ctrl-C, pytest-xdist worker death, host crash) and the NEXT run flakes with "vm-ready never asserted", UDS "connection refused", or mysterious HTTP 500s -- the cause is companion processes from the interrupted run still alive under PID 1. pkill -f "target/debug/capsem-(service|process|gateway|tray|mcp)" will make the flake vanish, but that is cleanup-after-the-fact. The fix is on the COMPANION side: every spawned companion (gateway, tray, and any new one) must use capsem-guard::install(parent_pid, lock_path) to enforce (a) refuse-standalone, (b) singleton, (c) self-exit on parent death. See /dev-rust-patterns lesson 18. Regression tests live in tests/capsem-service/test_companion_lifecycle.py -- never remove them; when adding a new companion, extend that file.

Never `pkill -f capsem-` with a broad pattern during test debugging: capsem- matches --crate-name capsem-core in running rustc/cargo invocations and will SIGKILL the compiler mid-build. Use a binary-path pattern like pkill -f "target/debug/capsem-(service|process|gateway|tray|mcp)" instead.

Apple VZ lifecycle serialization is part of the product

Apple's Virtualization.framework does not tolerate overlapping checkpoint lifecycle operations (saveMachineStateToURL and restoreMachineStateFromURL) on sibling VMs, and teardown must not cross those checkpoint edges. Capsem uses ServiceState::save_restore_lock plus the host-wide VzHostLock flock: cold starts and teardown take shared/read guards, save and restore take exclusive/write guards. The rail holds even when pytest-xdist spawns one capsem-service per worker, while independent cold starts can still run together for the boot-latency gate.

Do not demote suspend/resume, lifecycle, provisioning, or teardown tests to -n 1 to sidestep VZ races. just test at -n 4 is the contract; if a concurrent run sees restore permission errors, loop-device corruption, connection-refused startup races, or readiness misses, fix the lifecycle rail. Full context and failure signatures live in docs/src/content/docs/gotchas/concurrent-suspend-resume.md.

Adversarial testing

Capsem is a security product. Every security-relevant feature needs tests that actively try to break invariants. Think like an attacker:

• Can a corp-blocked domain be snuck through another provider's list?
• Does an overlapping wildcard in allow+block always deny?
• Does malformed input (empty strings, unicode, huge payloads, invalid JSON) get rejected?
• Can path traversal escape the VirtioFS sandbox?
• Can a guest process modify its own binaries?

Stress-test boundary conditions. Write tests for the attacks you'd attempt yourself.

Security invariants to verify in tests

When touching security-relevant code, check these invariants have test coverage:

Test fixture anti-pattern: masking races with polling

If all test fixtures wait/poll before asserting, the tests will never catch server-side race conditions. For every endpoint that talks to a VM socket, write at least one test that calls it IMMEDIATELY after provision (no wait_exec_ready, no ready_vm fixture). The server must handle readiness internally.

Pattern to avoid (masks the bug -- server never needs wait logic because client always waits):

fixture calls provision -> fixture polls wait_exec_ready -> test calls exec

Required test pattern (catches the bug -- if server doesn't wait, test fails):

test calls provision -> test immediately calls exec -> server handles wait

See tests/capsem-service/test_svc_exec_ready.py for the regression tests that enforce this.

wait_exec_ready is a single call, not a loop

wait_exec_ready (in tests/helpers/service.py, tests/helpers/mcp.py, tests/capsem-gateway/test_gw_e2e.py) makes one exec call with the server-side timeout passed through. The server's handle_exec calls wait_for_vm_ready internally, which polls until the VM is ready. Do NOT add client-side retry loops -- that creates a double-wait where each retry can block for the full server timeout (30s client retries x 30s server wait = pathological cascade). One wait, one place.

Exec latency regression gate

tests/capsem-serial/test_boot_timing.py::test_exec_latency_under_1_5_seconds asserts that provision-to-first-exec completes in under 1.5s. If this test fails, investigate boot time (process.log boot_timeline spans), not the wait mechanism.

Where tests live

• Rust unit: sibling `tests.rs` file, not inline `mod tests { ... }`. See the next subsection.
• Rust integration: crates/capsem-core/tests/
• In-VM diagnostics: guest/artifacts/diagnostics/test_*.py (see dev-testing-vm)
• Hypervisor: KVM + Apple VZ tests (see dev-testing-hypervisor)
• Frontend: frontend/src/lib/__tests__/ (see dev-testing-frontend)
• Python (builder): tests/test_*.py
• Python integration (service daemon): tests/capsem-*/ directories, each with its own conftest.py and pytest marker
• Ironbank release ledger: tests/ironbank/ (black-box only; no Rust

implementation-derived expectations)

Rust unit tests: sibling tests.rs pattern

Every Rust module keeps its unit tests in a sibling `tests.rs`, not an inline `mod tests { ... }` block. The parent module declares:

// foo.rs  OR  foo/mod.rs
// ... production code ...
 
#[cfg(test)]
mod tests;

and the tests go in tests.rs in the same directory:

// tests.rs -- sibling of foo.rs or child of foo/
use super::*;
 
#[test]
fn roundtrip() { ... }

Why. Inline #[cfg(test)] mod tests { ... } blocks are appended at the bottom of prod files and commonly hit 50–99% of the file's line count. That means every Read, grep, and scroll to reach production code walks past thousands of test lines first. Several modules in this codebase hit 4,000+ lines that way before extraction. Agents and humans both read faster when prod code isn't buried.

Mechanics.

• tests.rs is a submodule of the parent file -- use super::*; works, private items are visible, #[cfg(test)] on the mod tests; declaration still gates compilation.
• For files that don't yet have a sibling directory (e.g. lib.rs, foo.rs), put tests.rs next to them in the same src/ directory.
• For files that are already foo/mod.rs, put tests.rs inside foo/.
• Attributes on the inline mod tests block (e.g. #[allow(unused_imports)]) move onto the declaration: #[cfg(test)]\n#[allow(unused_imports)]\nmod tests;.

Extraction recipe (for any remaining inline mod tests { ... }):

1. Move the block body (everything between the outer { and }) into a new sibling tests.rs.
2. Dedent one indentation level so contents read as top-level items.
3. Replace the old inline block with #[cfg(test)] mod tests; (plus any attributes that were on the original).
4. cargo test -p <crate> -- should pass identically.

When to push back. If you see a new PR or agent output adding an inline mod tests { ... } block, request it be moved to tests.rs before merge. Exceptions are narrow: tiny helper modules under ~50 lines total where inline tests plus prod code fit on one screen, or a module that's already a test-only helper.

Integration test suites

All Python integration tests live under tests/capsem-*/ and use pytest markers. Each suite has a dedicated just recipe.

Composite recipe: just test-vm runs build-chain + guest + cleanup + codesign + serial + session-lifecycle + config-runtime + recovery. just test-install runs the install suite in Docker with systemd. just test runs everything.

Test matrix: what runs where

Rust crate CI matrix

Python integration suite tier map

"Run" = tests execute in PR CI. "Collect" = imports verified (--collect-only) but tests do not execute in that PR lane. Artifact-dependent no-VM suites still execute in the full just test gate after their build/sign prerequisites exist. "Yes (Docker)" = runs in dedicated Docker+systemd CI job.

Coverage targets

Coverage

• Rust: cargo llvm-cov via just test (floor: 65% line coverage)
• Python: PR top-level contract lane uses 89%; full just test uses 90%.
• codecov.yml maps components to code paths. Update it when files or directories are added, moved, or renamed.

Fast debug with capsem MCP tools

When the capsem MCP server is configured, Claude Code has direct VM control via MCP tools -- no shell commands or just recipes needed. This is the fastest way to test changes interactively because you stay in the conversation loop: create a VM, run commands, inspect results, fix code, repeat.

The tools

Debug workflow

Quick one-shot (no VM management): capsem_run with the command you want to test.

Iterative debugging (long-lived VM):

1. Create: capsem_create -- boots a fresh VM in ~10s
2. Test: capsem_exec with the command you want to verify (e.g., capsem-doctor -k net, cat /etc/resolv.conf, curl https://example.com)
3. Inspect: capsem_read_file to check config files/logs; typed stats, timeline, security, detection, and enforcement routes for telemetry
4. Iterate: fix code on host, rebuild (just build), create a new VM to test again
5. Cleanup: capsem_delete when done

When to use MCP tools vs just recipes

MCP tools are for fast, targeted checks during development. Just recipes are for comprehensive validation before committing.

Common debug queries

-- Check network events for a domain
SELECT * FROM net_events WHERE domain LIKE '%example%' ORDER BY timestamp DESC LIMIT 10;
 
-- Verify MCP-origin tool calls were logged
SELECT server_name, tool_name, decision, duration_ms
FROM tool_calls
WHERE origin = 'mcp'
ORDER BY timestamp DESC;
 
-- Check model API calls
SELECT provider, model, status_code, duration_ms FROM model_calls ORDER BY timestamp DESC;
 
-- File system events
SELECT operation, path, success FROM fs_events ORDER BY timestamp DESC LIMIT 20;

End-to-end validation is not optional

After any change touching guest binaries, network policy, telemetry, MCP, or VM lifecycle:

1. just exec "capsem-doctor" -- verifies sandbox integrity inside the VM
2. After telemetry/logging changes: run a real session and verify with just inspect-session that net_events, model_calls, tool_calls, tool_responses, fs_events, dns_events, and security_rule_events are populated correctly for the exercised protocols

When tests fail

Never dismiss a test failure as "pre-existing" or "unrelated." Every failure must be investigated. Follow the dev-debugging workflow:

1. Do not change the test to make it pass. The test is evidence. Changing the assertion to match broken behavior destroys that evidence.
2. Reproduce and diagnose first. Understand why it fails before writing any fix. See the dev-debugging skill for the full methodology: reproduce with a test, diagnose root cause, then fix comprehensively.
3. Fix the code, not the test. If the test is genuinely wrong (not the code), explain in detail why the test's expectation is incorrect before changing it.

Platform gating tests

cargo test --test platform_gating scans all .rs files under crates/ for macOS-only and Linux-only symbols (libc::clonefile, AppleVzHypervisor, KvmHypervisor, FICLONE, etc.) and verifies they appear inside #[cfg(target_os = "...")] blocks. This catches ungated platform APIs before they reach CI. Run this test when adding any platform-specific code.

Testable design

Extract logic into capsem-core -- never embed business logic in the app layer where it's coupled to Tauri. If you can't test something without booting a VM or launching the GUI, it belongs in core.