version: "1.0.0" name: dev-testing description: Capsem testing policy and workflow. Use whenever running tests, writing new tests, or verifying changes work. Covers the three test tiers (unit, smoke, full), TDD red-green-refactor, adversarial security testing, coverage policy, and the mandatory end-to-end VM validation. For VM-specific tests see dev-testing-vm, for hypervisor tests see dev-testing-hypervisor, for frontend tests see dev-testing-frontend.

Testing

Test tiers

Three tiers, fast to thorough. Every change must pass all three before it ships.

just test is the single source of truth. There is no "fast" tier that skips integration tests -- that's how the "Connection refused" bug shipped while tests said green. Individual test-* recipes exist for targeted debugging but just test is the gate.

TDD workflow

Write tests first:

1. Write failing tests that capture expected behavior
2. Verify they fail for the right reason
3. Write minimal implementation to pass them
4. Refactor

Without a failing test first, it's easy to write tests that pass by accident or don't actually verify the behavior you intended.

Functional slice proof matrix

Every non-trivial feature slice needs evidence in all of these categories before it can be called done. A green unit suite or a benchmark is not a substitute for functional or end-to-end proof.

If a category is genuinely impossible or deliberately deferred, record it as missing with a reason, owner, and follow-up task. Silent deferral is the bug. "Covered by later E2E" is not enough unless the tracker names the later test and the current milestone is explicitly scoped as internal-only.

For policy, MITM, MCP, telemetry, networking, filesystem, process lifecycle, or sandbox-boundary work, the functional slice matrix is mandatory. The tests should prove not only that the happy path succeeds, but also that enforcement happens at the intended boundary: a blocked MCP tool does not dispatch, a blocked return does not leak, a denied URL does not reach the network, a malformed frame does not poison the stream, and telemetry records the truth.

Parallel tests as dogfooding (n=4 is non-negotiable)

just test runs the python suite under pytest -n 4 --dist=loadfile. Four real VMs boot simultaneously. This is the canary, not just a speed-up. We ship Capsem as a multi-VM sandbox for AI agents -- if our own test suite cannot safely boot 4 concurrent VMs, real users running an agent farm will hit the exact same bug. Treat any concurrency flake as a Capsem-side bug, not a test-tuning problem:

• "Suspend timed out" under load -> service IPC handling is racy, not "bump the timeout"
• "Session did not become ready" -> Apple VZ resource serialization, VirtioFS lock contention, or service handling concurrent provisions; investigate, don't suppress
• Two tests both want the same VM name -> name-collision bug in validate_vm_name / registry, not "isolate test names better"
• Stale socket between tests -> service didn't reap a child cleanly, real production bug

Anti-patterns when a test flakes under -n 4:

• Adding time.sleep() to "let things settle" -- masking a race
• Bumping the per-test timeout -- buying time for a real bug to manifest in prod instead of CI
• Marking the test serial so it runs alone -- defeating the dogfooding signal

The host has plenty of headroom (48 GB RAM, 14 cores; 4 VMs at 2 GB / 2 CPU each = 8 GB / 8 cores). If concurrency surfaces a flake, fix the product, then re-run. Bumping -n higher (8, 12) is the natural follow-on once n=4 is stable -- real users will run more.

Orphan processes across runs are a product bug (not a test bug)

If a previous just test -n 4 run was interrupted (ctrl-C, pytest-xdist worker death, host crash) and the NEXT run flakes with "vm-ready never asserted", UDS "connection refused", or mysterious HTTP 500s -- the cause is companion processes from the interrupted run still alive under PID 1. pkill -f "target/debug/capsem-(service|process|gateway|tray|mcp)" will make the flake vanish, but that is cleanup-after-the-fact. The fix is on the COMPANION side: every spawned companion (gateway, tray, and any new one) must use capsem-guard::install(parent_pid, lock_path) to enforce (a) refuse-standalone, (b) singleton, (c) self-exit on parent death. See /dev-rust-patterns lesson 18. Regression tests live in tests/capsem-service/test_companion_lifecycle.py -- never remove them; when adding a new companion, extend that file.

Never `pkill -f capsem-` with a broad pattern during test debugging: capsem- matches --crate-name capsem-core in running rustc/cargo invocations and will SIGKILL the compiler mid-build. Use a binary-path pattern like pkill -f "target/debug/capsem-(service|process|gateway|tray|mcp)" instead.

When -n 1 is actually the right answer: multi-service-only gotchas

One narrow class of concurrency bug belongs at -n 1, not -n 4: bugs that only exist when two `capsem-service` processes run on the same host. Apple's Virtualization.framework does not tolerate overlapping saveMachineStateToURL / restoreMachineStateFromURL calls on sibling VMs, and we serialize with a per-service tokio::sync::Mutex (ServiceState::save_restore_lock). That lock is in-process, so it only serializes VMs inside one service. Production always has exactly one service per host per user, so the lock is sufficient in real deployments.

tests/capsem-mcp/test_stress_suspend_resume.py runs under pytest-xdist, which spawns one capsem-service per worker. At -n 2+, worker A's service can't see worker B's lock, and you re-expose the bug that never happens in production. This is the one case where the "n=4 dogfoods concurrency" rule doesn't apply -- the concurrency being tested would never happen outside the test harness. Keep this harness at -n 1. Full context and the failure signature live in docs/src/content/docs/gotchas/concurrent-suspend-resume.md.

This is NOT a blanket license to run any flaky test at -n 1. If you're tempted to demote another test, first ask: "Would this failure occur in production with one capsem-service and N VMs?" If yes, it belongs at -n 4; fix the product.

Adversarial testing

Capsem is a security product. Every security-relevant feature needs tests that actively try to break invariants. Think like an attacker:

• Can a corp-blocked domain be snuck through another provider's list?
• Does an overlapping wildcard in allow+block always deny?
• Does malformed input (empty strings, unicode, huge payloads, invalid JSON) get rejected?
• Can path traversal escape the VirtioFS sandbox?
• Can a guest process modify its own binaries?

Stress-test boundary conditions. Write tests for the attacks you'd attempt yourself.

Security invariants to verify in tests

When touching security-relevant code, check these invariants have test coverage:

Test fixture anti-pattern: masking races with polling

If all test fixtures wait/poll before asserting, the tests will never catch server-side race conditions. For every endpoint that talks to a VM socket, write at least one test that calls it IMMEDIATELY after provision (no wait_exec_ready, no ready_vm fixture). The server must handle readiness internally.

Pattern to avoid (masks the bug -- server never needs wait logic because client always waits):

fixture calls provision -> fixture polls wait_exec_ready -> test calls exec

Required test pattern (catches the bug -- if server doesn't wait, test fails):

test calls provision -> test immediately calls exec -> server handles wait

See tests/capsem-service/test_svc_exec_ready.py for the regression tests that enforce this.

wait_exec_ready is a single call, not a loop

wait_exec_ready (in tests/helpers/service.py, tests/helpers/mcp.py, tests/capsem-gateway/test_gw_e2e.py) makes one exec call with the server-side timeout passed through. The server's handle_exec calls wait_for_vm_ready internally, which polls until the VM is ready. Do NOT add client-side retry loops -- that creates a double-wait where each retry can block for the full server timeout (30s client retries x 30s server wait = pathological cascade). One wait, one place.

Exec latency regression gate

tests/capsem-serial/test_boot_timing.py::test_exec_latency_under_1_5_seconds asserts that provision-to-first-exec completes in under 1.5s. If this test fails, investigate boot time (process.log boot_timeline spans), not the wait mechanism.

Where tests live

• Rust unit: sibling `tests.rs` file, not inline `mod tests { ... }`. See the next subsection.
• Rust integration: crates/capsem-core/tests/
• In-VM diagnostics: guest/artifacts/diagnostics/test_*.py (see dev-testing-vm)
• Hypervisor: KVM + Apple VZ tests (see dev-testing-hypervisor)
• Frontend: frontend/src/lib/__tests__/ (see dev-testing-frontend)
• Python (builder): tests/test_*.py
• Python integration (service daemon): tests/capsem-*/ directories, each with its own conftest.py and pytest marker

Rust unit tests: sibling tests.rs pattern

Every Rust module keeps its unit tests in a sibling `tests.rs`, not an inline `mod tests { ... }` block. The parent module declares:

// foo.rs  OR  foo/mod.rs
// ... production code ...
 
#[cfg(test)]
mod tests;

and the tests go in tests.rs in the same directory:

// tests.rs -- sibling of foo.rs or child of foo/
use super::*;
 
#[test]
fn roundtrip() { ... }

Why. Inline #[cfg(test)] mod tests { ... } blocks are appended at the bottom of prod files and commonly hit 50–99% of the file's line count. That means every Read, grep, and scroll to reach production code walks past thousands of test lines first. Several modules in this codebase hit 4,000+ lines that way before extraction. Agents and humans both read faster when prod code isn't buried.

Mechanics.

• tests.rs is a submodule of the parent file -- use super::*; works, private items are visible, #[cfg(test)] on the mod tests; declaration still gates compilation.
• For files that don't yet have a sibling directory (e.g. lib.rs, foo.rs), put tests.rs next to them in the same src/ directory.
• For files that are already foo/mod.rs, put tests.rs inside foo/.
• Attributes on the inline mod tests block (e.g. #[allow(unused_imports)]) move onto the declaration: #[cfg(test)]\n#[allow(unused_imports)]\nmod tests;.

Extraction recipe (for any remaining inline mod tests { ... }):

1. Move the block body (everything between the outer { and }) into a new sibling tests.rs.
2. Dedent one indentation level so contents read as top-level items.
3. Replace the old inline block with #[cfg(test)] mod tests; (plus any attributes that were on the original).
4. cargo test -p <crate> -- should pass identically.

When to push back. If you see a new PR or agent output adding an inline mod tests { ... } block, request it be moved to tests.rs before merge. Exceptions are narrow: tiny helper modules under ~50 lines total where inline tests plus prod code fit on one screen, or a module that's already a test-only helper.

Integration test suites

All Python integration tests live under tests/capsem-*/ and use pytest markers. Each suite has a dedicated just recipe.

Composite recipe: just test-vm runs build-chain + guest + cleanup + codesign + serial + session-lifecycle + config-runtime + recovery. just test-install runs the install suite in Docker with systemd. just test runs everything.

Test matrix: what runs where

Rust crate CI matrix

Python integration suite tier map

"Run" = tests execute in PR CI. "Collect" = imports verified (--collect-only) but tests do not execute in that PR lane. Artifact-dependent no-VM suites still execute in the full just test gate after their build/sign prerequisites exist. "Yes (Docker)" = runs in dedicated Docker+systemd CI job.

Coverage targets

Coverage

• Rust: cargo llvm-cov via just test (floor: 65% line coverage)
• Python: PR top-level contract lane uses 89%; full just test uses 90%.
• codecov.yml maps components to code paths. Update it when files or directories are added, moved, or renamed.

Fast debug with capsem MCP tools

When the capsem MCP server is configured, Claude Code has direct VM control via MCP tools -- no shell commands or just recipes needed. This is the fastest way to test changes interactively because you stay in the conversation loop: create a VM, run commands, inspect results, fix code, repeat.

The tools

Debug workflow

Quick one-shot (no VM management): capsem_run with the command you want to test.

Iterative debugging (long-lived VM):

1. Create: capsem_create -- boots a fresh VM in ~10s
2. Test: capsem_exec with the command you want to verify (e.g., capsem-doctor -k net, cat /etc/resolv.conf, curl https://example.com)
3. Inspect: capsem_read_file to check config files, logs; capsem_inspect to query telemetry tables
4. Iterate: fix code on host, rebuild (just build), create a new VM to test again
5. Cleanup: capsem_delete when done

When to use MCP tools vs just recipes

MCP tools are for fast, targeted checks during development. Just recipes are for comprehensive validation before committing.

Common debug queries

-- Check network events for a domain
SELECT * FROM net_events WHERE domain LIKE '%example%' ORDER BY timestamp DESC LIMIT 10;
 
-- Verify MCP tool calls were logged
SELECT server_name, tool_name, decision, duration_ms FROM mcp_calls ORDER BY timestamp DESC;
 
-- Check model API calls
SELECT provider, model, status_code, duration_ms FROM model_calls ORDER BY timestamp DESC;
 
-- File system events
SELECT operation, path, success FROM fs_events ORDER BY timestamp DESC LIMIT 20;

End-to-end validation is not optional

After any change touching guest binaries, network policy, telemetry, MCP, or VM lifecycle:

1. just run "capsem-doctor" -- verifies sandbox integrity inside the VM
2. After telemetry/logging changes: run a real session and verify with just inspect-session that all 6 tables (net_events, model_calls, tool_calls, tool_responses, mcp_calls, fs_events) are populated correctly

When tests fail

Never dismiss a test failure as "pre-existing" or "unrelated." Every failure must be investigated. Follow the dev-debugging workflow:

1. Do not change the test to make it pass. The test is evidence. Changing the assertion to match broken behavior destroys that evidence.
2. Reproduce and diagnose first. Understand why it fails before writing any fix. See the dev-debugging skill for the full methodology: reproduce with a test, diagnose root cause, then fix comprehensively.
3. Fix the code, not the test. If the test is genuinely wrong (not the code), explain in detail why the test's expectation is incorrect before changing it.

Platform gating tests

cargo test --test platform_gating scans all .rs files under crates/ for macOS-only and Linux-only symbols (libc::clonefile, AppleVzHypervisor, KvmHypervisor, FICLONE, etc.) and verifies they appear inside #[cfg(target_os = "...")] blocks. This catches ungated platform APIs before they reach CI. Run this test when adding any platform-specific code.

Testable design

Extract logic into capsem-core -- never embed business logic in the app layer where it's coupled to Tauri. If you can't test something without booting a VM or launching the GUI, it belongs in core.