microsoft/vscode/chat-perf

Chat Performance Testing

When to use

• Before/after modifying chat rendering code (chatListRenderer.ts, chatInputPart.ts, markdown rendering)
• When changing the streaming response pipeline or SSE processing
• When modifying disposable/lifecycle patterns in chat components
• To compare performance between two VS Code releases
• In CI to gate PRs that touch chat UI code

Quick start

# Run perf regression test (compares local dev build vs VS Code 1.115.0):
npm run perf:chat -- --scenario text-only --runs 3
 
# Run all scenarios with no baseline (just measure):
npm run perf:chat -- --no-baseline --runs 3
 
# Compare two local builds (apples-to-apples):
npm run perf:chat -- --build /path/to/build-A --baseline-build /path/to/build-B --runs 5
 
# Build a local production package and compare against a release:
npm run perf:chat -- --production-build --baseline-build 1.115.0 --runs 5
 
# Run memory leak check (10 messages in one session):
npm run perf:chat-leak
 
# Run leak check with more messages for accuracy:
npm run perf:chat-leak -- --messages 20 --verbose

Perf regression test

Script: scripts/chat-simulation/test-chat-perf-regression.js npm: npm run perf:chat

Launches VS Code via Playwright Electron, opens the chat panel, sends a message with a mock LLM response, and measures timing, layout, and rendering metrics. By default, downloads VS Code 1.115.0 as a baseline, benchmarks it, then benchmarks the local dev build and compares.

You don't always need a baseline. A baseline exists only for comparison (regression detection). If you just want the current build's numbers — profiling a single change, capturing traces/heap snapshots, or iterating on a scenario — pass --no-baseline to skip downloading and benchmarking the baseline entirely (roughly halves runtime). Baseline comparison is what turns raw measurements into a pass/fail verdict; without it you still get all the metrics, just no verdict.

Key flags

Comparing two remote builds

# Compare 1.110.0 against 1.115.0 (no local build needed):
npm run perf:chat -- --build 1.110.0 --baseline-build 1.115.0 --runs 5

Comparing two local builds

Both --build and --baseline-build accept local paths to VS Code executables. This enables apples-to-apples comparisons between any two builds:

# Compare two dev builds (e.g. feature branch vs main):
npm run perf:chat -- \
  --build .build/electron/Code\ -\ OSS.app/Contents/MacOS/Code\ -\ OSS \
  --baseline-build /path/to/other/Code\ -\ OSS.app/Contents/MacOS/Code\ -\ OSS \
  --runs 5
 
# Compare two production builds:
npm run perf:chat -- \
  --build ../VSCode-darwin-arm64-feature/Code\ -\ OSS.app/Contents/MacOS/Code\ -\ OSS \
  --baseline-build ../VSCode-darwin-arm64-main/Code\ -\ OSS.app/Contents/MacOS/Code\ -\ OSS \
  --runs 5

Local path baselines are never cached (the build may change between runs). Version string baselines are cached for reuse.

Build modes and mismatch detection

The tool classifies builds into three modes based on the executable path:

When test and baseline builds have different modes (e.g. dev vs release), the tool shows a warning and prompts for confirmation. Use --force or --ci to skip the prompt.

Using --production-build builds a local bundled package via gulp vscode for fair comparison against a release baseline. This eliminates dev-mode overhead while still testing your local changes.

# Production build vs release baseline (fair comparison):
npm run perf:chat -- --production-build --baseline-build 1.115.0 --runs 5

Settings overrides

Use --setting, --test-setting, and --baseline-setting to inject VS Code settings into the launched instance. This is useful for A/B testing experimental features:

# Enable a feature for the test build only:
npm run perf:chat -- --test-setting chat.experimental.incrementalRendering.enabled=true --runs 3
 
# Compare two builds with different settings:
npm run perf:chat -- \
  --baseline-build "../vscode2/.build/electron/Code - OSS.app/Contents/MacOS/Code - OSS" \
  --baseline-setting chat.experimental.incrementalRendering.enabled=true \
  --test-setting chat.experimental.incrementalRendering.enabled=false \
  --runs 3
 
# Set a value for both builds:
npm run perf:chat -- --setting chat.mcp.enabled=false --runs 3

Precedence: --test-setting / --baseline-setting override --setting for the same key. Values are auto-parsed: true/false become booleans, numbers become numbers, everything else stays a string.

Resuming a run for more confidence

When results exceed the threshold but aren't statistically significant, the tool prints a --resume hint. Use it to add more iterations to an existing run:

# Initial run with 3 iterations — may be inconclusive:
npm run perf:chat -- --scenario text-only --runs 3
 
# Add 3 more runs to the same results file (both test + baseline):
npm run perf:chat -- --resume .chat-simulation-data/2026-04-14T02-15-14/results.json --runs 3
 
# Keep adding until confidence is reached:
npm run perf:chat -- --resume .chat-simulation-data/2026-04-14T02-15-14/results.json --runs 5

--resume loads the previous results.json and its associated baseline-*.json, runs N more iterations for both builds, merges rawRuns, recomputes stats, and re-runs the comparison. The updated files are written back in-place. You can resume multiple times — samples accumulate.

Statistical significance

Regression detection uses Welch's t-test to avoid false positives from noisy measurements. A metric is only flagged as REGRESSION when it both exceeds the threshold AND is statistically significant (p < 0.05). Otherwise it's reported as (likely noise — p=X, not significant).

With typical variance (cv ≈ 20%), you need:

• n ≥ 5 per build to detect a 35% regression at 95% confidence
• n ≥ 10 per build to detect a 20% regression reliably

Confidence levels reported: high (p < 0.01), medium (p < 0.05), low (p < 0.1), none.

Exit codes

• 0 — all metrics within threshold, or exceeding threshold but not statistically significant
• 1 — statistically significant regression detected, or all runs failed

Scenarios

Scenarios are defined in scripts/chat-simulation/common/perf-scenarios.js and registered via registerPerfScenarios(). There are three categories:

• Content-only — plain streaming responses (e.g. text-only, large-codeblock, rapid-stream)
• Tool-call — multi-turn scenarios with tool invocations (e.g. tool-read-file, tool-edit-file)
• Multi-turn user — multi-turn conversations with user follow-ups, thinking blocks (e.g. thinking-response, multi-turn-user, long-conversation)

Run npm run perf:chat -- --help to see the full list of registered scenario IDs.

Metrics collected

• Timing: time to first token, time to complete, time to render complete (includes typewriter animation)
• Rendering: layout count, layout duration (ms), style recalculation count, forced reflows, long tasks (>50ms), long animation frame count and duration
• Memory: heap before/after, heap delta post-GC (informational, noisy for single requests)
• Extension host: heap before/after/delta via CDP inspector

Regression triggers vs informational metrics

Only these metrics trigger a regression failure (when they exceed the threshold with statistical significance):

• timeToFirstToken, timeToComplete — user-perceived latency
• layoutDurationMs — total layout time from the trace (the real layout cost)
• forcedReflowCount — forced synchronous layouts are always bad
• longTaskCount — main thread jank

These are reported but informational only (won't fail CI):

• longAnimationFrameCount — number of long animation frames; noisy and compositor-driven, so tracked for signal but does not gate. Real jank is gated via longTaskCount / layoutDurationMs.
• layoutCount — number of layout ops; inflated by CSS animations (compositor-driven, cheap). A build can do more but cheaper layouts, so gate on layoutDurationMs, not this count.
• recalcStyleCount — number of style recalcs; inflated by CSS animations (compositor-driven, cheap)
• timeToRenderComplete — includes typewriter animation tail
• Memory/heap metrics — too noisy for single-request benchmarks

Statistics

Results use IQR-based outlier removal and median (not mean) to handle startup jitter. The coefficient of variation (cv) is reported — under 15% is stable, over 15% gets a ⚠ warning. Baseline comparison uses Welch's t-test on raw run values to determine statistical significance before flagging regressions. Use 5+ runs to get stable results.

Memory leak check

Script: scripts/chat-simulation/test-chat-mem-leaks.js npm: npm run perf:chat-leak

Launches one VS Code session, sends N messages sequentially, forces GC between each, and measures renderer heap and DOM node count. Uses linear regression on the samples to compute per-message growth rate, which is compared against a threshold.

Key flags