version: "1.0.1" name: unblock description: "Use when a fetched URL returns 4xx/blocked, hits a WAF or captcha, or when a JS-heavy SPA returns no usable body. Escalates through public APIs, Jina Reader, header-diverse curl, TLS impersonation, headless browsers, and free archive mirrors until validated content is returned. Zero API keys." effort: medium

Unblock — Adaptive Fetch Chain (Zero-Key)

No API keys. No signup. No config. Every phase runs on public endpoints,

rate-limited free tiers, or self-hosted binaries that auto-install. Paid

providers exist only behind explicit --allow-paid.

Iron Law

First HTTP 200 is not success. Validate the body first.

A 200 with a Cloudflare challenge HTML, a login wall, or an empty SPA shell is failure. The chain stops only when content passes engine/validate.mjs.

Phases

The chain stops at the first probe whose body passes validate.mjs. Phase 6 never runs implicitly even with env keys present. Phase 0d and URL normalization detail: references/native-cleaners.md.

When to Use

• Eevee researcher hits 403/blocked during /scc:research
• Manual fetch of a hostile site
• WebFetch returns truncated/empty body for JS-heavy SPAs

Not for plain keyword search (use Jina Search via s.jina.ai) or bulk crawls.

Quick Start

node skills/unblock/engine/cli.mjs "<URL>"
node skills/unblock/engine/cli.mjs "<URL>" --device mobile --json
node skills/unblock/engine/cli.mjs "<URL>" --selector "article" --trace
node skills/unblock/engine/cli.mjs "<URL>" --max-phase 4

Exit codes: 0 success, 1 exhausted, 2 invalid input.

Phase 0a — Public APIs (instant win)

Dispatched by URL host pattern.

Phase 5 — Free Archive Fallback

When Phases 1–4 all fail, the chain tries free mirrors: Wayback Machine, archive.today, AMP cache, RSS/Atom feed discovery, OG-tag rescue. Results carry via_archive: true; OG-rescue results also carry partial: true. See references/archive-fallbacks.md.

Phase 2 — Cookie Warming + Referrer Chain

curl-impersonate does not just spoof TLS. The probe runs a 2-hop sequence:

1. GET https://<host>/ with empty referrer → captures Set-Cookie
2. GET target URL with referrer = locale-matched search engine, cookies = step 1 jar

Defeats homepage-cookie gates and referrer-checking news sites without per-site rules.

Phase 1 — Header Diversity

Each retry rotates User-Agent and: Accept-Language matched to TLD, Sec-Ch-Ua / Sec-Ch-Ua-Mobile / Sec-Ch-Ua-Platform (Chrome 131 client hints), Sec-Fetch-Dest / Mode / Site, Accept-Encoding: gzip, deflate, br, zstd. WAFs gating on missing client hints fall here without escalation.

Hard Rules

R1 — No site-name hardcoding. engine/profiles.json and code under engine/** must not contain brand-specific selectors or domain matches except the Phase 0a public-API allowlist (which routes to public APIs, not selector scraping). Site-specific scraping hints flow only via --user-hint. Enforced by tests/skills/unblock/no-brand-hardcode.test.mjs.

R2 — Validate before declare. Every probe result passes through validate.mjs's 4-layer check (status, body length, challenge body, content-type) before being returned.

R3 — Auto-install on miss. Missing binary triggers a one-time install attempt. If install fails, skip to next phase. Never block the chain.

R4 — Trace is mandatory on failure. Failure mode always returns a JSON trace listing phase outcomes, error codes, elapsed times.

R5 — Read trace before retry. Identical retries are forbidden. Adjust --device, --selector, or --user-hint based on trace.

R6 — Paid is opt-in. Phase 6 never runs without explicit --allow-paid.

R7 — Single-user CLI, not a service. This skill fetches arbitrary user-supplied URLs without internal-IP filtering. Do not expose its CLI behind a network endpoint without adding host allowlisting upstream — it can otherwise be used to probe 169.254.169.254 and other private ranges (SSRF).

Eevee Researcher Integration

Eevee invokes Unblock automatically when WebFetch returns 4xx/5xx, an under-200-char body, a known challenge signature, or content-type mismatch. See references/eevee-flow.md.

Output

Returns JSON with ok, url, phase, probe, elapsed_ms, content, title, meta, trace. Trace records every probe outcome. On exhaustion, content may still be present with partial: true if Phase 5 OG-rescue caught anything.

Configuration

All optional. Defaults work zero-key.

Files

engine/
  cli.mjs           argv, exit code, output
  chain.mjs         orchestrator, escalation
  detect.mjs        challenge / WAF signatures
  validate.mjs      4-layer success validator
  transforms.mjs    URL variants + locale-aware headers
  intent.mjs        URL vs keyword router
  cookie-jar.mjs    in-memory Set-Cookie jar
  install.mjs       auto-install binaries
  profiles.json     UAs, transforms, validator thresholds
  probes/{public-api,jina,yt-dlp,curl,impersonate,lightpanda,playwright,archive,paid-api}.mjs
references/{waf-detection,tls-impersonation,archive-fallbacks,eevee-flow}.md