version: "1.0.1" name: verify-environment description: Run and inspect the script-owned local two-sibling validation before spending GPU credits.

Verify the environment

Inspect main.py, then run:

uv run python main.py validate

The validation stage must first obtain its example through env.create_dataset("train", Path(".")), then call castform.validate_environment once for one real Environment.run_group containing exactly two siblings of that example locally and in the hosted sandbox. This exercises the public data materialization and deployment contract as part of validation. Hosted validation always runs against the exact assets that were just uploaded — the same ones a launch would train on. Keep the local and hosted rollout-model context budget shared through VALIDATE_CONFIG["max_context_tokens"]; the local wall-clock backstop is VALIDATE_CONFIG["local_timeout_seconds"].

validate_environment first performs static model-parameter checks, then uses tracked model sessions locally and remotely to enforce the same sampling and multi-turn history contract as training. Review static and runtime warnings as well as outcomes. A max_tokens or max_completion_tokens warning is allowed when the effective cap is acceptable. Sampling conflicts, unsupported controls, changed tools, overlapping generations, and rewritten assistant history are errors. Do not launch while any contract error remains. If validation made only one model call, treat the “multi-turn history was not exercised” warning as a coverage gap for harnesses expected to loop.

Read both outcomes

For each sibling, inspect:

• termination_reason;
• the complete reward mapping and total;
• evidence that the response was actually scored by the intended reward;
• any environment, tool, sandbox or judge error logs.

A successful outcome has termination_reason == "finished" and the reward components produced by its scoring hooks. Its scores may legitimately all be zero. An operational failure has a different termination reason, no rewards, and a visible log entry. It must not cancel the other sibling.

Do not call the baseline green when:

• an outcome failed, even if its reward mapping looks structurally valid;
• rewards are malformed, non-finite or missing declared keys;
• the reward is constant for reasons the task does not justify;
• the judge or verifier failure was mistaken for a valid zero score;
• group-relative scoring depends on failed siblings or cross-group state.

Targeted checks

Before launch, add unit tests for empty, wrong, partial and correct answers. Put them in tests/ next to main.py (its conftest.py pins the import path so from main import ... resolves) and run uv run pytest tests. Exercise tool exceptions and judge exceptions and assert the failure termination reason, zeroed declared shape and log message. For a group-relative reward, verify that one failed sibling does not alter successful siblings' scoring inputs.

If the environment uses InjectedAuth("judge") for the Castform LLM endpoint, Castform validation binds that name to its call-time credential provider for the duration of the run. Rollout model_auth and named environment bindings are independent; overriding one must not silently override the other. A missing or unknown binding should fail visibly; the environment must not read a platform token itself.

When the baseline is green, report both outcomes and ask whether to iterate or load launch-run. Do not launch automatically.