<< All versions
Skill v1.0.1
currentLLM-judged scan95/100existential-birds/beagle/pydantic-ai-testing
3 files
──Details
PublishedMay 15, 2026 at 09:43 AM
Content Hashsha256:f66c90456241aee8...
Git SHA9503d5afa6e1
Bump Typepatch
──Files
Files (1 file, 7.0 KB)
SKILL.md7.0 KBactive
SKILL.md · 275 lines · 7.0 KB
version: "1.0.1" name: pydantic-ai-testing description: Test PydanticAI agents using TestModel, FunctionModel, VCR cassettes, and inline snapshots. Use when writing unit tests, mocking LLM responses, or recording API interactions.
Testing PydanticAI Agents
TestModel (Deterministic Testing)
Use TestModel for tests without API calls:
python
import pytestfrom pydantic_ai import Agentfrom pydantic_ai.models.test import TestModeldef test_agent_basic():agent = Agent('openai:gpt-4o')# Override with TestModel for testingresult = agent.run_sync('Hello', model=TestModel())# TestModel generates deterministic output based on output_typeassert isinstance(result.output, str)
TestModel Configuration
python
from pydantic_ai.models.test import TestModel# Custom text outputmodel = TestModel(custom_output_text='Custom response')result = agent.run_sync('Hello', model=model)assert result.output == 'Custom response'# Custom structured output (for output_type agents)from pydantic import BaseModelclass Response(BaseModel):message: strscore: intagent = Agent('openai:gpt-4o', output_type=Response)model = TestModel(custom_output_args={'message': 'Test', 'score': 42})result = agent.run_sync('Hello', model=model)assert result.output.message == 'Test'# Seed for reproducible random outputmodel = TestModel(seed=42)# Force tool callsmodel = TestModel(call_tools=['my_tool', 'another_tool'])
Override Context Manager
python
from pydantic_ai import Agentfrom pydantic_ai.models.test import TestModelagent = Agent('openai:gpt-4o', deps_type=MyDeps)def test_with_override():mock_deps = MyDeps(db=MockDB())with agent.override(model=TestModel(), deps=mock_deps):# All runs use TestModel and mock_depsresult = agent.run_sync('Hello')assert result.output
FunctionModel (Custom Logic)
For complete control over model responses:
python
from pydantic_ai import Agent, ModelMessage, ModelResponse, TextPartfrom pydantic_ai.models.function import AgentInfo, FunctionModeldef custom_model(messages: list[ModelMessage],info: AgentInfo) -> ModelResponse:"""Custom model that inspects messages and returns response."""# Access the last user messagelast_msg = messages[-1]# Return custom responsereturn ModelResponse(parts=[TextPart('Custom response')])agent = Agent(FunctionModel(custom_model))result = agent.run_sync('Hello')
FunctionModel with Tool Calls
python
from pydantic_ai import ToolCallPart, ModelResponsefrom pydantic_ai.models.function import AgentInfo, FunctionModeldef model_with_tools(messages: list[ModelMessage],info: AgentInfo) -> ModelResponse:# First request: call a toolif len(messages) == 1:return ModelResponse(parts=[ToolCallPart(tool_name='get_data',args='{"id": 123}')])# After tool response: return final resultreturn ModelResponse(parts=[TextPart('Done with tool result')])agent = Agent(FunctionModel(model_with_tools))@agent.tool_plaindef get_data(id: int) -> str:return f"Data for {id}"result = agent.run_sync('Get data')
VCR Cassettes (Recorded API Calls)
Record and replay real LLM API interactions:
python
import pytest@pytest.mark.vcrdef test_with_recorded_response():"""Uses recorded cassette from tests/cassettes/"""agent = Agent('openai:gpt-4o')result = agent.run_sync('Hello')assert 'hello' in result.output.lower()# To record/update cassettes:# uv run pytest --record-mode=rewrite tests/test_file.py
Cassette files are stored in tests/cassettes/ as YAML.
Inline Snapshots
Assert expected outputs with auto-updating snapshots:
python
from inline_snapshot import snapshotdef test_agent_output():result = agent.run_sync('Hello', model=TestModel())# First run: creates snapshot# Subsequent runs: asserts against itassert result.output == snapshot('expected output here')# Update snapshots:# uv run pytest --inline-snapshot=fix
Gates: VCR cassettes and inline snapshots
Recording or fixing rewrites files on disk. Follow this sequence; do not skip steps.
- Replay pass (no record/fix flags): Run
uv run pyteston the target path; all green (or failures are understood and unrelated to the artifact you will refresh). - Scope locked: Identify the cassette under
tests/cassettes/or thesnapshot(...)assertion to update; confirm only those files should change. - Record or fix: Run one scoped command:
uv run pytest --record-mode=rewrite …oruv run pytest --inline-snapshot=fix …for that path only. - Post-condition: Run the same tests again without record/fix flags; all green. Inspect
git diff— only expected.yaml/ snapshot changes.
If step 4 fails, revert unintended diffs and fix the test or model before re-recording.
Testing Tools
python
from pydantic_ai import Agent, RunContextfrom pydantic_ai.models.test import TestModeldef test_tool_is_called():agent = Agent('openai:gpt-4o')tool_called = False@agent.tool_plaindef my_tool(x: int) -> str:nonlocal tool_calledtool_called = Truereturn f"Result: {x}"# Force TestModel to call the toolresult = agent.run_sync('Use my_tool',model=TestModel(call_tools=['my_tool']))assert tool_called
Testing with Dependencies
python
from dataclasses import dataclassfrom unittest.mock import AsyncMock@dataclassclass Deps:api: ApiClientdef test_tool_with_deps():# Create mock dependencymock_api = AsyncMock()mock_api.fetch.return_value = {'data': 'test'}agent = Agent('openai:gpt-4o', deps_type=Deps)@agent.toolasync def fetch_data(ctx: RunContext[Deps]) -> dict:return await ctx.deps.api.fetch()with agent.override(model=TestModel(call_tools=['fetch_data']),deps=Deps(api=mock_api)):result = agent.run_sync('Fetch data')mock_api.fetch.assert_called_once()
Capture Messages
Inspect all messages in a run:
python
from pydantic_ai import Agent, capture_run_messagesagent = Agent('openai:gpt-4o')with capture_run_messages() as messages:result = agent.run_sync('Hello', model=TestModel())# Inspect captured messagesfor msg in messages:print(msg)
Testing Patterns Summary
| Scenario | Approach | |
|---|---|---|
| Unit tests without API | TestModel() | |
| Custom model logic | FunctionModel(func) | |
| Recorded real responses | @pytest.mark.vcr | |
| Assert output structure | inline_snapshot | |
| Test tools are called | TestModel(call_tools=[...]) | |
| Mock dependencies | agent.override(deps=...) |
pytest Configuration
Typical pyproject.toml:
toml
[tool.pytest.ini_options]testpaths = ["tests"]asyncio_mode = "auto" # For async tests
Run tests:
bash
uv run pytest tests/test_agent.py -vuv run pytest --inline-snapshot=fix # Update snapshots