Attest
Testing framework that checks what AI agents do, not just what they say
Visit website →About
Attest is an open-source testing framework for AI agents in Python and TypeScript. It checks an agent reply, a sequence of tool calls, or a change to external state using pytest or Vitest.
It follows a rec, replay, gate pattern: capture the real incident, replay the agent against recorded model responses with zero live model calls, then gate the merge. This removes live-model variability from tests while keeping other dependencies under control.
Attest provides six evaluation layers: schema validation, cost and token constraint checks, trace and behavioral checks, content pattern matching, embedding similarity, and LLM-as-judge. The first four layers are free and fully deterministic.
It ships adapters for OpenAI, Anthropic, Gemini, LangChain, CrewAI, Ollama, and more, with beta support for multi-agent trace delegation and simulation mode with mock tools and seeded faults.
AI-assisted draft, human-reviewed before publishing — see how we choose & review tools.
Why we picked it
Testing AI agents is hard because models are non-deterministic. Attest solves this elegantly with strict replay: record model responses then replay with zero live calls. The four free deterministic layers mean most tests need no API key. A mature framework that fills a real gap in agent testing tooling.