What is LangWatch Scenario?
Scenario, from the LLM observability company LangWatch, tests an agent by talking to it. You describe a situation in plain language; a UserSimulatorAgent writes the user's messages, your agent replies through a single call() adapter, and a JudgeAgent checks the conversation against your criteria on every turn and ends it with a verdict. Scripts can mix fixed messages, generated ones, assertions on tool calls and mocked tool results, and a RedTeamAgent swaps the simulated user for a multi-turn attacker using Crescendo-style escalation.
Tests run as ordinary pytest (Python) or vitest (TypeScript) tests; Go is listed but undocumented in the README. Voice agents can be tested through adapters for ElevenLabs, OpenAI Realtime, Twilio, Pipecat and Gemini Live, with injected audio, background noise and p50 and p95 latency figures. A cache_key makes the simulated user repeat the same inputs across runs.
The repository has about 1,000 stars and is Apache-2.0. The simulator and judge need an LLM key (examples default to openai/gpt-4.1-mini); a LangWatch key is optional and adds a live view of runs.
Is LangWatch Scenario worth it?
Single-prompt evals miss how agents fail. An agent can answer the first question well and then forget a constraint on turn four, or call the wrong tool when the user changes their mind. Scenario tests that by simulating the whole conversation, and its main achievement is making that look like any other test in the suite: a pytest or vitest function with a description, a list of judge criteria and an assert on the result.
The scripting model is the strongest part. You can let the simulator run on autopilot up to max_turns, or pin the first message, inject a mocked tool result, assert that a particular tool was called and then let the LLM continue. That mix keeps tests focused on behaviour you care about instead of whatever the simulator happens to say. Voice support goes further than rivals in the open-source space, with interruption tests and latency percentiles.
The catch: two models drive every test, and both cost tokens and both vary. Caching the simulated user with a cache_key, and wrapping your agent's own LLM calls with @scenario.cache(), restores repeatability, but a changed prompt invalidates the cache. Even voice-only tests need an OpenAI key, because the simulator and judge are text models by default. The project is small, about 1,000 stars with 165 open issues, and the README has visible slips such as a voice section quoting the wrong package name.
LangWatch's cloud is optional. Without a key, results stay in your terminal; with one, the free Developer tier allows 3 simulations, so teams wanting a shared dashboard will land on Growth at €29 a seat.
Scenario fits best next to an existing eval library rather than in place of one: Giskard or DeepEval for metrics and scans, Scenario for the conversations that only show up across several turns.
Where it earns its place
A developer building a booking agent for a restaurant group
Make sure the agent still asks for party size when a user changes the date mid-conversation
Write a scenario where the simulated user changes plans on turn three, and give the judge the criterion that party size is confirmed before booking; run it in CI with a cache_key so failures reproduce.
A voice-AI team at a 30-person telehealth startup
Check the phone agent copes with background noise and interruptions
Use the Twilio or ElevenLabs adapter, inject cafe noise at 0.3 and an interruption mid-sentence, and track p95 latency across runs; an OpenAI key is still needed for the simulator and judge.
Before you commit
- Setup
- uv add langwatch-scenario pytest, or pnpm install @langwatch/scenario vitest; set OPENAI_API_KEY, write an adapter with one call() method and run the test file.
- Integrations
- pytest and vitest; LiteLLM and the Vercel AI SDK in examples; agents already connected to LangWatch need no adapter; voice adapters for ElevenLabs, OpenAI Realtime, Twilio Media Streams, Pipecat and Gemini Live.
- Data and privacy
- Runs locally; conversations go to the LLM provider configured for the simulator and judge, and to LangWatch only if a LANGWATCH_API_KEY is set. The cache sits in ~/.scenario/cache.
- Cost at scale
- The library is free; the cost is two LLMs per turn per test. LangWatch Growth is €29 per core seat a month with 200k events, then €5 per 100k, and each simulation step counts as an event.
- Limits
- max_turns caps a conversation; LiveKit, Vapi and generic WebRTC voice adapters are planned, not shipped.
What’s good
- Multi-turn agent tests written as ordinary pytest or vitest tests
- Scripts mix fixed and simulated turns, tool-call assertions and mocks
- Red-team agent with Crescendo-style multi-turn escalation
- Voice adapters with noise injection and latency percentiles
- Caching for repeatable runs
What’s not
- Simulator and judge LLM calls on every run cost tokens
- Results vary between runs unless cached
- Go support listed but undocumented
- Small project with 165 open issues and README errors
Best for
Teams building chat or voice agents who want conversation-level regression tests in their existing test runner.
LangWatch Scenario alternatives
LangWatch Scenario: common questions
- Is LangWatch Scenario worth it?
- Scenario is the clearest way to write multi-turn agent tests as normal unit tests, provided you accept that an LLM on both sides makes every run a little different and a little costly. We rate it 3.9 out of 5.
- How much does LangWatch Scenario cost?
- Free and open source (Apache-2.0); you pay for simulator and judge tokens. Optional LangWatch cloud: Developer €0 (50k events, 3 simulations); Growth €29 per core seat a month with 200k events, then €5 per 100k.
- What are the downsides of LangWatch Scenario?
- Simulator and judge LLM calls on every run cost tokens; results vary between runs unless cached; go support listed but undocumented; small project with 165 open issues and README errors.
- Who is LangWatch Scenario best for?
- Teams building chat or voice agents who want conversation-level regression tests in their existing test runner.
- How much does LangWatch Scenario cost for a team or in production?
- The library is free; the cost is two LLMs per turn per test. LangWatch Growth is €29 per core seat a month with 200k events, then €5 per 100k, and each simulation step counts as an event.
- Is LangWatch Scenario safe for company data?
- Runs locally; conversations go to the LLM provider configured for the simulator and judge, and to LangWatch only if a LANGWATCH_API_KEY is set. The cache sits in ~/.scenario/cache.
- What are the best LangWatch Scenario alternatives?
- The closest alternatives we have reviewed are Giskard, DeepEval, promptfoo and Langfuse.
Answers checked Nov 6, 2026.