Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Eval-Driven Development

A worked example of a Pydantic AI agent whose prompt, model and eval rules sit in one versioned file, graded against stored traces.

Reviewed by ReviewRamble editors · Updated

Share this page:XLinkedInFacebookRedditWhatsAppEmail

Our rating

3.0/5

A tidy blueprint for versioning an agent's prompt with its tests and grading stored traces, but with no licence and four days of commits it is a reference to read, not a dependency.

What is Eval-Driven Development?

This repository is a reference project, not a library. It builds one small agent, a financial-news assistant on Pydantic AI that calls an OpenAI model and a SearchAPI MCP search server, and wraps it in the testing and monitoring a production agent would need.

The centre is prompt_eval.json, a versioned contract that keeps the agent's prompt, model and pass rules together: words the answer must contain, limits on model requests and tool calls, and a rubric for an LLM judge. Three scripts check behaviour against it: repeated live runs, deterministic checks on traces already stored in Grafana Tempo, and an LLM-as-judge pass over those stored answers. A Docker Compose file brings up Prometheus, Loki, Tempo, SeaweedFS and Grafana locally, and two long write-ups in docs/blog explain the reasoning.

It has a single author (alexcpn), was created on 14 August 2026 and has 17 commits, the last on 17 August 2026.

Is Eval-Driven Development worth it?

Eval-Driven Development is a clear, compact demonstration of treating an agent's prompt as code with tests, and a demonstration is all it is. It has 10 GitHub stars, 2 forks and 17 commits made over four days in August 2026, with no activity since.

The idea worth taking away is the contract file. Because the prompt, its version, the model and the eval policy live in one JSON record, and every trace carries the agent_id and prompt_version, a stored trace can always be graded against the rules in force when it ran. The trace mode counts model requests and tool calls from OpenTelemetry span names without spending tokens; the judge mode spends only judge-model tokens (gpt-4o-mini by default). The live mode runs each case 3 times by default and exits with status 1 on any failure, so it drops into a CI pipeline. A small custom MinToolCalls evaluator fills a gap in Pydantic Evals, which ships only a maximum.

The catch: the stack is a teaching setup, not a safe default. The Compose file publishes Grafana, Prometheus, Loki and Tempo on all network interfaces rather than localhost, and Grafana starts with admin/admin; only SeaweedFS is bound to 127.0.0.1. Run it on a machine no one else can reach, or change the port bindings first. The README says traces are kept for 24 hours, while tempo.yml sets 720 hours (30 days). There is no licence file, so you may read the code but have no granted right to copy it into your own project. The sample contract's agent model is openai:gpt-4, and the example covers a single OpenAI-plus-search agent.

Our take: this is a well-annotated blueprint for wiring Pydantic Evals to Tempo traces, best rebuilt by hand in your own code; for a maintained tool to run, Promptfoo, DeepEval or Inspect cover evaluation with a licence and a user base.

Where it earns its place

  • A backend engineer adding an LLM agent to a fintech product

    Set up a regression gate before the agent ships

    Follow the contract-file pattern: one JSON record per agent version with required words and request and tool-call limits, run the live eval 3 times per case in CI and fail the build on exit code 1. Each live run spends OpenAI and SearchAPI quota.

  • A platform team already running Grafana Tempo

    Check production traces against the prompt version that produced them

    Adapt the trace script to sample 10% of the last hour's traces and grade tool-call counts from span names; it makes no model calls, so an hourly check costs nothing. The judge script adds rubric grading at gpt-4o-mini rates.

Before you commit

Setup
Needs Python 3.10+, uv, Docker Compose, an OpenAI key and a SearchAPI MCP token; start the observability stack with docker compose, then uv run python main.py.
Integrations
Pydantic AI and Pydantic Evals, OpenAI models, MCP through MCPToolset, OpenTelemetry, Prometheus, Loki, Tempo, SeaweedFS and Grafana.
Data and privacy
Prompts and answers go to OpenAI and search queries to SearchAPI; traces and logs stay in local Docker volumes. Tempo keeps traces for 720 hours per tempo.yml, and Loki has no retention limit set.
Limits
No licence; Grafana, Prometheus, Loki and Tempo ports bind to all interfaces with default Grafana credentials; one example agent; no tagged releases.

What’s good

  • One versioned file ties prompt, model and eval rules together
  • Grades stored Tempo traces without new model or search calls
  • Exit codes plus Markdown and JSON reports suit CI
  • Detailed README and two design write-ups

What’s not

  • No licence file
  • Grafana, Prometheus, Loki and Tempo exposed on all interfaces; Grafana login admin/admin
  • Single author, 17 commits, none since 17 August 2026
  • README and config disagree on trace retention

Best for

Engineers building their first production agent on Pydantic AI who want a worked pattern for evals and tracing to reimplement.

Eval-Driven Development alternatives

Eval-Driven Development: common questions

Is Eval-Driven Development worth it?
A tidy blueprint for versioning an agent's prompt with its tests and grading stored traces, but with no licence and four days of commits it is a reference to read, not a dependency. We rate it 3.0 out of 5.
How much does Eval-Driven Development cost?
Free to read, but the repository has no licence file, so reuse rights are not granted. Running it costs OpenAI and SearchAPI usage.
What are the downsides of Eval-Driven Development?
No licence file; grafana, Prometheus, Loki and Tempo exposed on all interfaces; Grafana login admin/admin; single author, 17 commits, none since 17 August 2026; rEADME and config disagree on trace retention.
Who is Eval-Driven Development best for?
Engineers building their first production agent on Pydantic AI who want a worked pattern for evals and tracing to reimplement.
Is Eval-Driven Development safe for company data?
Prompts and answers go to OpenAI and search queries to SearchAPI; traces and logs stay in local Docker volumes. Tempo keeps traces for 720 hours per tempo.yml, and Loki has no retention limit set.
What are the best Eval-Driven Development alternatives?
The closest alternatives we have reviewed are promptfoo, DeepEval, Inspect and Langfuse.

Answers checked Nov 18, 2026.