LLM apps are easy to build and hard to trust.
EvalStack is an open source eval framework for LLM and multi-agent systems. Write down what a good answer looks like, run your model against it, and know whether it works before your users find out.
Get started Learn the basicsWhy this exists
Most teams ship an LLM app, try a few prompts by hand and call it done. Then it fails quietly in production. EvalStack gives you a repeatable way to check model output, so a change to a prompt or a model shows up as a number, not a surprise.
What works today
Eval loop
Send a prompt, score the reply, get pass/fail with a reason. Run one case or a batch.
Scorers
Exact match, contains, and an LLM judge that grades meaning from 0.0 to 1.0.
Any provider
Anthropic and OpenAI built in, or plug in any function that takes a prompt and returns text.
Honest failures
API errors, refusals and bad judge replies raise errors. A broken eval never looks like a pass.
Quick start
Clone the repo and install it with the Anthropic SDK:
git clone https://github.com/ajithjoseph/evalstack
cd evalstack
pip install -e ".[anthropic]"Then run your first eval:
from evalstack.core import EvalRunner
from evalstack.scorer import LLMJudgeScorer
runner = EvalRunner(model="claude-opus-5-5")
result = runner.run("What is the capital of France?", expected="Paris")
print(result.passed, result.metadata["reason"])
# Grade meaning instead of matching text:
judge = LLMJudgeScorer(model="claude-opus-5-5")
runner = EvalRunner(model="claude-opus-5-5", scorer=judge)Use provider="openai" with pip install -e ".[openai]"
for OpenAI models. Python 3.11 or newer.
Roadmap
- Core eval loop (single LLM call)
- Multi-agent workflow eval
- RAG retrieval quality scoring
- MCP server integration
- Production dashboard
Building in public
EvalStack is early and moving fast. Follow the build log on LinkedIn, star the repo on GitHub, and open a PR if you want to help.