Then I looked at what the agent actually did. Nothing. It never touched the data. It guessed.

That is the moment I stopped trusting text-based evals for agents, and the reason I have been playing with Silo since.

Your eval grades essays. Your agent does surgery.

Here is the thing nobody says out loud: most agent evals are essay grading. You show the agent a prompt, it writes something back, and you (or an LLM judge) score how good the writing looks. That was fine when agents were chatbots.

Modern agents are not chatbots. They call tools, update records, move inventory, change the state of systems. An agent can write a beautiful summary of a pipeline while leaving every deal in the wrong stage. If your eval only reads the transcript, that failure is invisible. You ship it, and production finds out for you.

There are roughly three levels to this:

1. Unit tests. Does the code work?

2. Prompt evals. Does the answer look right?

3. Actually watching what the agent did. Did the world end up in the right state?

Silo is a tool for level 3. It is open source (MIT), local-first, TypeScript-first, and it gives your existing agent a realistic simulated world to work in: tools, seeded data, tasks, and verifiers that grade outcomes deterministically. Nothing leaves your machine.

The five-minute version

You need Node 22+ and "type": "module" in your package.json. Then:

npm i @burn0/silo
npx @burn0/silo init demo --template crm
npx @burn0/silo env validate --env demo

OK   demo
data=9 tasks=6 tools=42 verifiers=6

You now have a realistic simulated B2B sales pipeline on your laptop: 9 data collections, 6 tasks, 42 tools an agent can call, 6 verifiers that decide pass or fail. Small enough to hold in your head, which is exactly why it is the best template to start with. (There is also an ERP template with 185 tools and deliberately messy seed data. An invoice that bills more than was received. A payment that failed on stale bank details. Someone on the Silo team has seen things.)

Write an agent. Anything that default-exports a function counts, any framework or a raw API loop:

// silo.agent.js
export default async function agent({ callTool }) {
  const forecast = await callTool("forecast_report", {});
  const { weightedAmount } = forecast.output;

  return { output: `Open pipeline is worth $${weightedAmount.amount}` };
}

Run it against a real task:

npx @burn0/silo run --env demo --task TASK-004 --agent ./silo.agent.js

  Task          TASK-004 — Report the weighted value of open pipeline
  Result        PASS
  Reward        1.00

  Tool calls    1
  Checks        3 / 3
  Required      1 / 1

  Run saved: .silo/runs/run_20260915012734_4t01

Fine. Now here is the part that got me.

The verifier does not read your agent's essay

In Silo, your agent never sees the world state. It gets the task instruction, the tool schemas, and cloned tool output. If it wants to know something, it has to call a tool for it, exactly like production.

Grading works the same way in reverse. A task passes because the world changed correctly, never because the agent said the right words. Remember my lying agent from the top of this post? Under Silo it fails loudly: zero tool calls, checks unmet, done. No partial credit for a confident tone.

Two design rules keep this honest:

• Time is simulated. state.now is the only clock. Nothing reads Date.now(), so a run from Tuesday reproduces exactly on Friday.

• Every run leaves evidence. Each rollout writes a directory: trace.jsonl (every event, append-only), result.json (checks, reward, tool errors), state-diff.json (exactly what changed), run.json (task, verifier, timings).

My favorite detail: result.json and state-diff.json contain no timestamps or run ids. That makes them an exact regression oracle. Diff two runs and any difference is a genuine behavioral change, not clock noise. And when your agent is non-deterministic, --runs 5 repeats the task so you see the real distribution instead of the lucky run you would have screenshotted.

The honest caveats

Silo is early, v0.4.0, and the maintainers say plainly to expect breaking changes before 1.0. Packaged adapters for LangChain, Vercel AI SDK, OpenAI, Anthropic, and Mastra are still roadmap, as are an MCP server and LLM judges to sit alongside the deterministic checks. If you need those today, you will be writing some glue.

But the core loop works right now, and it rearranged how I think about agent testing in about an afternoon. Stop asking "did it write a good answer" and start asking "is the world right." Those are different questions, and only one of them protects production.

Links, since you will ask:

• Site: silo.burn0.dev

• Docs: docs.burn0.dev/silo/introduction

• Repo: github.com/burn0-dev/silo

Start with the CRM template. Then try to make your agent lie to the verifier. It is harder than you think, and that is the point.