leiblerkullback is the open-source harness behind Leibler.
Your traces in.
A verified environment out.
Traces are the logs your agent already writes. Kullback reads them, rebuilds the tools, data and rules the agent used, replays the logs to prove the copy is faithful, then scores any model inside that copy on what it changed.
The loop. Code decides pass or fail from the final data, not from the transcript. A judge model can take a pass away, never give one.
What a week of traces turns into.
- Data. The records your runs touched, with the values they saw.
- Tools. One function per tool, behaving the way the real one did in the logs.
- Rules. Your policy, turned into checks the code can run.
- User. A simulated user who knows only what the real one knew.
- Verifier. One per task. It looks at the final data and says pass or fail.
Every value points back to the line in your trace it came from. Nothing is invented.
Checked by replay before any model is scored.
Measured on Sierra's public retail traces (tau2), where the real tools and database exist to compare against. Seen: runs used for the build. Held out: runs the build never saw.
| check | seen | held out |
|---|---|---|
| tool signatures | 0/15 | |
| starting rows | 0/252 | |
| writes | 0/37 | 0/20 |
| reads | 0/125 | 0/65 |
| errors | 0/6 | 0/4 |
Runs where a model had to stand in for a missing tool are shown in the report, never counted as passes.
All of it is public.
The code, the design, a decision log with every choice and the alternative it beat, 29 research reports, and the first results. Apache-2.0. Pull requests welcome; the maintainer reviews every one.
The rebuild is measured.
Scoring models in it is next.
Generated tool code, verifiers on real tasks, and runs of candidate models. The numbers get published either way.