LobeHubFrom the builder
Saying "Done" Isn’t Enough: How Lobe Eval Tests Agents on Real-World Tasks
LobeHub describes Lobe Eval, which runs agents through the product's real server-side execution path in a stateful, resettable tool environment and judges each run on execution traces, tool results and artifact evidence. Each test case freezes the facts, permissions and tools the agent sees, including distractor tools, without dictating a solution path, and external services are simulated after calibrating against live MCP servers. Its main metric, Pass³, requires all three isolated runs to pass, unlike pass@3, which counts a single success out of three.
A Case is not a prompt or a reference answer. It is a resettable task environment paired with an acceptance contract.
Saved . A shortened excerpt from the official blog; the original publisher retains authorship.