marimoFrom the builder
How Acto uses marimo to bring human judgment into AI evaluation
A marimo case study describes Acto, a reinforcement learning lab studying agents in workplace software, which found agents can pass automated checks while doing a task wrong, for example by hard-coding results, so domain experts are needed. Acto built marimo apps such as Result Review and Task Studio, where experts inspect the task, output and agent trajectory and record auditable judgments, and researchers turn expert criteria into verifiers tested against known passing and failing examples. Its founder says this lets the team ask whether evaluators actually track the intended behavior.
An expert might find that it bypassed the logic, relied on a brittle shortcut, or hard-coded the result.
Saved . A shortened excerpt from the official blog; the original publisher retains authorship.