IBM Research: same model, different harness — AppWorld from 30% to 81%

IBM Research published a controlled comparison on Hugging Face: the same open-weight model, GLM-5.3-flash, on 90 AppWorld tasks (30 scenarios), changing only the agent harness. Task completion (TGC) was 81.1% (73/90) for CUGA, 46.7% (42/90) for Hermes, and 30.0% (27/90) for Deep Agents; stricter scenario-all-pass rates (SGC) were 66.7%, 26.7%, and 6.7%. The authors stress this is not a verdict on Hermes or Deep Agents—neither was built for multi-app API work—but that the harness must fit the job. CUGA is IBM's open-source enterprise harness that can enforce policies in the harness rather than only in the prompt; the evaluation setup is public in harbor-eval.

Written by dalao.ai: the title, summary and why it matters are our own words. The original reports are the record; their publishers keep authorship.