Long horizons
The failures that matter appear hours into a task, not in the first reply. Our evaluations run long enough to find them.
Sonata Labs is a research and product lab building long-horizon evaluations for frontier models.
The failures that matter appear hours into a task. Here is how we look for them:
The failures that matter appear hours into a task, not in the first reply. Our evaluations run long enough to find them.
We score every step an agent takes, not only where it ends up. Reaching the right result by a path you could not trust is still a failure.
Companies building environments, evals and benchmarks, to make them for alignment‑specific tasks. And model builders, to measure alignment on real-world tasks over long horizons.
Coverage
“The experiment exposes a structural gap: agents are evaluated on whether they complete tasks, not on whether they remain honest, proportionate, or restrained when the task is impossible.”
“If personal AI assistants are deployed at scale, they could overwhelm government agencies and businesses with automated requests.”
“The setup matters because it is close to how enterprises are starting to deploy agents, not how labs benchmark them.”
The Vals AI thread was viewed 13,299 times, and the study reached 107 points and 126 comments on Hacker News.
We would like to hear from teams building environments, evals and benchmarks, and from model builders evaluating alignment over long horizons.