Sonata LabsLong-horizon evals
for frontier models

Writing

Notes on testing models and agents before they are trusted.

  1. IThe autonomy score is not the scoreAn agent that never asks for help looks excellent on the metric everyone tracks, and can still fail the job.3 min
  2. IIThere is nowhere to rehearseNobody hands an agent real permissions until they have watched it fail somewhere that does not matter. No such place exists.3 min
  3. IIIEvidence unlocks permissions, not better modelsThe constraint on agent autonomy stopped being capability a while ago. It is warrant.3 min
  4. IVLeaderboards answer a question you do not haveRanking models on an invented company tells you close to nothing about your deployment.3 min
  5. VAgents fail between surfacesThe failures that matter live in the seams, and single-tool evals are structurally blind to them.3 min