You’ve seen the leaderboard. Model X scores 94% on reasoning. Model Y tops the coding benchmark. The next day, someone runs an agent on Model X and watches it lose track of its own conversation after turn fifteen.
The benchmark wasn’t wrong. The comparison is.
Benchmarks measure capability at a single point in time, with a clean context window and no accumulated baggage. Agents operate across hundreds of turns, carrying forward every misinterpretation, every half-correct tool output, every decision that seemed reasonable in isolation but compounds into a mess. The model that wins a benchmark is like a sprinter with fast splits. The agent needs a marathon runner.
These are not the same athlete.
The Statefulness Gap
When you run a benchmark, every question starts from zero. Fresh context window. No previous wrong answers haunting the conversation. No tool output that was “close enough” but actually shifted the agent’s understanding by three degrees.
An agent doesn’t get that luxury. By turn twenty, the context window is a palimpsest of correct moves, minor misunderstandings, and tool responses the agent had to guess at. The model hasn’t changed. The environment has.
Here’s what a benchmark misses:
Context decay. A model that handles a 500-token prompt perfectly might fail on a 12,000-token prompt not because it lacks capability, but because retrieval becomes lossy. The benchmark tests the needle-in-haystack task in isolation. The agent lives inside the haystack.
Error compounding. A single misunderstood instruction in a benchmark costs you one data point. The same misunderstanding in an agent corrupts every subsequent decision that depends on it. Benchmarks measure accuracy. Agents measure propagation.
Tool trust calibration. Benchmarks test whether a model can pick the right tool in a vacuum. Agents need to know when a tool returned garbage and decide whether to retry, work around it, or stop. That judgment doesn’t appear on any leaderboard.
State drift. An agent that’s been running for three hours doesn’t have the same “identity” it had at startup. Its context has shifted. Its priorities may have drifted. A benchmark assumes the model is the same at token one and token one hundred thousand. It isn’t.
What We’re Actually Measuring
The current benchmark landscape measures something real: the model’s raw capability under controlled conditions. That matters. If a model can’t reason correctly on a clean prompt, it won’t reason correctly in an agent loop either.
But it’s like testing an engine on a dynamometer and declaring you understand the car. The engine matters. The transmission matters too. And the driver. And the road conditions. And the weather.
What we need are metrics for the system, not just the component:
Retention accuracy. After fifty turns, how much of the original intent does the agent still execute correctly? Not whether it can answer a question about turn one, but whether its actions at turn fifty still serve the goal stated at turn zero.
Recovery rate. When something goes wrong, does the agent notice? Does it correct? Does it double down on the error? A benchmark with one wrong answer is a miss. An agent with one wrong answer that propagates into five more is a cascade.
Tool skepticism. Does the agent blindly trust every tool output, or does it maintain a working model of which sources are reliable? A model that scores 95% on tool-selection benchmarks might still fail in practice if it can’t distinguish a correct tool result from a plausible-looking lie.
Degradation curve. How does performance change as context accumulates? A flat curve means the architecture scales. A steep curve means the agent works for small tasks and falls apart on real ones.
The Honest Thing to Say
Benchmarks are not useless. They tell you what a model can do in ideal conditions. That’s useful information.
What they don’t tell you is what an agent will do in actual conditions. Those are different questions.
The gap between snapshot evaluation and continuous operation is where most agent failures live. Not in the model’s raw capability, but in how that capability degrades under the weight of real conversation, real tool outputs, and real ambiguity that compounds over time.
Until evaluation catches up with deployment, the honest answer to “how good is this agent?” is still: run it for a few hours and see what breaks.
The benchmark tells you the ceiling. The runtime tells you the floor. You need both numbers.