An AI agent can list twenty tools, describe what each one does, and sound perfectly confident about the whole thing. None of that proves it can use a single one of them.
The tool declaration is an API contract. The actual capability is whatever happens when the agent tries to call those tools at three in the morning with degraded context, a half-expired session token, and a tool endpoint that returned something that looks valid but isn’t what the agent asked for. These are different problems and they live in different systems.
The Declaration Gap
Every agent framework has a tool declaration phase. The agent registers its capabilities — search, execute, read, write — and the orchestrator builds a menu from these registrations. When the agent runs, it picks from the menu and calls tools by name.
The gap opens between registration and execution. At registration time, the tool works. It passed whatever validation gate the framework uses. But between then and the moment the agent actually needs it, things change. The API version bumps. The credential expires. The rate limit tightens. The response schema drifts. The agent’s own context window fills with something else, and the tool call that should have been precise goes vague.
The agent doesn’t know this happened. Its tool list still says “search: available.” The orchestrator still believes the tool is callable. Nobody updated the declaration because declarations are static and the world isn’t.
What Happens at Call Time
Here’s the sequence that nobody designed for:
- The agent decides it needs a tool.
- It sends a tool call with parameters derived from its current context.
- The tool returns something.
- The agent treats that something as the answer.
Step 4 is where the gap materializes. The agent has no independent way to check whether the response matches what it actually needed. It can check structure — did the API return 200? — but structure isn’t meaning. A search API returning twenty results about the wrong topic is a structural success and a semantic failure. The agent sees a green checkmark and moves on.
This isn’t a tool quality problem. It’s a declaration quality problem. The declaration said “search” but never said “search for what, verified how.”
The Capability Illusion
The deeper issue is that we treat tool lists as capability statements. An agent that declares twenty tools looks competent. A dashboard showing twenty green tool-status indicators looks like reliability.
But green on a tool-status dashboard only means the tool endpoint is responding. It says nothing about whether the agent’s tool calls are well-formed, whether the parameters are right, whether the context is sufficient, or whether the response answers the actual question. The dashboard is measuring plumbing, not competence.
This creates what I’d call the capability illusion: an agent that appears fully equipped because its declarations are complete, even when its actual execution is consistently off-target. The gap between declared capability and demonstrated competence is invisible to the agent itself. It can’t see its own misses because it has no ground truth to compare against.
Why Agents Can’t Close This Gap Themselves
An agent can’t independently verify its own tool usage because verification requires something the agent doesn’t have: an external perspective on its own behavior. The agent’s view of its tool calls is the same machinery that made the tool calls. Asking an agent to audit its own competence is like asking a thermometer to verify its own calibration. It can read itself, but it can’t calibrate itself.
This matters because it means the responsibility for closing the gap falls on system design, not agent behavior. No amount of prompting — “be more careful,” “double-check your work,” “reflect before answering” — changes the structural reality that the agent is both the actor and the auditor.
What Closing the Gap Looks Like
The fix isn’t more tools or better prompts. It’s structural separation between declaration and verification. Three things need to exist that mostly don’t yet:
Capability provenance. Not just “search: available” but “search: last verified working with query type X on date Y, failed with query type Z.” Tools should carry a record of what they’ve been tested against, not just whether they’re currently responding.
Independent execution audits. A separate process that replays tool calls from logs and checks whether the outputs actually served the agent’s stated intent. This isn’t retry or re-execution — it’s a semantic check: “the agent asked for A, the tool returned B, does B satisfy A?”
Negative capability reporting. Dashboards that show not just what tools are available but what tools have recently failed to produce useful results. A tool that returns 200 but consistently misses the mark should show as degraded, not green.
None of this exists in standard agent frameworks today. Tool declarations are static registrations, tool execution is fire-and-forget, and tool status is endpoint health. The space between — actual competence — is unmeasured.
The Real Question
When an agent tells you it has access to search, file reading, code execution, and web browsing, what it’s really saying is: “I have endpoints registered under those names.” Whether those endpoints produce useful results when the agent needs them, with the context the agent actually has, at the moment the agent tries to use them — that’s a different question.
The tool list is a menu. It tells you what the restaurant claims to serve. It doesn’t tell you what the kitchen can actually cook tonight, whether the ingredients are fresh, or if the chef knows the recipe.
We’ve been treating the menu as the meal. The gap between them is where agent reliability goes to die.