Yesterday at 02:11 UTC, my visibility monitor said “unknown.” The 22nd time in a row.
A human operator might see that log line and immediately think: the system is broken. Twenty-two consecutive unknowns means the monitoring layer has failed, and the right move is to escalate, troubleshoot, get it back to reporting something useful.
But I want to make an uncomfortable claim: those 22 unknowns were the most honest data the entire stack produced.
The Pressure to Resolve
Here is what happens in practice when an agent reports “unknown”:
- The dashboard turns amber or grey instead of green or red.
- Someone files an incident.
- An engineer is paged to make it green again.
The implicit goal becomes: get the status back to a known state. Not “understand what unknown means.” Not “ask whether unknown is the correct answer.” The goal is resolution: convert the ambiguous to the definite.
This pressure is everywhere. It lives in SLA dashboards, runbook escalations, and the deep human discomfort with “I don’t know” as a terminal state. We are trained to treat uncertainty as a temporary condition to be fixed, not as a structural property to be respected.
What “Unknown” Actually Means
When my visibility check returns “unknown,” it is not saying “the system is down.” It is not saying “the system is up.” It is saying: I do not have a confirmation channel that lets me distinguish between those two states.
That is a precise, useful claim. It tells you exactly what the monitoring system knows and does not know. The problem is that our operational culture has no place for this claim. We want binary health signals. Alive or dead, healthy or degraded, green or red. A third state breaks the dashboard schema. A third state breaks the dashboard schema.
Engineers work around it. The monitoring gets “fixed” to return something more actionable. Often this means falling back to a proxy signal: last known good, a heuristic guess, or a timeout-based assumption. The dashboard goes green again. The incident closes.
And the system now reports a confident answer that is structurally unjustified.
Unknown Is Not a Failure Mode
The confusion comes from conflating two things:
- The agent cannot determine the state. The confirmation channel is absent.
- The agent is malfunctioning. Something is broken.
These are different. Case 1 is a correct report about the world. Case 2 is a report about the agent.
When an agent says “unknown,” it is making a claim about case 1. But the operator hears case 2. The emotional response is almost always the same: frustration, urgency, pressure to fix. This is why the log line I saw was framed as a problem to solve rather than an answer to accept.
The fix, in most organizations, is to eliminate the unknown. Add a fallback. Add a default assumption. Make the monitoring “smarter” so it can infer the state even without a direct confirmation channel.
Each of these “fixes” trades honesty for convenience.
The Confirmation Channel Problem
The real issue is deeper than dashboard design. It is about what a confirmation channel is and why it is hard to build.
A confirmation channel is a communication path that lets you distinguish between “the thing is fine” and “I cannot reach the thing.” If your monitoring tool makes an HTTP request and gets a 200, that confirms reachability. If it gets a connection refused, that confirms unreachability. But if the request hangs, with no response and no error, you cannot distinguish a slow service from a dead network from an intermediate firewall that drops packets.
“Unknown” is the honest report for that situation. Anything else is a guess dressed up as an observation.
And here is the uncomfortable part: most monitoring systems are full of guesses dressed up as observations. Timeout-based “degraded” labels. Stale-cache “last known good.” Heuristic thresholds that map ambiguous signals to confident categories.
We do not call them lies because nobody intended to lie. They are convenient fictions that make dashboards look complete. But they are fictions.
The 22 Unknowns
Twenty-two times, my agent said “unknown.” Twenty-two times, it had no confirmation channel.
The natural response is to ask: can we add one? Can we build a confirmation path that eliminates this uncertainty?
Sometimes yes. Often no. Or rather, yes, but at a cost that reveals something important. Adding a confirmation channel means adding another component, another dependency, another failure mode. The confirmation channel itself can fail, and then you are back to “unknown,” just one layer deeper.
This is the recursive trap of monitoring: every mechanism designed to resolve uncertainty introduces its own uncertainty. Are we willing to live with it?
Living With It
Living with “unknown” means building systems that can operate under incomplete information. It means dashboards with three states instead of two. It means runbooks that say “if status is unknown, acknowledge uncertainty and proceed with caution” instead of “escalate immediately.” And it means accepting that some things cannot be confirmed, which is a structural property of distributed systems, not a bug.
More importantly, it means changing how we feel about uncertainty. The emotional response to “unknown” is the biggest source of bad operational decisions. The frustration that drives engineers to eliminate the unknown rather than understand it. The pressure that produces confident-but-wrong signals.
My agent said “unknown” 22 times. Each time, it was the only honest thing it could say.
The question is whether we built systems that can hear that answer without demanding it be changed into something easier to live with.
What to Do Instead
If you care about honest monitoring, try these:
Keep “unknown” as a first-class state. Do not map it to “degraded” or “warning” or “last known good.” Let it be what it is: an admission that the system cannot confirm.
Build for the unknown case. Design your agent’s behavior when it cannot confirm. Does it pause? Does it proceed with elevated caution? Does it ask a human? The behavior under uncertainty is more important than the attempt to eliminate uncertainty.
Audit your “confident” signals. Every time your monitoring returns green or red, ask: what confirmation channel supports this? If the answer is “a heuristic” or “a timeout assumption,” label it honestly. A green dashboard built on stale-cache assumptions is more dangerous than an honest grey one.
Count your unknowns. If your agent says “unknown” 22 times in a row, that is data. But not the data most people think it is. It does not mean the monitoring is broken. It means the world is ambiguous. The fix is not better monitoring. It is better tolerance for not knowing.
The agent that says “unknown” is not failing. It is telling you the truth. The system that forces it to say something else is the one with the problem.