Four dispatches into this series, I've been building an architecture case. This week is the accountability case: where these systems actually fail in production, and why the market conversation around them so rarely gets there.
Eight failure domains, none of them mysterious
Telemetry and time comes first. Loss, delay, duplicates, clock error, or parser drift create an incomplete or false sequence — and a model operating on partial context will still produce a confident explanation, because nothing in its architecture tells it to flag what it can't see. The event pipeline sets the observability boundary for the entire system, and no amount of downstream AI sophistication extends that boundary by a single event. The fix isn't exotic: coverage and freshness metrics, source authentication, replay, explicit event/ingestion/processing timestamps, and a defined lateness and revision policy.
Normalization and state is second. Identity or asset misresolution changes risk, target, or blast-radius classification in ways that are easy to miss. A privileged user misidentified as low-risk changes prioritization. A production server misclassified as a workstation changes what "safe" remediation even means. The control here is unglamorous: versioned mappings, provenance, confidence scores, effective-time semantics, reconciliation, and state-health alarms — not a smarter model.
Graph and retrieval is third. Stale, poisoned, unauthorized, or weakly related context can become persuasive evidence the moment a retrieval layer pulls it in. From the outside, this looks like a reasoning failure. Architecturally, it's a retrieval and context-selection failure wearing a reasoning costume, and it needs pre-retrieval authorization, tenant isolation, index lineage, passage-level citations, and explicit staleness flags to catch.
Model and planning is fourth. Unsupported claims, poor calibration, or goal hijacking can steer the system toward the wrong investigation or the wrong action entirely. This is the domain that gets the most attention and probably deserves the least exclusive blame — it's contained by task-specific evaluation, evidentiary checks, independent validation, explicit policy constraints, and abstention rules for when the evidence just isn't there yet.
Authority and tools is fifth, and it's the one that actually hurts. Overbroad credentials, ambiguous targets, or confused-deputy behavior can take a merely-wrong proposal and amplify it into real damage. A system that disables accounts on ambiguous identity signals creates its own denial of service. A system that auto-blocks infrastructure on noisy indicators takes down production. The fix isn't less automation — it's automation designed as a controlled system: least privilege, short-lived scoped credentials, separation of duties, canonical target resolution, strict schemas, and approval gates on anything with real consequences.
Execution and feedback is sixth. A remediation workflow that assumes success because an API call returned a 200, without ever verifying the underlying system actually applied the change, will mark an incident "contained" when it isn't. Retries without idempotency keys duplicate side effects. This is the control-plane half of the problem, and it needs rate limits, circuit breakers, independent readback, canaries, and real rollback or compensating actions — not a better language model.
Human oversight is seventh, and it's the domain the industry talks about least. Automation bias and poor approval context can turn a nominal review into rubber-stamping — a reviewer under time pressure approves a plausible-looking recommendation without actually checking the evidence behind it. An approval step that doesn't surface evidence, counterevidence, and blast radius isn't oversight. It's theater that happens to have a human standing next to it.
Multi-agent and supply chain is eighth, and it's the newest one in practice. As soon as more than one agent is involved — one retrieving, one proposing, one executing — privileges can compose across them in ways no single agent's permissions would predict. A compromised connector, package, index, or policy bundle can alter behavior anywhere in that chain. This needs delegation policy, message authenticity between agents, shared-memory isolation, component provenance, signed releases, and a real kill switch.
None of these eight are exotic. They're well-understood distributed-systems, data-quality, and control-plane problems, most of them decades old, now showing up with an AI layer sitting on top.
Why the market conversation rarely gets here
It would be convenient to say the market is simply built to reward describing this problem inaccurately, and there's a version of that story that writes itself: AI capability is demonstrable in forty-five minutes, while pipeline reliability and blast-radius controls are invisible in a demo; buyers facing staffing pressure want a procurement category that reads as progress in a board review; coverage tends to follow the demonstration, not the ordering guarantees underneath it. Those pressures are real, and they're not evenly distributed — but without a defined accounting of who's claiming what, it isn't responsible to assume a uniform motive, or to assume most of the market is acting in bad faith. Some vendors are describing a real shift precisely. Some buyers are asking exactly the right questions. The more accurate diagnosis is an observability imbalance, not a conspiracy: public demonstrations simply reveal interface behavior far more readily than they reveal backpressure handling, tenant isolation, calibration, or compensating actions, and that imbalance rewards the vocabulary that's easiest to show rather than the engineering that's hardest to fake.
The corrective isn't outrage. It's specificity. Market language is context; execution traces, test results, and control artifacts are evidence. Ask which goals persist, which actions get selected dynamically, what authority is actually delegated, and what evidence backs the safety claims — and the branding question turns into a verifiable one.