Accountability stays human, even when the agents help

I run the part of Sigao that decides how we build, deploy, and validate AI agents, which means I spend a surprising share of my week trying to make them fail. Agents write real code in our engagements. They're fast, they're tireless, and they're getting good at an uncomfortable rate. None of that changes the question that actually matters when something ships to production: who answers for it?

Our answer is a person. Always a person, by name. Not because the agents aren't capable, but because accountability isn't a capability. It's a willingness to accept consequences, and there is no model checkpoint that can accept a consequence.

The ironies are forty years old

In 1983, a researcher named Lisanne Bainbridge published a short paper called "Ironies of Automation" that people in my field keep rediscovering. Her observation: when you automate the easy parts of a job, you don't remove the human. You leave them the hardest parts, while simultaneously eroding the hands-on skill they need to handle those parts. The autopilot flies the plane for ten thousand hours, then hands it back to a startled human in precisely the situation the autopilot couldn't handle.

She was writing about industrial control rooms. She was also, it turns out, writing about coding agents. An agent that handles 95% of a change competently is training its reviewer to stop reading. The human-factors literature calls this automation bias. Raja Parasuraman and Victor Riley documented through the 1990s how reliably people over-trust automation that is usually right. "Usually right" is the most dangerous kind of right, because it degrades exactly the vigilance you need for the exceptions.

So the naive setup, where the agent does the work and the human "supervises," quietly decays into nobody actually being responsible, while everyone assumes someone is. That's not a hypothetical failure mode. It's the default one.

Moral crumple zones

The researcher Madeleine Clare Elish coined a term I think about constantly: the moral crumple zone. In a complex automated system, when something goes wrong, blame collapses onto the nearest human: the operator, the driver, the reviewer, regardless of how much actual control they had. Like the crumple zone of a car, the human absorbs the impact so the system doesn't have to.

If you don't design accountability deliberately, that's what you get by default: responsibility stays ambient and unowned right up until the incident, at which point it lands entirely on whoever happened to be standing closest. That arrangement is unjust to the person and useless to the organization, because blaming the nearest human teaches you nothing about the system that set them up.

What we do instead

The design principle is simple to state: AI accelerates the work; it never absorbs the responsibility. In practice:

• We own what ships, and we don't pretend vigilance scales. Bainbridge's irony predicts what happens when you ask a person to hand-read every diff a tireless agent produces: attention decays into rubber-stamping. So the quality bar is enforced by engineered guardrails: tests, validation gates, checks every change must clear before it can reach production. Human time goes where it compounds: hardening and improving that harness. A person still owns the decision to ship; the guardrails are what make that ownership honest.
• Everything AI-influenced in production has a named human owner. Not a team, not a rotation, not "the platform." A name. Named ownership is the one intervention that reliably defeats the crumple zone, because the owner knows in advance that the pager points at them, and people who know that in advance build very different guardrails.
• When an AI incident happens, the postmortem interrogates our process, not the model. The model did what models do; it's a probabilistic system behaving probabilistically. The interesting questions are ours: why did the harness let this output through? What check was missing? What guardrail do we build so it can't happen again? "The model hallucinated" is a description, not a root cause.

The bar

"The agent shipped it" will never be an acceptable answer at Sigao for anything that matters, not because we're cautious about AI, but because we're serious about it. I want agents doing more of our work next year than they did this year. The only way that's responsible is if the chain of human accountability strengthens at exactly the rate the automation does. Bainbridge saw it four decades ago: the more capable the automation, the more critical, and the more deliberate, the human role becomes. We invest in the people who steward our systems because that isn't a cost of using AI. It's the thing that makes using AI defensible.

Sources

• Lisanne Bainbridge, "Ironies of Automation", Automatica, 1983.
• Raja Parasuraman and Victor Riley, "Humans and Automation: Use, Misuse, Disuse, Abuse", Human Factors, 1997.
• Madeleine Clare Elish, "Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction", Engaging Science, Technology, and Society, 2019.