A Rational Optimist View Of Preventing Agency Decay

Becoming Aware Of The Silent Migration Of Cognition

“Ensuring that AI expands access, agency, and opportunity is a central challenge.” ~ OpenAI Policy paper

The shift from human oversight to AI integration is often framed as a miracle of efficiency, but beneath the surface of productivity gains lies a quieter, more systemic transformation. This is not a sudden takeover by a rogue superintelligence. It is a potential cognitive offloading that leads to the gradual hollowing out of human agency.

“This shift will reshape how organizations run, how knowledge is created, and how people find meaning and opportunity.” ~ OpenAI Policy paper

The business case for AI is undeniable. Companies like AES, an American utility and power generation company with significant operations across North and South America, have seen audit processes that previously took fourteen days shrink to just one hour thanks to AI agents. On paper, this is a triumph. By handling half the workload, AI allows for a doubling of output while maintaining a human in the loop for the remaining half.

However, this human in the loop may be a fragile tether. As the Gradual Disempowerment paper argues, societal systems remain aligned with human interests only because they historically needed human participation to function. When AI becomes a superior substitute for human cognition across a very large number of domains, the incentives for these employment systems to ensure human flourishing become untethered. We are not just speeding up the audit. We are beginning to remove the structural necessity of the auditor.

OpenAI’s latest policy paper warns that wealth and power could become more concentrated, that democratic values could be undermined, and that workers need a voice in how these systems are deployed. This is something that I wholeheartedly agree with. The interesting thing is not that critics say this, the interesting thing is that the builders now feel compelled to say it too.

The Boiling Frog

We are very good at talking about artificial intelligence as if it were either a miraculous assistant or an apocalyptic sovereign. We are much worse at describing the long middle, the quiet migration in which initiative slips, task by task and judgment by judgment, from human beings to corporate systems that do not exactly rule us and yet steadily reorganize the terms under which we think, approve, create, and contest. Not a coup. A substitution. Not the clang of conquest, but the administrative transfer of authorship.

This is why the word productivity, so innocent in boardrooms, now deserves suspicion. Productivity is not a false god. It is simply a greedy one. It asks the same question over and over: faster than what, cheaper than what, at greater scale than what? Those are not foolish questions. They are often necessary questions. But they are incomplete. They do not ask what kind of human being remains on the other side of the optimization. They do not ask what capacities are being exercised, and which are being quietly retired. They do not ask whether the person left supervising the machine is still practicing judgment or merely performing assent.

A one-hour audit is not merely a better audit process. It is a different theory of the auditor. But, there is hope. In an excellent research study from Stanford, the authors show that “the acceleration is remarkably slow because of the prominence of “weak links,” i.e., an elasticity of substitution among tasks.” It will not be possible to fully rely on 100% AI automation; there are still tasks that will have to be done manually.

That is a long ‘take off’. The boiling frog scenario.

Hollowing Out

The problem is not simply displacement in the vulgar sense of jobs disappearing. That is real enough, and bad enough, but it is only the visible edge of a deeper shift. The more consequential danger is the hollowing out of the human role before the human role disappears. First the worker is aided. Then the worker is supervised by the tool he is said to supervise. Then the worker becomes the legal residue of accountability in a process whose substantive intelligence now resides elsewhere. The signature remains human. The reasoning has migrated.

The authors of the paper on Gradual Disempowerment understand this with unusual clarity. Their central claim is not that some rogue superintelligence will wake up one morning and push humanity off the stage. It is more unsettling than that. The authors argue that human societies remain aligned with human interests partly because they still require human participation. Economies need our labor and consumption. States need our taxes, compliance, and sometimes our consent. Culture still depends on our attention, imitation, aspiration, and response. The system does not love us. It needs us. And because it needs us, it has had reasons, however imperfect, to remain at least partly answerable to us.

Now imagine that this necessity begins to weaken.

Imagine a production system that no longer needs much human cognition. Imagine firms that can create, analyze, sort, forecast, persuade, and coordinate at scale with machine substitutes that are cheaper, faster, more docile, and easier to replicate than human beings. Imagine that the institutions around us continue to function, perhaps even more efficiently than before, while relying less and less on human judgment as an input. In that world, the old bargain changes. Human flourishing becomes less a structural requirement than a sentimental afterthought.

That is the real sting in the phrase human in the loop. It sounds reassuring because it suggests sovereignty. In practice it can mean anything from genuine review to ceremonial ratification. The loop is not a guarantee of power.

Task And Risk

The Stanford Enterprise AI Playbook, to its credit, is more honest than most of the corporate genre. It notes that successful deployments vary by task and risk, and that structured human oversight will be still be required. especially, in high-stakes work where human review remains the strategically correct choice. Yet the same document also reports that escalation-based models, where AI handles more than 80 percent of the work and humans review only exceptions, delivered the highest median productivity gains. One does not need to be a cynic to see where the pressure points will gather. Once the organization has learned that the machine can do most of the work, the remaining human fraction begins to look, from the spreadsheet’s point of view, like an inefficiency waiting for courage.

This is how agency is lost in modern institutions. Not because anyone formally abolishes it, but because the price of keeping it starts to look indulgent.

There is an older vocabulary for this problem, and it is more illuminating than our current jargon. We speak casually of cognitive offloading, and sometimes that is exactly what is happening. There is nothing inherently tragic in refusing to memorize every phone number or in letting software carry out routine calculation. Civilization is built on forms of offloading. Writing was an offloading. The ledger was an offloading. The printed book was an offloading. Nobody sensible wants to return to the memorization feats of preliterate administration.

But what the resilience literature now calls agency decay is a different matter. Cognitive offloading concerns the transfer of tasks. Agency decay concerns the transfer of consequential judgment. One can lose neither memory nor skill in any dramatic sense and still become a passive observer of one’s own decisions. This is the more insidious atrophy, because it hides beneath apparent competence. The person is still present. The institution still says he is responsible. The workflow still contains a human checkpoint. Yet the actual conditions that make authorship meaningful have thinned. He or she no longer initiates inquiry, no longer frames uncertainty, no longer builds the internal muscle required to say, with confidence and cost, no.

This is why diversity of thought is important in a much deeper sense than the usual innovation cliché. Jascha Sohl-Dickstein has argued that the survival of complex systems depends on diversity. That insight belongs not only to biology but to civilization. A culture able to generate multiple strategies, rival interpretations, and genuinely different forms of judgment is more resilient than one optimized around a single high-performing template. The trouble with large-model mediation is not merely that it may produce errors. It is that it may produce a smooth and persuasive sameness. The same cadences. The same framings. The same sanitized competence. The same latent habits of inference appearing across audits, reports, strategies, essays, and public language.

A monoculture is efficient until reality changes. Then it is a famine.

The NIST AI Risk Management Framework is useful precisely because it does not accept the fantasy that AI risk is only a technical matter. It insists on governance, mapping, measurement, and management across the full organizational setting. That sounds bureaucratic, which is unfortunate, because the underlying insight is rather profound. The risk is never just the model. The risk is the arrangement into which the model is inserted. Who can contest its outputs? Who understands the system well enough to challenge it? What incentives are attached to speed? What happens to the human capacities the system displaces? How is accountability preserved when judgment is distributed across software, workflow, legal form, and exhausted employees clicking approve before lunch?

Human Buffer

The best recent writing on resilience pushes this farther. Human resilience, at its strongest, is not mere endurance. It is not the ability to keep functioning inside a system that has quietly stripped one of authorship. A well-fed dependency is still a dependency. Real resilience means retaining the capacity to remain an active author of meaning, judgment, and responsibility even when one works alongside powerful non-human systems. That requires more than literacy in prompts or dashboards. It requires protected zones of human difficulty. It requires occasions when the answer does not arrive in nano-seconds, when deliberation is not bypassed, when ambiguity is not treated as a design flaw, when the institution deliberately preserves a human buffer against its own appetite for streamlining.

That phrase, human buffer, may sound modest. It is not. It is a civilizational demand. It means preserving domains in which human beings still practice the arts that make self-government possible: interpretation, contestation, doubt, refusal, narration, moral comparison, strategic patience. If every meaningful process is optimized for immediate machine mediation, these capacities do not disappear all at once. They soften. Then they rust. Then, one day, an institution discovers that it still has people on the org chart but very few adults left who can think without scaffolding.

This is why the economic question becomes political, though not quite in the fatalistic way the darker version of this argument might suggest. The pressure toward one-hour efficiency is real because, at the level of the firm, it is often rational. Faster throughput, lower labor costs, more scale, fewer delays, cleaner reporting. These are not hallucinations of finance. They are genuine advantages. A serious argument cannot begin by pretending otherwise. Yet, isn’t audit a crucial task?

But I have a rational optimism. I do not deny the greedy logic of productivity. I ask whether that logic, properly understood, is actually narrower than it first appears.

The first hopeful truth is that productivity is not a single thing. A workflow can be efficient in the crude sense of reducing labor minutes and still be profoundly inefficient in the larger sense that matters to institutions that expect to survive contact with reality. A system that moves quickly but cannot be contested, cannot recover from novel failure, cannot explain itself under pressure, and cannot preserve the expertise required to challenge it is not, in the deepest sense, efficient. It is merely fast. And speed, outside a narrow band of conditions, is one of the easiest virtues for a civilization to overpraise.

Positive Outcomes

This is where the rational optimist in me suggests that we reconsider the AI alarm and seek a positive outcome for humanity. I argue that the human buffer need not be defended as a nostalgic preference or a moral ornament. It can be defended as a performance asset. In high-stakes domains, retained human judgment is not a sentimental relic from a pre-automated age. It is reserve capacity. It is error recovery. It is strategic adaptability. It is the ability to notice when the world has changed in a way the model did not foresee. It is the capacity to say that the machine is wrong not because a dashboard signaled an anomaly, but because a trained human being, drawing on tacit knowledge, institutional memory, and moral seriousness, recognized that something essential had been missed.

Once one sees this, the economic case becomes less one-sided than the evangelists suggest. The question is not whether firms will always be tempted to replace judgment with automation. Of course they will. The question is whether they can be made to see that some forms of retained judgment are themselves productive. A company does not call a fire exit inefficient because most days no one uses it. A hospital does not call sterile backups wasteful because the primary system worked yesterday. Human competence of the right kind belongs in the same category. It is infrastructure.

That means the task is not to defeat productivity. It is to rescue productivity from its most primitive definition.

A more mature definition would include resilience, contestability, trust, error recovery, reputational durability, and the preservation of expertise. It would ask not merely how much labor was removed, but whether the institution remains capable of understanding and governing the process it has accelerated. It would distinguish between organizations that have truly augmented their people and those that have simply consumed their own future competence for a pleasing quarter or two of metrics.

This is also where governance can be less a brake than a clarifier. Good governance does not exist to protect society from progress. It exists to make explicit what kinds of progress are real. If a firm must be able to explain an AI-mediated decision, if a citizen must be able to contest it, if a professional must remain substantively rather than ceremonially accountable for it, then the organization begins to discover something important: some measure of human authorship is not external to the system’s success. It is one of the conditions of that success.

The rational optimist therefore does not place his or her faith in corporate benevolence. He or she places it in a broader and sturdier possibility: that our incentives can be redesigned, our metrics widened, our reporting standards improved, our professional norms sharpened, and our institutions taught to distinguish between mere acceleration and genuine capability. We should assume, correctly, that firms respond to cost, liability, and competitive pressure. But also we should assume that those pressures are malleable. They are shaped by law, custom, accounting, reputation, insurance, procurement rules, professional accreditation, and the slowly changing moral vocabulary through which a society decides what counts as competent stewardship.

In that sense, optimism is not softness. It is design discipline.

It says that one-hour efficiency should be welcomed where it truly serves human purposes. Let the machine do the drudgery. Let it clear the underbrush. Let it compress the parts of work that are repetitive, mechanical, and genuinely beneath the best use of trained minds. But let us not confuse this with a license to retire the difficult faculties by which institutions remain humanly governed. The aim is not to preserve friction for its own sake. It is to preserve authorship where authorship remains significant.

Work on High Risk Tasks

That is the more hopeful ending, and also the more demanding one. We may yet prove capable of resisting the narrowest version of the productivity god, not by smashing the machine or romanticizing inefficiency, but by learning to price what the spreadsheet initially cannot see. Judgment. Redundancy. Contestability. Strategic patience. Moral seriousness. The trained capacity to notice when the map is still elegant and the territory has already caught fire.

If we can do that, then AI will not mark the end of human agency. It will force us, perhaps for the first time in a very long while, to specify what human agency is actually for.

And that, though less glamorous than the old fantasies of salvation or doom, is a rationally optimistic future. Not one in which the machine stops advancing, and not one in which the market stops optimizing, but one in which we become intelligent enough to govern what we have built without surrendering the very faculties that made the building possible.

Stay curious

Colin

Image - The rowing team from the La Cala del Moral Rowing Club during a traditional fishing boat race held on the beaches of La Carhuela (Málaga). This type of race is only held in the waters off Málaga.